Audio watermark embedding method and device, electronic equipment and storage medium

Abstract

The application provides an audio watermark embedding method and device, electronic equipment and storage medium, and belongs to the technical field of audio processing. The method comprises the following steps: obtaining the amplitude spectrum feature of an original audio signal and determining a watermark embedding energy mask; inputting the amplitude spectrum feature into a plurality of mask prediction network models to obtain a watermark embedding risk mask; determining a target watermark embedding risk mask according to the watermark embedding risk mask to generate a frame-level embedding mask; embedding watermark information into the feature component corresponding to the target audio frame to obtain a watermark-containing amplitude spectrum feature, and performing inverse time-frequency conversion on the watermark-containing amplitude spectrum feature to obtain a watermark-containing audio signal. The application combines audio energy and model prediction risk to adaptively select the most suitable audio frame for embedding a watermark, avoids modification in silent, transient and other auditory sensitive frames, solves the problem of audio quality degradation caused by indiscriminate watermark embedding, and significantly improves the imperceptibility of the watermark and the auditory quality of the watermark-containing audio without sacrificing the robustness of the watermark.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and in particular to an audio watermark embedding method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the widespread application of digital multimedia technologies (including but not limited to audio, images, and video) and the continuous evolution of their dissemination methods, the issues of copyright protection and authenticity verification of digital media content have become increasingly prominent. Audio, as one of the important information carriers in the digital multimedia field, faces numerous security challenges due to its rapid dissemination and ease of copying and modification. Digital watermarking technology, which embeds hidden markers into audio to declare copyright or verify integrity, has become a key technology in this field. Its core requirement is to achieve effective information embedding without affecting the normal auditory experience.

[0003] To meet the above requirements, a mainstream existing technical solution adopts an end-to-end architecture based on neural networks. This solution first performs a Short Time Fourier Transform (STFT) on the original audio signal, converting it into a two-dimensional time-frequency amplitude spectrum composed of multiple audio frames; then, based on the watermark information to be embedded, a modulation matrix is ​​generated through an encoding network; finally, this modulation matrix is ​​uniformly and indiscriminately applied to the entire amplitude spectrum, globally modifying all its time-frequency units, and then reconstructing the watermarked audio through an inverse transform.

[0004] However, the aforementioned existing technical solutions have inherent design flaws. This approach, which uniformly applies watermark modulation across the entire time spectrum, introduces noise or distortion perceptible to the auditory system after reconstructing the amplitude spectrum of the watermarked signal and converting it back to the time-domain audio signal. This distortion directly compromises the key technical indicator of the watermark's imperceptibility and leads to a decline in the final audio listening quality. Summary of the Invention

[0005] This invention provides an audio watermark embedding method, apparatus, electronic device, and storage medium, aiming to solve the technical problem in the prior art that the use of a global and indiscriminate embedding strategy introduces perceptible distortion in the reconstructed audio, thereby impairing the imperceptibility of the watermark and reducing the audio listening quality.

[0006] This invention provides an audio watermark embedding method, comprising the following steps:

[0007] Perform time-frequency transformation on the original audio signal to obtain the amplitude spectrum characteristics of multiple audio frames ordered by time;

[0008] Based on the energy value of each audio frame in the original audio signal, a watermark embedding energy mask is determined;

[0009] The amplitude spectrum features are input into a model combination consisting of multiple mask prediction network models to obtain multiple different watermark embedding risk masks output by the model combination.

[0010] Based on all the watermark embedding risk masks, determine a target watermark embedding risk mask;

[0011] Based on the watermark embedding energy mask and the target watermark embedding risk mask, a frame-level embedding mask is generated. The frame-level embedding mask is used to indicate that at least a portion of the target audio frames among all the audio frames are selected for embedding watermark information.

[0012] The watermark information is embedded into the feature components corresponding to each target audio frame in the amplitude spectrum feature to obtain the watermarked amplitude spectrum feature.

[0013] The watermarked amplitude spectrum features are subjected to inverse time-frequency transformation to obtain the watermarked audio signal.

[0014] According to an audio watermark embedding method provided by the present invention, determining a target watermark embedding risk mask based on all the watermark embedding risk masks includes:

[0015] All the watermarks are embedded into a risk mask and concatenated to form a first mask feature map;

[0016] The first mask feature map is input into multiple convolutional neural network models with the same structure but different random initialization parameters, and the second mask feature map output by each convolutional neural network model is obtained.

[0017] Multiply at least two features from all the second mask feature maps by matrix and then normalize them to obtain the third mask feature map.

[0018] The third mask feature map is multiplied by any fourth mask feature map to obtain a global mask feature map; any fourth mask feature map is any one of the second mask feature maps except for the two second mask feature maps that generate the third mask feature map;

[0019] Each of the multiple risk values ​​of the watermark embedded in the risk mask at each identical feature position is merged into a single risk value.

[0020] The risk value at each feature position of the watermark embedding risk mask is binarized to generate the target watermark embedding risk mask.

[0021] According to an audio watermark embedding method provided by the present invention, the step of binarizing the risk value at each feature position of the watermark embedding risk mask to generate the target watermark embedding risk mask includes:

[0022] Sort the risk values ​​at all feature positions in the watermark embedding risk mask in descending order;

[0023] The feature positions with the highest preset number of risk values ​​after arrangement are set to non-selected in the risk mask embedded in the binarized watermark.

[0024] Set the positions corresponding to the remaining elements to the selected state.

[0025] According to an audio watermark embedding method provided by the present invention, determining a target watermark embedding risk mask based on all the watermark embedding risk masks includes:

[0026] All the watermark embedding risk masks are sorted by size, and a binarization operation is performed on each watermark embedding risk mask according to the obtained watermark embedding risk mask sequence to obtain the binarized embedding risk mask corresponding to each watermark embedding risk mask.

[0027] Based on the mask dispersion of each of the binarized embedding risk masks, the mask dispersion corresponding to the maximum mask dispersion is determined as the target watermark embedding risk mask.

[0028] According to an audio watermark embedding method provided by the present invention, the mask dispersion of each binarized embedding risk mask is determined based on the following method:

[0029] Sort the frame indices of all audio frames with a value of 0 in the binarized embedded risk mask by size to obtain the frame index sequence;

[0030] The mask dispersion is obtained by summing the differences between all adjacent frame indices in the frame index sequence.

[0031] According to an audio watermark embedding method provided by the present invention, the step of determining a watermark embedding energy mask based on the energy value of each audio frame in the original audio signal includes:

[0032] Determine the average energy value of all audio frames in the original audio signal;

[0033] The energy value of each audio frame in the original audio signal is compared with the average energy value frame by frame to determine the watermark embedding energy mask based on the comparison results of all audio frames.

[0034] The comparison result is that for audio frames with energy values ​​greater than the average energy value, the elements mapped in the watermark embedded energy mask are selected; otherwise, the elements mapped in the watermark embedded energy mask are not selected.

[0035] According to an audio watermark embedding method provided by the present invention, the frame-level embedding mask is generated based on the watermark embedding energy mask and the watermark embedding risk mask, including:

[0036] Perform an element-wise logical OR operation on the watermark embedding energy mask and the watermark embedding risk mask to obtain the frame-level embedding mask.

[0037] According to an audio watermark embedding method provided by the present invention, the watermark information is embedded into the feature components corresponding to each target audio frame in the amplitude spectrum feature to obtain a watermarked amplitude spectrum feature, including:

[0038] By splicing the amplitude spectrum feature and the watermark information, a spliced ​​feature is obtained;

[0039] The splicing features are input into the watermark encoder network model to obtain the initial watermark modulation weight matrix output by the watermark encoder network model. The initial watermark modulation weight matrix has the same dimension as the amplitude spectrum features.

[0040] The initial watermark modulation weight matrix is ​​masked using the frame-level embedding mask to obtain the final watermark modulation weight matrix.

[0041] The final watermark modulation weight matrix is ​​applied to the feature components corresponding to each target audio frame in the amplitude spectrum feature to obtain the watermarked amplitude spectrum feature.

[0042] According to an audio watermark embedding method provided by the present invention, the final watermark modulation weight matrix is ​​applied to the feature components corresponding to each target audio frame in the amplitude spectrum feature to obtain the watermarked amplitude spectrum feature, including:

[0043] The elements corresponding to each target audio frame in the final watermark modulation weight matrix are multiplied or added to the feature components corresponding to each target audio frame in the amplitude spectrum feature, respectively, to obtain the watermarked amplitude spectrum feature.

[0044] According to the audio watermark embedding method provided by the present invention, the watermark encoder network model is a deep residual network model.

[0045] According to the audio watermark embedding method provided by the present invention, the mask prediction network model and the watermark encoder network model are obtained through the following steps:

[0046] Determine each audio signal sample in the training set, and the initial watermark information for watermark embedding operation on each audio signal sample;

[0047] Based on the initial watermark information, a watermarked audio signal sample is generated for each audio signal sample, and a frame-level embedding mask sample is determined for each audio signal sample.

[0048] The watermarked audio signal sample is input into the watermark extraction network model to obtain the watermark information output by the watermark extraction network model.

[0049] The total training loss is calculated based on the audio signal sample, the watermarked audio signal sample, the initial watermark information, the identified watermark information, and the frame-level embedding mask sample.

[0050] The trainable parameters of the mask prediction network model, the watermark encoder network model, and the watermark extraction network model are updated based on the total training loss.

[0051] According to an audio watermark embedding method provided by the present invention, the total training loss includes at least the adjacent mask frame index loss, which is calculated based on the following steps:

[0052] Identify the frame index corresponding to the element with a value of 1 in the frame-level embedding mask sample to obtain the frame index set;

[0053] Calculate the interval distance between every two temporally adjacent frame indices in the frame index set, and when the interval distance is less than a preset distance threshold, determine a loss value based on the difference between the interval distance and the preset distance threshold;

[0054] The adjacent mask frame index loss is obtained by summing all the loss values.

[0055] According to an audio watermark embedding method provided by the present invention, the total training loss includes at least an audio distortion loss, which is calculated based on the following steps:

[0056] Obtain the original amplitude spectrum features corresponding to the audio signal sample;

[0057] Obtain the watermark amplitude spectrum features corresponding to the watermarked audio signal sample;

[0058] The difference between the original amplitude spectrum feature and the watermarked amplitude spectrum feature is calculated to obtain the audio distortion loss.

[0059] According to an audio watermark embedding method provided by the present invention, the total training loss includes at least a watermark extraction loss, which is calculated based on the following steps:

[0060] The difference between the initial watermark information and the identified watermark information is calculated to obtain the watermark extraction loss.

[0061] According to an audio watermark embedding method provided by the present invention, before inputting the watermarked audio signal sample into the watermark extraction network model, the method further includes:

[0062] At least one attack simulation is applied to the watermarked audio signal sample;

[0063] The attack simulation includes at least one of adding Gaussian white noise, performing lossy compression, random frame cutting, or resampling.

[0064] According to an audio watermark embedding method provided by the present invention, the step of performing inverse time-frequency transform on the watermarked amplitude spectrum features to obtain a watermarked audio signal includes:

[0065] Obtain the phase spectrum features synchronously generated during time-frequency transformation of the original audio signal;

[0066] The watermarked audio signal is obtained by combining the amplitude spectrum feature and the phase spectrum feature of the watermark and performing an inverse time-frequency transform on the combined result.

[0067] According to an audio watermark embedding method provided by the present invention, the watermark information is determined based on the following steps:

[0068] A baseline watermark information is randomly selected from the watermark codebook;

[0069] The reference watermark information is encoded and then copied at the frame level to generate watermark information that matches the number of audio frames.

[0070] The present invention also provides an audio watermark embedding device, comprising:

[0071] The time-frequency conversion unit performs time-frequency transformation on the original audio signal to obtain the amplitude spectrum characteristics of multiple audio frames ordered by time.

[0072] The energy mask determination unit determines a watermark embedded energy mask based on the energy value of each audio frame in the original audio signal.

[0073] The risk mask acquisition unit inputs the amplitude spectrum features into a model combination composed of multiple mask prediction network models to obtain multiple different watermark embedding risk masks output by the model combination.

[0074] The target mask determination unit determines a target watermark embedding risk mask based on all the watermark embedding risk masks.

[0075] A frame-level mask generation unit generates a frame-level embedding mask based on the watermark embedding energy mask and the target watermark embedding risk mask. The frame-level embedding mask is used to indicate that at least a portion of the target audio frames among all the audio frames are selected for embedding watermark information.

[0076] The watermark embedding unit embeds the watermark information into the feature components corresponding to each target audio frame in the amplitude spectrum feature to obtain the watermarked amplitude spectrum feature.

[0077] The inverse time-frequency conversion unit performs inverse time-frequency transformation on the watermarked amplitude spectrum features to obtain the watermarked audio signal.

[0078] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the audio watermark embedding method as described above.

[0079] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio watermark embedding method as described above.

[0080] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the audio watermark embedding method as described above.

[0081] The audio watermark embedding method, apparatus, electronic device, and storage medium provided by this invention combine audio energy and model prediction risk to adaptively select the most suitable audio frames for watermark embedding, avoiding modifications in auditory sensitive frames such as silence and transients. This solves the problem of audio quality degradation caused by indiscriminate watermark embedding, and significantly improves the imperceptibility of watermarks and the auditory quality of watermarked audio without sacrificing watermark robustness. Attached Figure Description

[0082] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0083] Figure 1 This is one of the flowcharts illustrating the audio watermark embedding method provided by the present invention.

[0084] Figure 2 This is a flowchart illustrating the target watermark embedding risk mask generation method provided by the present invention.

[0085] Figure 3 This is a flowchart illustrating the watermark embedding energy mask generation method provided by the present invention.

[0086] Figure 4 This is a schematic diagram of the method for obtaining watermark amplitude spectrum features provided by the present invention.

[0087] Figure 5 This is a flowchart illustrating the model training method provided by the present invention.

[0088] Figure 6 This is the second flowchart of the audio watermark embedding method provided by the present invention.

[0089] Figure 7 This is a schematic diagram of the audio watermark embedding device provided by the present invention.

[0090] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0091] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0092] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0093] In addition, "and / or" indicates that at least one of the connected objects is connected, and the character " / " generally indicates that the objects before and after are in an "or" relationship.

[0094] With the development of digital multimedia technology, the issues of copyright protection and authenticity verification for audio content have become increasingly prominent. Currently, there are various low-cost methods of tampering, forgery, and splicing that can modify or generate digital media content circulating on the mobile internet in bulk, posing a potential threat to the cognitive security of ordinary audiences.

[0095] Traditional audio content protection mechanisms typically borrow from codec patterns in communication systems. This involves encrypting the original audio signal, converting it into unreadable ciphertext. In this encrypted form, the information is invisible to the outside world during transmission. Even if intercepted, unauthorized users without the corresponding decoding algorithm cannot access the valid content, thus achieving copyright and content protection. This mechanism relies on the assumption that unauthorized parties cannot obtain the codec algorithm. However, with technological advancements, the difficulty of cracking codec algorithms has significantly decreased. Unauthorized individuals may illegally decrypt ciphertext data transmitted over the mobile internet, implant malicious information, re-encrypt it using the same encoding algorithm, and then publicly release it. Due to the lack of effective authentication mechanisms, it is difficult to identify such illegally tampered audio content, leading to vulnerabilities in the public transmission environment and causing confusion and disruption to user perception.

[0096] To address this, digital watermarking technology offers a covert and effective method to identify copyright ownership or verify content authenticity by embedding imperceptible watermark information into the original audio signal. The key challenge of this technology lies in ensuring that the embedded watermark information is not only transparent to the end user but also resistant to attacks from various signal processing operations. Particularly in the field of digital audio, audio digital watermarking can achieve multiple functions such as copyright protection, content verification, and copy control, making it of significant research value for improving audio security.

[0097] However, the development of audio watermarking faces more complex challenges than image and video watermarking. Besides balancing imperceptibility and robustness, the high sensitivity of the human auditory system must also be considered, making the requirements for audio quality more stringent. Therefore, in-depth research into audio digital watermarking technology is not only a technological necessity but also an inevitable trend in the pursuit of high-quality audio experiences.

[0098] Current mainstream neural network-based audio watermarking embedding methods generally adopt an end-to-end architecture. Based on the watermark information to be embedded, the amplitude spectrum characteristics of all audio frames obtained after the original audio signal is subjected to short-time Fourier transform (STFT) are globally corrected, and then combined with the original phase spectrum. Finally, the watermarked audio signal is reconstructed through inverse short-time Fourier transform (ISTFT).

[0099] Research has revealed that while existing methods are relatively simple to implement and possess a certain degree of robustness, their homogenization strategies neglect the local perceptual characteristics and non-uniform energy distribution of audio signals in the time-frequency domain. Specifically, not all amplitude spectrum feature components of audio frames are suitable for embedding watermark information. For example, forcibly embedding watermarks in low-energy audio frames (such as silent segments or weak signal segments), transient audio frames (such as high dynamic range segments like impact sounds and plosives), or audio frames corresponding to frequency bands sensitive to human hearing, can easily introduce perceptible distortion or noise, leading to a decrease in the auditory fluency and naturalness of the watermarked audio signal, severely compromising the imperceptibility of the watermark. Especially in speech signals, some phonemes (such as vowel transition segments or consonant plosive segments) are extremely sensitive to spectral perturbations; even minor alterations can weaken the intelligibility and prosodic naturalness of the speech signal, thus affecting the auditory experience.

[0100] Based on the above research findings, this invention provides an audio watermark embedding method based on adaptive frame-level masking, which can maintain the readability of audio content while enhancing its security. The following section combines... Figures 1-8 This invention describes the audio watermark embedding method, apparatus, electronic device, and storage medium provided by the present invention.

[0101] It should be noted that the audio watermark embedding method provided by this invention can be deployed on various types of computing devices according to different application needs, performance requirements, and business scenarios. For example, it can be deployed on a cloud server, or directly on a personal computer or mobile smart device, or integrated into dedicated hardware as an embedded system. This invention does not specifically limit this. For ease of description, the entity executing the method of this invention will be collectively referred to as the system in the following embodiments.

[0102] Figure 1 This is one of the flowcharts illustrating the audio watermark embedding method provided by the present invention, such as... Figure 1 As shown, including but not limited to the following steps:

[0103] Step 11: Perform time-frequency transformation on the original audio signal to obtain the amplitude spectrum features of multiple audio frames sorted by time.

[0104] The original audio signal can be any format of digital audio data, such as WAVE, MP3, or AAC, which the system reads from local storage media or receives via a network interface. Considering that audio signals are typical non-stationary time-varying signals, this embodiment uses time-frequency transformation, such as short-time Fourier transform, to convert the one-dimensional time domain signal to a two-dimensional time-frequency domain for refined analysis and processing.

[0105] Specifically, the system can first divide the original audio signal into a series of audio frames that are continuous in time and partially overlap, such as short segments with a frame length of 1024 sampling points and a frame shift of 512 sampling points, and apply a window function (such as the Hanning window) to each audio frame to reduce spectral leakage.

[0106] Subsequently, a Fast Fourier Transform (FFT) is performed on each windowed audio frame to obtain its complex spectrum. The magnitude of this complex spectrum constitutes the feature components of that audio frame. Arranging the feature components of all audio frames in chronological order creates a two-dimensional amplitude spectrum feature, which describes the distribution of signal energy in the time-frequency domain.

[0107] Step 12: Determine a watermark embedding energy mask based on the energy value of each audio frame in the original audio signal.

[0108] Specifically, the system calculates the average energy value of all audio frames. Then, it compares the energy value of each audio frame with this average energy value frame by frame. If the energy value of the current frame is greater than the average energy value, the audio frame is considered suitable for watermarking and is marked as "selected" (e.g., a value of 1) at the corresponding position in the watermark embedding energy mask; conversely, if the energy value is less than or equal to the average energy value (e.g., a silent or weak signal segment), the audio frame is considered unsuitable for embedding and is marked as "unselected" (e.g., a value of 0). This generates a one-dimensional binary vector of the same length as the number of audio frames, which is the watermark embedding energy mask.

[0109] Step 13: Input the amplitude spectrum features into a model combination consisting of multiple mask prediction network models to obtain multiple different watermark embedding risk masks output by the model combination.

[0110] This embodiment utilizes a deep learning model to predict the distortion risk that may be introduced by embedding watermarks in each frame from the perspective of auditory perception. The model combination consists of multiple (e.g., M) mask prediction network models (such as deep neural networks, DNNs) with the same structure but different randomly initialized parameters.

[0111] The system inputs the amplitude spectrum features obtained in step 11 into these M mask prediction network models, and each mask prediction network model independently outputs a watermark embedding risk mask. Due to the different initialization parameters of the models, their judgments on risk will have slight differences, resulting in M ​​different risk masks, which enhances the robustness of subsequent decisions.

[0112] Step 14: Determine a target watermark embedding risk mask based on all the watermark embedding risk masks.

[0113] To obtain a final mask for decision-making from the multiple risk masks generated in the previous step, the system fuses or optimizes these M different watermark embedding risk masks. For example, the information from these M masks can be integrated using a specific fusion algorithm, or an optimal mask can be selected based on preset evaluation metrics (such as mask dispersion). The output of this process is a single, comprehensively evaluated target watermark embedding risk mask.

[0114] Step 15: Generate a frame-level embedding mask based on the watermark embedding energy mask and the target watermark embedding risk mask.

[0115] The system combines the watermark embedding energy mask generated in step 12 with the target watermark embedding risk mask determined in step 14 to generate the final frame-level embedding mask.

[0116] One specific approach is to perform an element-wise logical OR operation. This operation ensures that an audio frame is marked as a selectable target audio frame for watermark embedding only if both the energy mask and the target risk mask simultaneously determine it to be "selected." This guarantees that the watermark is only embedded in high-energy audio frames with low perceived risk.

[0117] Step 16: Embed the watermark information into the feature components corresponding to each target audio frame in the amplitude spectrum feature to obtain the watermarked amplitude spectrum feature.

[0118] The watermark information is a pre-defined binary code stream. Based on the frame-level embedding mask generated in step 15, the system modifies the corresponding feature components in the amplitude spectrum characteristics of only the audio frames marked as target audio frames to embed the watermark. For all other unselected audio frames, the corresponding feature components remain unchanged. The embedding operation can be achieved by adding or multiplying a modulation signal associated with the watermark information with the feature components of the target audio frame. After the operation, the original amplitude spectrum characteristics are transformed into watermarked amplitude spectrum characteristics.

[0119] Step 17: Perform inverse time-frequency transformation on the watermarked amplitude spectrum features to obtain the watermarked audio signal.

[0120] This step is the inverse of the time-frequency transformation in step 11, preferably the inverse short-time Fourier transform (ISTFT). For accurate reconstruction, the system combines the original phase spectrum features acquired and saved synchronously in step 11 with the watermarked amplitude spectrum features generated in step 16 to form a complete complex spectrum matrix. Then, the ISTFT is performed on this complex spectrum matrix to ultimately reconstruct a one-dimensional time-domain, playable watermarked audio signal.

[0121] The audio watermark embedding method provided by this invention combines audio energy and model prediction risk to adaptively select the most suitable audio frames for watermark embedding, avoiding modifications in auditory sensitive frames such as silence and transients. This solves the problem of audio quality degradation caused by indiscriminate watermark embedding. Without sacrificing watermark robustness, it significantly improves the imperceptibility of watermarks and the auditory quality of watermarked audio.

[0122] Figure 2 This is a flowchart illustrating the target watermark embedding risk mask generation method provided by the present invention, as shown below. Figure 2 As shown, the main steps include, but are not limited to, the following:

[0123] Step 141 involves concatenating all the watermarks embedded in the risk mask to form a first mask feature map. The purpose of this step is to integrate the independent risk mask vectors output from multiple parallel mask prediction network models into a unified multi-channel data structure. Through this operation, subsequent neural networks can simultaneously process the prediction results of all mask prediction network models and analyze the correlations between them.

[0124] Optionally, the splicing method adopted is to stack multiple one-dimensional risk mask vectors along a new dimension to form a two-dimensional matrix, which constitutes the first mask feature map.

[0125] Step 142: Input the first mask feature map into multiple convolutional neural networks (CNNs) with the same structure but different random initialization parameters to obtain the second mask feature map output by each CNN model.

[0126] The purpose of this step is to leverage the powerful feature learning capabilities of convolutional neural networks to extract deeper and more abstract feature representations from the first mask feature map. By employing multiple convolutional neural network models with identical structures but different initialization parameters, each branch can learn slightly different feature representations, adding diversity for subsequent fusion.

[0127] Step 143: Multiply at least two of the second mask feature maps by matrix and then normalize them to obtain the third mask feature map.

[0128] The purpose of this step is to enable information exchange between the feature representations extracted from different branches. Matrix multiplication here can be viewed as an attention mechanism, that is, using one mask feature map to weight another mask feature map, thereby highlighting key information. The normalization operation ensures that the fused feature values ​​are distributed within a reasonable range.

[0129] Specifically, two of the multiple second mask feature maps output in step 142 are randomly selected, and after performing matrix multiplication, a normalization function (such as Softmax, Sigmoid, etc.) is applied to the matrix multiplication operation to obtain the third mask feature map.

[0130] Step 144: Perform matrix multiplication between the third mask feature map and any fourth mask feature map to obtain the global mask feature map.

[0131] The purpose of this step is to ensure that the feature information from all branches participates in the final fusion process. By interacting the fused third mask feature map with a feature map that has not yet been fused (such as the fourth mask feature map), comprehensive information aggregation is achieved.

[0132] Wherein, any fourth mask feature map is any one of the second mask feature maps other than the two second mask feature maps that generate the third mask feature map.

[0133] Step 145: Merge multiple risk values ​​of all watermarks embedded in the risk mask at each identical feature position into a single risk value.

[0134] The purpose of this step is to convert the global mask feature map, which is obtained after multi-level fusion and still contains multiple risk values ​​(corresponding to different fusion channels) at each audio frame position, into a vector that corresponds to only one unique risk value for each audio frame.

[0135] Specifically, a pooling operation, such as max pooling or average pooling, can be performed on the global mask feature map along the dimensions representing different fusion channels to obtain a single risk value at each feature location.

[0136] Step 146: Binarize the risk value at each feature position of the watermark embedding risk mask to generate the target watermark embedding risk mask.

[0137] The purpose of this step is to make a clear binary decision on whether to embed a watermark for each audio frame, based on the single risk value vector generated in the previous step.

[0138] A threshold or rule can be set first to convert continuous or multi-valued values ​​in the risk vector into two states: "selected" and "unselected," thereby generating the final target watermark embedding risk mask.

[0139] The following example uses three convolutional neural network models to briefly introduce the process of generating a target watermark embedding risk mask provided in this embodiment of the invention.

[0140] The system stacks three one-dimensional watermark embedding risk masks output by three mask prediction network models (assuming the size of each watermark embedding risk mask is 1). f), which can form a first mask feature map A with size (3, f).

[0141] The first mask feature map of (3, f) is obtained. A When these features are input into three CNN models with identical structures but different randomly initialized parameters, three second mask feature maps of size (3, f) are obtained, denoted as... , and .

[0142] Choose , The two second mask feature maps are multiplied by matrix, and the result is normalized using the Softmax function to obtain the third mask feature map. .

[0143] The third mask feature map obtained in the previous step With the remaining fourth mask feature map By performing matrix multiplication again, the global mask feature map can be obtained. .

[0144] Furthermore, the system will analyze the global mask feature map. Max pooling is performed on its first dimension (i.e., the channel dimension, M=3). That is, for each audio frame position, the largest risk value is selected from the three channels, and finally a one-dimensional risk vector of size (1, f) is obtained.

[0145] Finally, the one-dimensional risk vector is binarized to generate a target watermark embedding risk mask.

[0146] This invention goes beyond simple voting or averaging of multiple risk masks. Instead, it actively learns the complex, non-linear relationships between multiple risk predictions through a specialized neural network structure. Matrix multiplication acts as an attention mechanism, dynamically assessing the importance of different predictions based on context, thus generating a more accurate and reliable final risk assessment. Simultaneously, this multi-branch collective decision-making mechanism effectively overcomes the randomness and instability that may exist in a single prediction model, ensuring that the final generated target watermark embedding risk mask has higher robustness and fundamentally improving the quality of watermark embedding decisions.

[0147] As another optional embodiment, the present invention also provides another optional, parallel implementation method for determining a target watermark embedding risk mask based on all the watermark embedding risk masks, which constitutes two different technical paths from the CNN-based fusion scheme proposed in the aforementioned embodiments. The embodiments of the present invention employ a strategy based on evaluation and dynamic optimization: the system first standardizes and binarizes each independent risk mask, then calculates the mask dispersion, a key performance indicator, for each binarized mask, and finally selects the binarized mask with the optimal performance indicator as the final decision result. The specific implementation process can be broken down into the following steps:

[0148] First, all the watermark embedding risk masks are sorted by size. Then, based on the obtained watermark embedding risk mask sequence, a binarization operation is performed on each watermark embedding risk mask to obtain the binarized embedding risk mask corresponding to each watermark embedding risk mask.

[0149] The purpose of this step is to convert the continuous risk value vectors with different values ​​from multiple mask prediction network models into a unified binary (0 or 1) mask that can be directly used for evaluation and comparison.

[0150] There are several ways to implement binarization. One option is to set a global fixed threshold, setting risk values ​​above the threshold to 0 (unselected) and those below the threshold to 1 (selected). Another option is to use an elite exclusion strategy based on sorting and proportion. For each independent risk mask vector, the risk values ​​are sorted from largest to smallest, and the frame position with the highest risk at a preset proportion is marked as unselected, while the rest are marked as selected.

[0151] Next, based on the mask dispersion of each of the binarized embedding risk masks, the mask dispersion corresponding to the maximum mask dispersion is determined as the target watermark embedding risk mask.

[0152] Mask dispersion is a custom metric used to quantify the uniformity of watermark embedding locations. Physically, a higher mask dispersion indicates that audio frames marked as unselected are more sparsely and dispersed along the timeline, rather than appearing in concentrated clusters. Since a dispersed embedding pattern significantly improves the robustness of watermarks against continuous data loss attacks (such as random cuts or segment deletions), if unwatermarked frames are too concentrated, a single cut attack could remove most or all of the watermarked frames, leading to watermark extraction failure. Therefore, maximizing mask dispersion is equivalent to selecting the strategy with the best anti-attack performance among all candidate embedding strategies.

[0153] Specifically, this embodiment determines the target watermark embedding risk mask by calculating the mask dispersion score of each binarized embedding risk mask. Then, these mask dispersion scores are compared, and the maximum value is identified. The target watermark embedding risk mask ultimately determined by the system is the binarized embedding risk mask corresponding to this maximum dispersion score. This means that for a specific original audio signal, the system dynamically selects the mask prediction network model that generates the most dispersed embedding pattern as the optimal model for this embedding task.

[0154] This invention addresses the problem of selecting the optimal embedding strategy by introducing mask dispersion as a key performance indicator and dynamically selecting the best mask from multiple candidate masks. Instead of fusing all risk predictions, this method directly aims to improve the robustness of the watermark by selecting the embedding scheme that distributes the watermark information most evenly and dispersedly across the timeline. This significantly enhances the ability of watermarked audio to resist continuous attacks such as cut-out and segment loss.

[0155] Based on the above embodiments, this invention provides a detailed description of a specific and quantifiable calculation method for the key indicator, mask dispersion. Specifically, the mask dispersion of each binarized embedded risk mask is determined based on the following method:

[0156] First, sort the frame indices of all audio frames with a value of 0 in the binarized embedded risk mask by size to obtain the frame index sequence.

[0157] The specific implementation of obtaining the frame index sequence can be as follows: the system traverses the binary embedding risk mask. During the traversal, it checks the value of each element. When an element with a value of 0 is detected, it records the position index of that element in the binary embedding risk mask.

[0158] After traversal, the system sorts all recorded index values ​​in ascending order, ultimately forming an ordered sequence of frame indexes. This sequence of frame indexes is an integer sequence that completely and orderly describes the distribution of all non-selected audio frames on the timeline.

[0159] Then, the differences between all adjacent frame indices in the frame index sequence are summed to obtain the mask dispersion.

[0160] The system further iterates through the acquired ordered frame index sequence, initializing an accumulator variable to 0 during this process. Then, starting from the second element of the frame index sequence and traversing to the end, the system calculates the difference between the current frame index and its previous frame index in each iteration. This difference represents the number of other frames between two adjacent non-selected audio frames, and the system accumulates this difference into the accumulator. After traversing the entire frame index sequence, the final value in the accumulator represents the mask dispersion corresponding to the binarized embedding risk mask.

[0161] The above calculation process will be illustrated below with a specific implementation example.

[0162] Suppose that in the previous steps, the system has generated M binary embedding risk masks based on M different convolutional neural network models. Now, the system needs to calculate the mask dispersion for one of the binary embedding risk masks. .

[0163] For ease of understanding, assume that the audio corresponding to the binary embedded risk mask has a total of 20 audio frames (i.e., f=20), and its corresponding binary vector is: [1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1], where a value of 1 represents the selected state and a value of 0 represents the non-selected state.

[0164] The method for obtaining and sorting the frame index sequence includes: the system traverses the above binary vector and obtains the index of all elements with a value of 0. In this embodiment, assuming the index starts from the 0th position, the obtained index positions are 2, 6, 9, and 14. Since these index positions are naturally ordered when obtained, the resulting frame index sequence is [2, 6, 9, 14].

[0165] Furthermore, the system performs a cumulative summation of adjacent differences, that is, it calculates the mask dispersion of the binarized embedded risk mask according to the following formula. It is 12.

[0166] ;

[0167] in, Indicates the first Group binarization embedding risk mask The first in i indivual Frame index, N It represents the number of index positions of elements with a value of 0.

[0168] The system will repeat the above calculation process for all M binary embedding risk masks to obtain M dispersion scores, and will select the binary embedding risk mask with the highest score as the final target watermark embedding risk mask.

[0169] This invention provides a simple yet physically meaningful quantitative index that transforms the abstract concept of watermark embedding uniformity into a calculable and comparable numerical mask dispersion. This enables the system to objectively and dynamically select the optimal embedding strategy from multiple candidate schemes based on a clear optimization objective, thereby significantly improving the robustness of the watermark against continuous data attacks such as cutting and fragment deletion.

[0170] As an optional embodiment, in response to the step of determining a watermark embedding energy mask based on the energy value of each audio frame in the original audio signal used in the above embodiments, the present invention provides a method to determine a dynamic and adaptive reference threshold by calculating the average energy value of the audio signal itself, and then generating a watermark embedding energy mask by comparing the energy value of each audio frame with this reference threshold.

[0171] Figure 3 This is a flowchart illustrating the watermark embedding energy mask generation method provided by the present invention, as shown below. Figure 3 As shown, the main steps include, but are not limited to, the following:

[0172] Step 211: Determine the average energy value of all audio frames in the original audio signal.

[0173] Step 212: Compare the energy value of each audio frame in the original audio signal with the average energy value frame by frame to determine the watermark embedding energy mask based on the comparison results of all audio frames.

[0174] Wherein, if the comparison result is that the element mapped in the watermark embedding energy mask of an audio frame with an energy value greater than the average energy value is selected; otherwise, the element mapped in the watermark embedding energy mask is not selected.

[0175] Compared to using a fixed, preset energy threshold, this embodiment uses the average energy of the audio signal itself as the benchmark threshold, which has better adaptability. For an audio signal with a generally high volume, its average energy value will naturally be higher; while for an audio signal with a generally low volume, its average energy value will be lower. This adaptive benchmark threshold ensures that subsequent comparisons have a similar filtering effect on audio signals of different volumes.

[0176] In this embodiment, the energy value of each audio frame needs to be calculated first. For example, the energy of any audio frame can be determined by calculating the sum of all amplitude values ​​in its amplitude spectrum characteristic components.

[0177] Calculate the energy values ​​of each of the K audio frames. Then, the average energy value can be obtained by taking the arithmetic mean of all these energy values. :

[0178] .

[0179] Furthermore, the system will iterate through each audio frame and compare the energy value corresponding to each audio frame calculated in step 211 with the calculated average energy value.

[0180] Considering that higher-energy audio frames have a stronger masking effect and are more suitable for embedding watermark information, the discrimination rule in this embodiment is set as follows:

[0181] If the energy value of any audio frame is greater than the average energy value mentioned above, the video frame is considered to have good embedding conditions, and therefore the element corresponding to it in the watermark embedding energy mask is set to the selected state (for example, set to 1).

[0182] If the energy value of any audio frame is less than or equal to the average energy value mentioned above, the video frame is considered to have relatively insufficient energy and a high risk of embedding. Therefore, the element corresponding to it in the watermark embedding energy mask is set to a non-selected state (e.g., a value of 0).

[0183] Assuming the original audio signal is divided into 1000 audio frames, the system first calculates the energy value of each of these 1000 frames, then calculates the arithmetic mean of these energy values ​​to obtain the average energy value. Next, a zero-based vector of length 1000 is created as the initial watermark embedding energy mask. The system iterates from frame 1 to frame 1000. For frame t, if its energy value is greater than the average energy value, the t-th element of the watermark embedding energy mask is set to 1. After the iteration is complete, the resulting vector containing 0s and 1s is the final watermark embedding energy mask.

[0184] The audio watermark embedding method provided by this invention compares the watermark with the average energy value of the audio signal itself, rather than with a fixed energy threshold. This allows the filtering method to automatically adapt to audio content of different loudnesses and maintain a robust filtering effect without manual parameter adjustment, thereby improving the automation and versatility of the entire watermark embedding method.

[0185] Based on the above embodiments, as an optional embodiment, the generation of the frame-level embedding mask based on the watermark embedding power mask and / or the watermark embedding risk mask mentioned in the above embodiments includes the content of generating the frame-level embedding mask by combining the two different features of the watermark embedding power mask and the watermark embedding risk mask, specifically including but not limited to the following steps:

[0186] Binarize each element in the watermark embedding risk mask;

[0187] Perform an element-wise logical OR operation on the watermark embedding energy mask and the binarized watermark embedding risk mask to obtain the frame-level embedding mask.

[0188] First, the watermark embedding risk mask itself is a one-dimensional real number vector with the same length as the number of audio frames, where the value of each element represents the perceptual distortion tolerance of the corresponding audio frame. The system binarizes the watermark embedding risk mask, which consists of continuous values, output by the mask prediction network model, to generate a binary mask consisting of discrete values ​​(such as 0 and 1) for subsequent logical operations.

[0189] As an alternative implementation, a preset threshold (e.g., 0.5) can be determined first. For each element in the watermark embedding risk mask, if its value is greater than the preset threshold, the binarized result is set to 1 (representing selection); otherwise, the binarized result is set to 0 (representing non-selection).

[0190] Another approach is to first sort the values ​​of all elements in the watermark embedding risk mask from largest to smallest (or smallest to largest), and then select a subset of elements to set to 1 according to a preset rule, while setting the rest to 0. For example, a preset number (e.g., t) of the elements with the largest values ​​can be set to 0, and the rest can be set to 1. This method allows for precise control over the proportion of target video frames contributed by the watermark embedding risk mask, providing better controllability.

[0191] After binarization, a preliminary binary mask is obtained, consisting of 0s and 1s and equal in length to the number of audio frames, representing the neural network's perceptual prediction decision. Then, the generated watermark is embedded into the energy mask and subjected to an element-wise logical OR operation.

[0192] The rules for the logical OR operation are as follows: for two elements at the same position in two input vectors, if at least one of them is 1, then the element at that position in the output vector is 1; the output is 0 only when both input elements are 0. The output result after performing the logical OR operation is the frame-level embedding mask.

[0193] Suppose an original audio signal contains 5 audio frames. After energy filtering, the resulting watermark embedding energy mask is [0, 1, 1, 0, 1], indicating that the 2nd, 3rd, and 5th audio frames have sufficient energy. Simultaneously, assume the binarized watermark embedding risk mask is [0, 0, 1, 1, 0], indicating that the mask prediction network model considers the 3rd and 4th video frames perceptually suitable for embedding. The system performs an element-wise logical OR operation on these watermark embedding energy and risk masks to obtain the final frame-level embedding mask [0, 1, 1, 1, 1], which indicates that the 2nd, 3rd, 4th, and 5th audio frames are ultimately selected as the target audio frames.

[0194] The audio watermark embedding method provided by this invention adopts suggestions suitable for embedding watermark information from any dimension by using a frame-level embedding mask obtained by logical OR operation. It clarifies that any audio frame will be selected as the target audio frame as long as it meets one of the two conditions of sufficient energy or perceptual suitability. Under the premise of ensuring security, it can maximize the number of target audio frames that can be selected for embedding watermark information, and provide more opportunities and flexibility for subsequent embedding of sufficiently robust watermark information.

[0195] As an optional embodiment, the present invention provides an implementation method for binarizing each element in a watermark embedded risk mask, specifically including:

[0196] Sort the values ​​of all elements in the watermark embedding risk mask in descending order;

[0197] The position corresponding to the element with the largest preset number of values ​​after sorting is set to a non-selected state in the watermark embedding risk mask after binarization.

[0198] Set the positions corresponding to the remaining elements to the selected state.

[0199] The system first obtains the real-valued vector of the watermark embedding risk mask, and then sorts all its element values ​​in descending order (from largest to smallest). While sorting in descending order, the system records the original frame index of each value in the vector.

[0200] Since the value corresponding to the watermark embedding risk mask can be understood as the embedding risk, the larger the value, the higher the risk of watermark embedding in the corresponding audio frame, and the lower the tolerance. In this embodiment, after sorting in descending order, the elements with the largest preset number of values ​​represent the audio frames that the mask prediction network model considers to have the highest embedding risk and are most likely to produce distortion.

[0201] The preset quantity is a hyperparameter that can be configured according to system requirements. For example, it can be set to 10% of the total number of frames or a fixed value, such as 10. Based on this preset quantity, the system finds the top 10 elements with the highest risk values ​​and obtains their original frame indices. Then, in a newly created binary vector with the same length as the number of audio frames and all elements in a selected state (i.e., value set to 1), the positions corresponding to these high-risk frame indices are set to a non-selected state (i.e., value set to 0). The final binary vector obtained after this processing is the binarized watermark embedding risk mask.

[0202] In a specific scenario, suppose an original audio signal contains 1000 audio frames. The mask prediction network outputs a watermark embedding risk mask of length 1000 (assuming the values ​​are risk values, with larger values ​​indicating less suitability for watermark embedding). The system sets a preset quantity k of 100. First, it sorts these 1000 risk values ​​in descending order and finds the frame indices corresponding to the top 100 highest values. Then, it creates a vector of length 1000 with all elements equal to 1. Finally, it changes the values ​​at the vector positions corresponding to the 100 high-risk frame indices from 1 to 0. The resulting vector, containing 100 zeros and 900 ones, is the binarized result, accurately masking the 10% of the highest-risk audio frames.

[0203] This invention provides a binarization method that can precisely control the selected ratio. Compared with the simple threshold comparison method, it is not affected by the overall distribution of risk mask values ​​and can select a fixed number or ratio of the best or worst audio frames. This controllability makes the watermark embedding behavior more stable and predictable, and facilitates the quantification and optimization of the performance of the entire system.

[0204] Figure 4 This is a schematic diagram of the method for obtaining watermark amplitude spectrum features provided by the present invention, as shown below. Figure 4 As shown in the above embodiments, the method of embedding the watermark information into the feature components corresponding to each target audio frame in the amplitude spectrum feature to obtain the watermarked amplitude spectrum feature may include, but is not limited to:

[0205] Step 301: Concatenate the amplitude spectrum feature and the watermark information to obtain a concatenated feature. The amplitude spectrum feature is a two-dimensional matrix. The watermark information is typically a one-dimensional binary code stream. To enable both to be input into the watermark encoder network model, the watermark information needs to be processed first.

[0206] In one specific embodiment, the original binary watermark bitstream (e.g., a 32-bit code) is first copied frame-by-frame to match its length with the number of frames (time dimension) of the amplitude spectrum feature.

[0207] Then, the processed watermark information is concatenated with the amplitude spectrum feature along the feature dimension, for example, if the dimension of the amplitude spectrum feature is T. F, the dimension of the copied watermark information is T. If W (where W is the number of bits in the watermark code) is used, then the dimension of the concatenated feature is T. (F+W).

[0208] Step 302: Input the splicing features into the watermark encoder network model to obtain the initial watermark modulation weight matrix output by the watermark encoder network model. The initial watermark modulation weight matrix has the same dimension as the amplitude spectrum features.

[0209] In one specific embodiment, the watermark encoder network model is a pre-trained deep neural network whose function is to learn an optimal mapping function, which can intelligently calculate how to modify the amplitude spectrum features based on the input amplitude spectrum features and watermark information.

[0210] The splicing features are input into the watermark encoder network model, which, after a series of complex nonlinear calculations, outputs a two-dimensional matrix with the same dimension as the amplitude spectrum features, namely the initial watermark modulation weight matrix. The initial watermark modulation weight matrix can be understood as a modulation proposal given by the watermark encoder network model for each time-frequency unit of the original audio signal without considering the final frame-level selection, containing all the modification information required to embed the specified watermark.

[0211] Step 303: Use the frame-level embedding mask to mask the initial watermark modulation weight matrix to obtain the final watermark modulation weight matrix.

[0212] After obtaining the frame-level embedding mask, the system uses it to mask the initial watermark modulation weight matrix obtained in the previous steps. Specifically, the masking process involves element-wise multiplication: first, the one-dimensional frame-level embedding mask is broadcast along the frequency dimension, transforming it into a two-dimensional mask matrix with the same dimension as the initial watermark modulation weight matrix. Then, this two-dimensional mask matrix is ​​multiplied element-wise with the initial watermark modulation weight matrix to obtain the final watermark modulation weight matrix. The final watermark modulation weight matrix is ​​characterized by retaining valid watermark embedding information only in the rows corresponding to the target audio frame, while all other rows are zero matrices.

[0213] In rows where the frame-level embedding mask value is 1 (i.e., the target audio frame), the values ​​of the corresponding rows of the initial watermark modulation weight matrix are preserved. In rows where the frame-level embedding mask value is 0 (i.e., non-target audio frames), all values ​​of the corresponding rows of the initial watermark modulation weight matrix are multiplied by 0, thus becoming 0.

[0214] Step 304: Apply the final watermark modulation weight matrix to the feature components corresponding to each target audio frame in the amplitude spectrum feature to obtain the watermarked amplitude spectrum feature.

[0215] In a specific embodiment, it is assumed that the following was obtained: The amplitude spectrum features of the dimension and the initial watermark information are concatenated and fed into the watermark encoder network model to obtain a... The initial watermark modulation weight matrix is ​​obtained. Simultaneously, the system also obtains a frame-level embedding mask of length 1000, assuming that the values ​​at positions 10, 25, 48, ... are 1, and the rest are 0. This frame-level embedding mask is then broadcast as 1000. After obtaining the 512 matrix, it is multiplied element-wise with the initial watermark modulation weight matrix to obtain the final watermark modulation weight matrix. This final watermark modulation weight matrix retains the original weights in rows 10, 25, 48, etc. (up to row 100), while the remaining 900 rows are all zero. Finally, the system adds this final watermark modulation weight matrix element-wise to the amplitude spectrum feature to obtain the watermarked amplitude spectrum feature.

[0216] This invention decouples the two tasks of modulation content generation and embedding position selection, making the entire watermark embedding process more refined and controllable. By generating an intermediate initial watermark modulation weight matrix, the watermark encoder network model can focus on learning the optimal modification mode, while the selection task is completely controlled by the frame-level embedding mask. The resulting final watermark modulation weight matrix accurately executes the intention of selectively embedding watermark information.

[0217] As an optional embodiment, the implementation step mentioned in step 302 of the above embodiments, which involves applying the final watermark modulation weight matrix to the feature components corresponding to each target audio frame in the amplitude spectrum feature to obtain the watermarked amplitude spectrum feature, includes, but is not limited to:

[0218] The elements corresponding to each target audio frame in the final watermark modulation weight matrix are multiplied or added to the feature components corresponding to each target audio frame in the amplitude spectrum feature, respectively, to obtain the watermarked amplitude spectrum feature.

[0219] In this embodiment, the element-wise operations performed on the two identical matrices—the final watermark modulation weight matrix and the amplitude spectrum feature—include both addition and multiplication operations.

[0220] Under the addition operation method, the amplitude spectrum feature M of the watermark is... w It is achieved by combining the amplitude spectrum feature M with the final watermark modulation weight matrix Δ final The result of adding elements one by one:

[0221] M w =M+Δ final .

[0222] Due to Δ final Since the rows corresponding to non-target audio frames are all zero, the addition operation only changes the value of the feature component corresponding to the target audio frame, while the values ​​of other feature components remain unchanged because zero has been added to them. This method can be understood as directly superimposing a perturbation signal composed of watermark information onto the amplitude spectrum feature.

[0223] Under the multiplication operation method, the amplitude spectrum feature M of the watermark is... w It is achieved by combining the amplitude spectrum feature M with the final watermark modulation weight matrix Δ final The result of adding elements one by one:

[0224] M w =M (1+Δ) final ).

[0225] Similarly, due to Δ final In rows corresponding to non-target audio frames, all values ​​are zero. The modulation factor for these rows is 1 + 0 = 1, so the values ​​of these rows remain unchanged after multiplication. Only in rows corresponding to the target audio frame is the modulation factor not equal to 1, thus changing the value of the amplitude spectrum feature. This method can be understood as scaling the amplitude of the original spectrum by a certain proportion based on the watermark information.

[0226] Through the above steps, the embodiments of the present invention clarify the specific path for achieving the fusion of watermark information and carrier features. By limiting the path to element-wise multiplication or addition, the technical operation is clearly defined and the selective embedding of watermark information is ensured. That is, the amplitude spectrum features are effectively modified only at the position of the target audio frame due to the non-zero value of the final watermark modulation weight matrix. These two operation methods are easy to implement and have high computational efficiency, providing clear guidance for the engineering application of the present invention.

[0227] As an optional embodiment, the watermark encoder network model used can be a deep residual network (DRN). The main reason for using a deep residual network model as a watermark encoder is its superior performance and training stability when dealing with complex feature mapping tasks.

[0228] Specifically, the deep residual network model used in this embodiment refers to a deep neural network that utilizes the concepts of residual connections or shortcut connections. Its core structure is a residual block. In a standard residual block, the input signal x not only undergoes nonlinear transformation through several convolutional or fully connected layers to obtain F(x), but also skips multiple layers directly through a shortcut connection, adding the result of the nonlinear transformation to the input signal, and the final output is H(x) = F(x) + x.

[0229] Analysis shows that this invention concretizes the watermark encoder network model into a deep residual network model, which has the following technical advantages:

[0230] On the one hand, watermark embedding is a complex task, requiring the watermark encoder network model to have sufficient depth to learn the mapping from "audio content + watermark information" to the optimal spectral modification scheme in the amplitude spectrum feature modification scheme. Traditional deep networks are prone to gradient vanishing or gradient exploding problems as the number of layers increases, leading to training difficulties. Deep residual networks effectively alleviate these problems through residual connections, making it possible to build and train very deep networks (e.g., containing dozens or even hundreds of convolutional layers), thereby enabling the extraction of higher-level, more abstract features and the generation of finer modulation weights.

[0231] On the other hand, residual learning essentially allows the model to learn the residual or increment between the output and the input, i.e., F(x) = H(x) - x. This is a natural fit for the task of audio watermark embedding. In a specific scenario, the final watermarked amplitude spectrum can be viewed as the original amplitude spectrum plus a perturbation caused by the watermark information. Allowing the network to directly learn this "perturbation" (i.e., the watermark modulation weight matrix) is much easier than having it learn the entire watermarked amplitude spectrum from scratch. The Watermark Encoder Network (DRN) only needs to focus on calculating this optimal increment, which greatly reduces the complexity of the learning task.

[0232] A pre-trained, high-performance network model is the foundation for achieving the desired watermark embedding effect in this invention. Therefore, this invention provides an end-to-end model training process.

[0233] Figure 5 This is a flowchart illustrating the model training method provided by the present invention, as shown below. Figure 5 As shown, the mask prediction network model and the watermark encoder network model are trained through the following steps:

[0234] Step 41: Determine each audio signal sample in the training set, and the initial watermark information for watermark embedding operation on each audio signal sample.

[0235] As a preparation stage for model training, in order to enable the network model to learn the rules of watermark embedding, the system first prepares a large number of diverse audio signal samples, which will cover various types of audio content, such as music, speech, and ambient sounds.

[0236] For each audio signal sample in the training set, an initial watermark is randomly generated or selected from a pre-defined set. This initial watermark is the label that is expected to be embedded and correctly extracted in this training iteration.

[0237] Step 42: Generate a watermarked audio signal sample corresponding to each audio signal sample based on the initial watermark information, and determine the frame-level embedding mask sample corresponding to each audio signal sample.

[0238] As part of the forward propagation process during actual training, for each audio signal sample in the training set and its corresponding initial watermark information, the system will completely execute the watermark embedding process provided in any of the above embodiments:

[0239] (1) Perform time-frequency transformation on the audio signal sample to obtain amplitude spectrum features.

[0240] (2) Generate a frame-level embedded mask sample based on the content features of the audio signal sample (such as energy and the current output of the mask prediction network model).

[0241] (3) Based on the output of the watermark encoder network model and combined with the frame-level embedding mask sample, the initial watermark information is embedded to obtain the watermark amplitude spectrum feature.

[0242] (4) Perform inverse time-frequency transformation to reconstruct the watermarked audio signal sample.

[0243] In this step, all network models, including the mask prediction network and the watermark encoder network, participate in the computation and jointly generate frame-level embedded mask samples and watermarked audio signal samples.

[0244] Step 43: Input the watermarked audio signal sample into the watermark extraction network model to obtain the watermark information output by the watermark extraction network model.

[0245] To construct a complete and trainable closed-loop system, this invention introduces a watermark extraction network model. The structure of the watermark extraction network model can be similar to or different from that of the encoder network. Its function is to receive an audio signal containing watermark information and attempt to recover the original embedded watermark information from it. By inputting the generated watermarked audio signal sample into the watermark extraction network model, the processed and output watermark information can be obtained.

[0246] Step 44: Calculate the total training loss based on the audio signal sample, the watermarked audio signal sample, the initial watermark information, the identified watermark information, and the frame-level embedding mask sample.

[0247] The system utilizes the various data generated or used in the previous steps to calculate one or more loss functions, and then sums them with weights to obtain the total training loss. The total training loss comprehensively measures the current model's performance across multiple dimensions, including imperceptibility, robustness and correctness, and the appropriateness of embedding positions. A lower total loss value indicates better overall performance of the current model.

[0248] Step 45: Update the trainable parameters of the mask prediction network model, the watermark encoder network model, and the watermark extraction network model based on the total training loss.

[0249] The system can use the backpropagation algorithm to calculate the gradient of the total training loss with respect to each trainable parameter of all neural networks (mask prediction network, watermark encoder network, watermark extraction network) in the system.

[0250] Then, using an optimizer algorithm (such as Adam, SGD, etc.), a small adjustment is made to all these parameters based on the calculated gradient. The adjustment is made in a direction that reduces the "total training loss".

[0251] By iteratively executing steps 41 to 45 over the entire training set, the parameters of all neural networks will be continuously optimized, making their overall performance better and better until the total loss converges to a low level or reaches the preset number of training rounds. At this point, the training is complete, and a well-trained network model that can achieve the expected function can be obtained.

[0252] This invention provides a systematic, end-to-end neural network model training method. By constructing the watermark embedding and extraction process into a unified, differentiable framework and driving it through a carefully designed total loss function, multiple neural networks in the system can co-evolve. This training method enables the model not only to learn how to embed and extract watermarks, but also to achieve an optimal balance among multiple mutually constraining objectives such as imperceptibility and robustness, providing a fundamental guarantee for the superior performance of the method of this invention.

[0253] As an optional embodiment, the total training loss includes at least the adjacent mask frame index loss, which is calculated based on the following steps:

[0254] Identify the frame index corresponding to the element with a value of 1 in the frame-level embedding mask sample to obtain the frame index set;

[0255] Calculate the interval distance between every two temporally adjacent frame indices in the frame index set, and when the interval distance is less than a preset distance threshold, determine a loss value based on the difference between the interval distance and the preset distance threshold;

[0256] Then, all the loss values ​​are summed to obtain the adjacent mask frame index loss.

[0257] The Adjacent Masked-Frame Index Loss (AMFI) adds an important structural constraint to the training of the network model, which can effectively prevent watermark information from being overly concentrated in time.

[0258] The adjacent mask frame index loss is calculated based on the following steps:

[0259] First, identify the frame index corresponding to the element with a value of 1 in the frame-level embedding mask sample to obtain the frame index set;

[0260] During the training forward propagation, the system generates a frame-level embedding mask sample for the current audio signal sample. Then, it iterates through this frame-level embedding mask sample, extracting the positions (i.e., indices) of all elements with a value of 1, forming a frame index set. Each integer in this frame index set represents the position of a frame selected as the target audio frame on the timeline.

[0261] Then, the interval distance between any two temporally adjacent frame indices in the frame index set is calculated, and when the interval distance is less than a preset distance threshold, a loss value is determined based on the difference between the interval distance and the preset distance threshold. This step is the core of calculating the AMFI loss. The system processes the obtained frame index set, which has been sorted in ascending order.

[0262] Assuming the frame index set is {10, 12, 50}, then the adjacent index pairs are (10, 12) and (12, 50), and their corresponding interval distances are 12-10=2 and 50-12=38, respectively.

[0263] Then, each calculated interval distance is compared with a preset distance threshold τ. This preset distance threshold τ represents the minimum safe distance between adjacent embedded frames that the system expects. For example, τ can be set to 1.5 or 2, and the following information can be obtained:

[0264] When the interval distance is less than the preset distance threshold τ, it means that the two target audio frames are too close in time, which is a behavior that needs to be avoided. In this case, a positive loss value is determined based on the difference between the interval distance and the preset distance threshold. For example, if the interval distance is 1 and τ is 1.5, then the loss value is 0.5.

[0265] Furthermore, if the interval distance is greater than or equal to τ, it is considered a reasonable interval, and its corresponding loss value is 0, so no penalty is imposed.

[0266] Finally, all the loss values ​​are summed to obtain the adjacent mask frame index loss.

[0267] The system will sum all the positive loss values ​​calculated for all closely adjacent frames. This final sum is the adjacent mask frame index loss L of the current training sample. AMFI .

[0268] This loss value L AMFI This will then be weighted and included in the total training loss, and during backpropagation, a non-zero L... AMFI This generates gradients, which drive the mask prediction network model to adjust its parameters, making it more inclined to generate frame-level embedding masks with larger intervals in subsequent iterations, thereby reducing L. AMFI The value of .

[0269] This invention introduces an explicit constraint on the spatial distribution of embedding locations during the training process of the network model. The resulting adjacent mask frame index loss effectively enables the network model to learn a strategy of dispersively embedding watermark information along the time axis. This dispersed embedding mode greatly improves the robustness of the watermarking system against time axis attacks (such as content clipping, packet loss, local noise, etc.). Even if part of the audio is corrupted, because the watermark information is distributed in multiple discontinuous locations, the decoder still has a high probability of recovering the complete watermark from the remaining part, thus significantly enhancing practicality and reliability.

[0270] As an optional embodiment, the total training loss includes at least the audio distortion loss, which is calculated based on the following steps:

[0271] Step 1101: Obtain the original amplitude spectrum features corresponding to the audio signal sample;

[0272] This step obtains the amplitude spectrum feature matrix, which is obtained from the original audio signal sample after time-frequency transformation and has not been modified in any way.

[0273] Step 1102: Obtain the watermarked amplitude spectrum features corresponding to the watermarked audio signal sample;

[0274] The matrix obtained here is the amplitude spectrum feature matrix, which has been partially modified and generated after the watermark encoder network model and frame-level embedding mask work together in the watermark information embedding process.

[0275] Step 1103: Calculate the difference between the original amplitude spectrum features and the watermarked amplitude spectrum features to obtain the audio distortion loss.

[0276] The system will compare the original amplitude spectrum features with identical dimensions obtained in the previous two steps with the watermarked amplitude spectrum features and calculate the distance or difference between them.

[0277] There are several ways to calculate the difference between the two matrices. In a preferred embodiment, the mean squared error (MSE) can be used, or other metrics that measure the difference between the two matrices, such as the mean absolute error (MAE), can be used. Specifically, the system calculates the square of the difference between all corresponding elements in both matrices and then averages all these squared differences.

[0278] The calculated audio distortion loss value is weighted and included in the total training loss. During backpropagation, a larger audio distortion loss will generate a corresponding gradient, penalizing network parameters that cause excessive spectral changes, thereby forcing the network model to learn a more moderate embedding strategy.

[0279] This invention ensures that by continuously minimizing audio distortion loss, the final trained model modifies the original audio spectrum as little as possible when embedding the watermark, thus fundamentally guaranteeing the imperceptibility of the watermark and making the watermarked audio sound almost indistinguishable from the original audio.

[0280] As an optional implementation, the total training loss may also include a watermark extraction loss, specifically calculated based on the following steps:

[0281] The difference between the initial watermark information and the identified watermark information is calculated to obtain the watermark extraction loss.

[0282] The initial watermark information is the original watermark bitstream that was determined for the current training sample in the steps of the aforementioned embodiment and is used as a reference.

[0283] The watermark information is the watermark bitstream that is recovered after the watermark extraction network model processes the watermark-containing audio signal sample.

[0284] The system compares the two (usually binary) bitstreams or their vector representations and calculates the differences between them. The method of calculating the differences depends on the representation of the watermark information.

[0285] If the initial watermark information and the identified watermark information are binary bitstreams, and the output of the watermark extraction network model is the probability value (between 0 and 1) for each bit, then the binary cross-entropy (BCE) loss is preferred. The BCE loss can effectively measure the difference between the predicted probability and the true label (0 or 1).

[0286] If both the initial watermark information and the identified watermark information are represented in vector form, then cosine similarity loss or mean square error can be used to calculate the difference between the two watermark information.

[0287] The calculated scalar value, i.e., the watermark extraction loss, is weighted and included in the total training loss. During backpropagation, a larger watermark extraction loss generates a strong gradient signal, which penalizes both the encoder (for failing to generate sufficiently robust embeddings) and the decoder (for failing to correctly recognize the embedded signals), forcing them to co-evolve to form a more efficient encoding and decoding scheme.

[0288] This invention, by continuously minimizing the watermark extraction loss, ensures that the watermark signal embedded in the final trained model has sufficient discriminative power and robustness. This enables the watermark extraction network to accurately recover the original embedded information from complex, even distorted, audio signals. This directly guarantees the effectiveness and reliability of the method in practical applications.

[0289] Figure 6 This is the second flowchart illustrating the audio watermark embedding method provided by the present invention. To more fully reveal the present invention, the following describes the process in conjunction with... Figure 6 This invention provides a detailed description of a specific embodiment of the audio watermark embedding method.

[0290] First, the system performs time-frequency transformation on the original audio signal, such as using Short Time Fourier Transform (STFT) technology to convert the one-dimensional time-domain signal into a time-frequency domain signal, obtaining amplitude spectrum features and phase spectrum features respectively. Among them, the amplitude spectrum features describe the energy distribution of the original audio signal in time and frequency, while the phase spectrum features retain the phase information of the original audio signal for subsequent accurate reconstruction of the time-domain signal.

[0291] At the same time, a baseline watermark information can be randomly selected from the watermark codebook library for encoding. This baseline watermark information is encoded as a fixed-length binary sequence, representing the original watermark information to be embedded.

[0292] The reference watermark information is encoded and copied at the frame level, that is, the encoding is repeatedly copied multiple times until the length matches the number of audio frames, thereby forming a complete watermark information corresponding to the amplitude spectrum time axis, ensuring that the watermark information and amplitude spectrum features maintain a one-to-one correspondence.

[0293] Furthermore, based on the aforementioned amplitude spectrum features and watermark information, these are fed into the watermark encoder network model. Through deep neural network processing, an initial watermark modulation weight matrix is ​​learned, representing the modulation of the amplitude spectrum features. This initial watermark modulation weight matrix has the same dimension as the amplitude spectrum features. The initial watermark modulation weight matrix is ​​a modulation proposal jointly calculated by the watermark encoder network based on the watermark information and the original audio signal content, indicating the amplitude adjustment to be applied at each time-frequency unit.

[0294] Meanwhile, the amplitude spectrum features are fed into two parallel processing modules to generate the following watermark embedding energy mask and watermark embedding risk mask.

[0295] The watermark embedding energy mask utilizes an energy comparison module to compare the energy value of each audio frame with the overall average energy. Audio frames with energy values ​​above the average are selected for their corresponding mask elements; otherwise, they are not selected. The watermark embedding energy mask ensures that watermark embedding occurs in audio frames with sufficient signal energy, reducing unpleasant auditory perception caused by embedding.

[0296] The watermark embedding risk mask is obtained by a mask prediction network model. The mask prediction network model estimates the perceptual distortion tolerance of each audio frame in the context of amplitude spectrum features. After processing by the activation function, the risk prediction result (the value range is generally 0~1) is obtained and then the watermark embedding risk mask is generated.

[0297] The watermark embedding risk mask is binarized, and the secure embedding area is preserved according to preset rules (e.g., setting a fixed number of elements with the largest value to unselected), thus forming the binarized watermark embedding risk mask.

[0298] Finally, the watermark embedding energy mask and the binarized watermark embedding risk mask are fused through an element-wise logical OR operation to form the final frame-level embedding mask. This mask is then directly applied to the initial watermark modulation weight matrix to filter out the modulation weights of audio frames that can truly be embedded with the watermark. The final watermark modulation weight matrix after frame-level embedding mask filtering contains only the components indicated by the frame-level embedding mask, with the remaining components having zero weight.

[0299] After multiplying or adding the corresponding elements of the final watermark modulation weight matrix and the amplitude spectrum feature to form the watermarked amplitude spectrum feature, the phase spectrum feature generated in the previous step is called to ensure accurate and distortion-free signal reconstruction. Combining the watermarked amplitude spectrum feature and the phase spectrum feature can form a complete complex spectrum matrix.

[0300] By performing an inverse short-time Fourier transform (ISTFT) on the complex spectral matrix, a one-dimensional watermarked audio signal can be reconstructed, which can be used for digital audio playback or subsequent processing.

[0301] As an optional implementation, to improve the robustness of the watermark, various attack simulations can be performed on the watermarked audio signal samples during the training phase, including but not limited to: adding Gaussian white noise, performing lossy compression, random frame cutting, or resampling. Attack simulations aim to simulate signal damage and tampering scenarios that may be encountered in actual transmission, thereby improving the adaptability of the trained model in real-world environments.

[0302] The audio watermark embedding method provided by this invention combines content-adaptive mask generation, intelligent watermark modulation weight prediction and selective embedding, frame-level multiplexing of watermark information, and accurate audio signal reconstruction. In particular, by fusing a watermark embedding energy mask with a binarized watermark embedding risk mask, it achieves intelligent selection of the watermark embedding position, effectively balancing imperceptibility and robustness. Combined with attack simulation, it enhances the system's resistance to variable noise and damage in real-world environments, ensuring reliable concealment and stable extraction of watermark information.

[0303] Figure 7 This is a schematic diagram of the audio watermark embedding device provided by the present invention, as shown below. Figure 7 As shown, it mainly includes:

[0304] The time-frequency conversion unit 71 performs time-frequency conversion on the original audio signal to obtain the amplitude spectrum characteristics of multiple audio frames ordered by time.

[0305] The energy mask determination unit 72 determines a watermark embedded energy mask based on the energy value of each audio frame in the original audio signal.

[0306] The risk mask acquisition unit 73 inputs the amplitude spectrum features into a model combination composed of multiple mask prediction network models to obtain multiple different watermark embedding risk masks output by the model combination.

[0307] The target mask determination unit 74 determines a target watermark embedding risk mask based on all the watermark embedding risk masks.

[0308] The frame-level mask generation unit 75 generates a frame-level embedding mask based on the watermark embedding energy mask and the target watermark embedding risk mask. The frame-level embedding mask is used to indicate that at least a portion of the target audio frames among all the audio frames are selected for embedding watermark information.

[0309] The watermark embedding unit 76 embeds the watermark information into the feature components corresponding to each target audio frame in the amplitude spectrum feature to obtain the watermarked amplitude spectrum feature.

[0310] The inverse time-frequency conversion unit 77 performs inverse time-frequency conversion on the watermarked amplitude spectrum features to obtain the watermarked audio signal.

[0311] It should be noted that the audio watermark embedding device provided by the present invention can execute the audio watermark embedding method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0312] The audio watermark embedding device provided by this invention adopts watermark embedding technology based on adaptive frame-level mask. It determines the frame-level embedding mask by performing content analysis on the amplitude spectrum features of the original audio signal, and selects the feature components of a certain proportion of audio frames based on the frame-level embedding mask to embed watermark information. This achieves high imperceptibility and robustness after embedding watermark information, and can meet the high sensitivity of the human auditory system to audio quality.

[0313] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communications bus 840. The processor 810 can call logic instructions in the memory 830 to execute an audio watermark embedding method, which includes: performing a time-frequency transformation on the original audio signal to obtain amplitude spectrum features of multiple audio frames ordered by time; determining a watermark embedding energy mask based on the energy value of each audio frame in the original audio signal; inputting the amplitude spectrum features into a model combination composed of multiple mask prediction network models to obtain multiple different watermark embedding risk masks output by the model combination; determining a target watermark embedding risk mask based on all the watermark embedding risk masks; generating a frame-level embedding mask based on the watermark embedding energy mask and the target watermark embedding risk mask, wherein the frame-level embedding mask is used to indicate that at least some target audio frames among all the audio frames are selected for embedding watermark information; embedding the watermark information into the feature components corresponding to each target audio frame in the amplitude spectrum features to obtain watermarked amplitude spectrum features; and performing an inverse time-frequency transformation on the watermarked amplitude spectrum features to obtain a watermarked audio signal.

[0314] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0315] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the audio watermark embedding method provided in the above embodiments, the method comprising: performing time-frequency transformation on an original audio signal to obtain amplitude spectrum features of multiple time-ordered audio frames; determining a watermark embedding energy mask based on the energy value of each audio frame in the original audio signal; and inputting the amplitude spectrum features into a module composed of multiple mask prediction network models. The model is combined to obtain multiple different watermark embedding risk masks output by the model combination; a target watermark embedding risk mask is determined based on all the watermark embedding risk masks; a frame-level embedding mask is generated based on the watermark embedding energy mask and the target watermark embedding risk mask, the frame-level embedding mask being used to indicate that at least some of the target audio frames among all the audio frames are selected for embedding watermark information; the watermark information is embedded into the feature components corresponding to each of the target audio frames in the amplitude spectrum feature to obtain the watermarked amplitude spectrum feature; the watermarked amplitude spectrum feature is subjected to inverse time-frequency transformation to obtain the watermarked audio signal.

[0316] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the audio watermark embedding method provided in the above embodiments. The method includes: performing a time-frequency transformation on an original audio signal to obtain amplitude spectrum features of multiple time-ordered audio frames; determining a watermark embedding energy mask based on the energy value of each audio frame in the original audio signal; inputting the amplitude spectrum features into a model combination composed of multiple mask prediction network models to obtain multiple different watermark embedding risk masks output by the model combination; determining a target watermark embedding risk mask based on all the watermark embedding risk masks; generating a frame-level embedding mask based on the watermark embedding energy mask and the target watermark embedding risk mask, the frame-level embedding mask indicating that at least some target audio frames among all the audio frames are selected for embedding watermark information; embedding the watermark information into the feature components corresponding to each target audio frame in the amplitude spectrum features to obtain a watermarked amplitude spectrum feature; and performing an inverse time-frequency transformation on the watermarked amplitude spectrum feature to obtain a watermarked audio signal.

[0317] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0318] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0319] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An audio watermark embedding method characterized by, The method comprises: performing time-frequency transformation on an original audio signal to obtain amplitude spectrum features of a plurality of time-ordered audio frames; determining a watermark embedding energy mask according to energy values of each audio frame in the original audio signal; inputting the amplitude spectrum features into a model combination composed of a plurality of mask prediction network models to obtain a plurality of different watermark embedding risk masks output by the model combination; determining a target watermark embedding risk mask according to all the watermark embedding risk masks; generating a frame-level embedding mask based on the watermark embedding energy mask and the target watermark embedding risk mask, the frame-level embedding mask being used to indicate that at least part of the target audio frames in all the audio frames are selected for embedding watermark information; embedding the watermark information into feature components corresponding to each of the target audio frames in the amplitude spectrum features to obtain watermark-containing amplitude spectrum features; performing inverse time-frequency transformation on the watermark-containing amplitude spectrum features to obtain a watermark-containing audio signal.

2. The audio watermark embedding method of claim 1, wherein, The method further comprises: splicing all the watermark embedding risk masks to form a first mask feature map; inputting the first mask feature map into a plurality of convolutional neural network models which are the same in structure but different in randomly initialized parameters to obtain second mask feature maps output by each of the convolutional neural network models; performing matrix multiplication on at least two of all the second mask feature maps and then performing normalization to obtain a third mask feature map; performing matrix multiplication on the third mask feature map and any fourth mask feature map to obtain a global mask feature map, the any fourth mask feature map being any one of all the second mask feature maps except the two second mask feature maps used to generate the third mask feature map; fusing a plurality of risk values of each watermark embedding risk mask at each same feature position into one risk value; performing binarization processing on the risk value at each feature position of the watermark embedding risk mask to generate the target watermark embedding risk mask.

3. The audio watermark embedding method of claim 2, wherein, The method further comprises: performing descending arrangement on risk values of all feature positions in the watermark embedding risk mask; setting a preset number of feature positions with the largest arranged risk values to a non-selected state in the binarized watermark embedding risk mask; setting positions corresponding to remaining elements to a selected state.

4. The audio watermark embedding method of claim 1, wherein, The method further comprises: sorting all the watermark embedding risk masks according to sizes to perform binarization operation on each watermark embedding risk mask according to a sequence of obtained watermark embedding risk masks to obtain a binarized embedding risk mask corresponding to each watermark embedding risk mask; determining a mask dispersion degree corresponding to a maximum mask dispersion degree as the target watermark embedding risk mask according to mask dispersion degrees of each binarized embedding risk mask.

5. The audio watermark embedding method of claim 4, wherein, A mask dispersion degree of each of the binarized embedding risk mask is determined in the following manner: frame indexes of all audio frames with a value of 0 in the binarized embedding risk mask are sorted by size to obtain a frame index sequence; a difference value of all adjacent frame indexes in the frame index sequence is accumulated to obtain the mask dispersion degree.

6. The audio watermark embedding method of claim 1, wherein, The watermark embedding energy mask is determined according to an energy value of each audio frame in the original audio signal, including: an average energy value of all audio frames in the original audio signal is determined; energy values of each audio frame in the original audio signal are compared with the average energy value frame by frame to determine the watermark embedding energy mask according to comparison results of all audio frames; the comparison result is that an element mapped in the watermark embedding energy mask of an audio frame with an energy value greater than the average energy value is in a selected state; otherwise, the element mapped in the watermark embedding energy mask is in a non-selected state.

7. The audio watermark embedding method of claim 1, wherein, The frame-level embedding mask is generated based on the watermark embedding energy mask and the watermark embedding risk mask, including: an element-wise logical OR operation is performed on the watermark embedding energy mask and the watermark embedding risk mask to obtain the frame-level embedding mask.

8. The audio watermark embedding method of claim 1, wherein, The watermark information is embedded into feature components corresponding to each of the target audio frames in the amplitude spectrum feature to obtain a watermark-containing amplitude spectrum feature, including: a spliced feature is obtained by splicing the amplitude spectrum feature and the watermark information; the spliced feature is input into a watermark encoder network model to obtain an initial watermark modulation weight matrix output by the watermark encoder network model, the initial watermark modulation weight matrix being the same in dimension as the amplitude spectrum feature; the initial watermark modulation weight matrix is processed by the frame-level embedding mask to obtain a final watermark modulation weight matrix; the final watermark modulation weight matrix is applied to the feature components corresponding to each of the target audio frames in the amplitude spectrum feature to obtain the watermark-containing amplitude spectrum feature.

9. The audio watermark embedding method of claim 8, wherein, The final watermark modulation weight matrix is applied to the feature components corresponding to each of the target audio frames in the amplitude spectrum feature to obtain the watermark-containing amplitude spectrum feature, including: element-wise multiplication or addition is performed between elements corresponding to each of the target audio frames in the final watermark modulation weight matrix and feature components corresponding to each of the target audio frames in the amplitude spectrum feature to obtain the watermark-containing amplitude spectrum feature.

10. The audio watermark embedding method of claim 8, wherein, The watermark encoder network model is a deep residual network model.

11. The audio watermark embedding method of any of claims 8-10, wherein, The mask prediction network model and the watermark encoder network model are trained in the following manner: each audio signal sample in a training set and initial watermark information for performing watermark embedding operation on each audio signal sample are determined; a watermark-containing audio signal sample corresponding to each audio signal sample is generated according to the initial watermark information, and a frame-level embedding mask sample corresponding to each audio signal sample is determined; the watermark-containing audio signal sample is input into a watermark extraction network model to obtain identified watermark information output by the watermark extraction network model; According to the audio signal sample, the watermark-containing audio signal sample, the initial watermark information, the identified watermark information, and the frame-level embedded mask sample, a total training loss is calculated; Based on the total training loss, trainable parameters of the mask prediction network model, the watermark encoder network model, and the watermark extraction network model are updated.

12. The audio watermark embedding method of claim 11, wherein, The total training loss at least includes an adjacent mask frame index loss, and the adjacent mask frame index loss is calculated based on the following steps: Identifying frame indexes corresponding to elements with a value of 1 in the frame-level embedded mask sample to obtain a frame index set; Calculating interval distances between every two adjacent frame indexes in the frame index set in time, and when the interval distance is less than a preset distance threshold, determining a loss value according to a difference between the interval distance and the preset distance threshold; Accumulating all the loss values to obtain the adjacent mask frame index loss.

13. The audio watermark embedding method of claim 11, wherein, The total training loss at least includes an audio distortion loss, and the audio distortion loss is calculated based on the following steps: Obtaining original amplitude spectrum features corresponding to the audio signal sample; Obtaining watermark-containing amplitude spectrum features corresponding to the watermark-containing audio signal sample; Calculating a difference between the original amplitude spectrum features and the watermark-containing amplitude spectrum features to obtain the audio distortion loss.

14. The audio watermark embedding method of claim 11, wherein, The total training loss at least includes a watermark extraction loss, and the watermark extraction loss is calculated based on the following steps: Calculating a difference between the initial watermark information and the identified watermark information to obtain the watermark extraction loss.

15. The audio watermark embedding method of claim 11, wherein, Before the watermark-containing audio signal sample is input into the watermark extraction network model, the method further includes: Applying at least one attack simulation to the watermark-containing audio signal sample; The attack simulation includes at least one of adding Gaussian white noise, performing lossy compression, randomly cutting frames, or resampling.

16. The audio watermark embedding method of claim 1, wherein, The inverse time-frequency transformation on the watermark-containing amplitude spectrum features to obtain a watermark-containing audio signal includes: Obtaining phase spectrum features synchronously generated when performing time-frequency transformation on the original audio signal; Combining the watermark-containing amplitude spectrum features and the phase spectrum features, and performing inverse time-frequency transformation on the combined result to obtain the watermark-containing audio signal.

17. The audio watermark embedding method of claim 1, wherein, The watermark information is determined based on the following steps: Randomly selecting a reference watermark information code from a watermark codebook library; Performing frame-level replication on the reference watermark information code to generate watermark information matching the number of audio frames.

18. An audio watermark embedding apparatus characterized by comprising: The method includes: A time-frequency conversion unit performs time-frequency transformation on an original audio signal to obtain amplitude spectrum features of multiple audio frames sorted by time; An energy mask determination unit determines a watermark embedding energy mask according to energy values of each audio frame in the original audio signal; A risk mask obtaining unit inputs the amplitude spectrum features into a model combination composed of multiple mask prediction network models to obtain multiple different watermark embedding risk masks output by the model combination; A target mask determination unit determines a target watermark embedding risk mask according to all the watermark embedding risk masks. a frame-level mask generating unit configured to generate a frame-level embedding mask based on the watermark embedding energy mask and the target watermark embedding risk mask, the frame-level embedding mask being used to indicate that at least part of the target audio frames in all the audio frames are selected for embedding watermark information; a watermark embedding unit configured to embed the watermark information into feature components corresponding to the target audio frames in the magnitude spectrum feature, to obtain a watermark-containing magnitude spectrum feature; a time-frequency inverse transforming unit configured to perform time-frequency inverse transform on the watermark-containing magnitude spectrum feature, to obtain a watermark-containing audio signal.

19. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor executes the computer program to implement the audio watermark embedding method in any one of claims 1 to 17. 20.A non-transitory computer-readable storage medium having stored thereon a computer program. The computer program is executed by the processor to implement the audio watermark embedding method in any one of claims 1 to 17.

21. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the audio watermark embedding method in any one of claims 1 to 17.

Images