Audio watermark generation method, device, equipment and computer storage medium

Abstract

The present application proposes an audio watermark generation method, an audio watermark generation device, an audio watermark generation equipment, and a computer storage medium. The audio watermark generation method includes: extracting a first latent vector representation of the target audio data; inputting the target audio data into the audio watermark embedding network of the audio watermark generation network to add watermark data to obtain watermarked audio data; extracting a second latent vector representation of the watermarked audio data; extracting a third latent vector representation of other audio data; constructing a triplet loss using the first latent vector representation, the second latent vector representation, and the third latent vector representation; and updating the audio watermark generation network using the triplet loss, wherein the updated audio watermark embedding network is used to generate an audio watermark. Through the above-mentioned audio watermark generation method, the audio features before and after the watermark is added are aligned in the latent space to update the network parameters of the audio watermark embedding network, thereby improving the audio watermark generation effect.

Description

Technical Field

[0001] The present application relates to the field of digital watermark technology, and in particular to an audio watermark generation method, an audio watermark generation device, an audio watermark generation equipment, and a computer storage medium. Background Art

[0002] This neural network-based audio watermark embedding and extraction method first performs a short-time Fourier transform (STFT) on the original audio signal to separate the amplitude and phase spectrum features. By concatenating the watermark information encoding and the amplitude spectrum features in the feature dimension, a joint feature tensor is constructed as the input to the convolutional neural network. This method then learns the watermark embedding weight matrix corresponding to each frequency point in the amplitude spectrum. This weight matrix is ​​then element-wise multiplied by the original amplitude spectrum and combined with the original phase spectrum to reconstruct the watermarked audio using an inverse short-time Fourier transform (iSTFT).

[0003] Existing watermark embedding networks based on neural networks only rely on calculating the mean square error between the STFT features of the original audio and the watermarked audio to update the network parameters. Summary of the Invention

[0004] In order to solve the above technical problems, the present application proposes an audio watermark generation method, an audio watermark generation device, an audio watermark generation equipment and a computer storage medium.

[0005] To solve the above technical problems, the present application proposes an audio watermark generation method, which includes:

[0006] Acquire a current batch of audio data sets, wherein the audio data sets include target audio data and other audio data;

[0007] extracting a first latent vector representation of the target audio data;

[0008] Inputting the target audio data into the audio watermark embedding network of the audio watermark generation network to add watermark data to obtain watermarked audio data;

[0009] extracting a second latent vector representation of the watermarked audio data;

[0010] extracting a third latent vector representation of the other audio data;

[0011] constructing a triplet loss using the first latent vector representation, the second latent vector representation, and the third latent vector representation;

[0012] The audio watermark generation network is updated using the triplet loss, wherein the updated audio watermark embedding network is used to generate an audio watermark.

[0013] The audio watermark generation method further includes:

[0014] Extracting a first short-time Fourier transform feature of the target audio data;

[0015] Extracting a second short-time Fourier transform feature of the watermarked audio data;

[0016] constructing an error loss using the first short-time Fourier transform feature and the second short-time Fourier transform feature;

[0017] The updating of the audio watermark generation network by using the triplet loss includes:

[0018] The audio watermark generation network is updated using the triplet loss and the error loss.

[0019] The step of inputting the target audio data into an audio watermark embedding network of an audio watermark generation network to add watermark data to obtain watermarked audio data comprises:

[0020] Inputting the target audio data into the audio watermark embedding network and adding a plurality of different watermark data respectively to obtain a plurality of watermarked audio data;

[0021] The constructing a triplet loss using the first latent vector representation, the second latent vector representation, and the third latent vector representation includes:

[0022] constructing an intermediate triplet loss using the second latent vector representation of each watermarked audio data, the first latent vector representation, and the third latent vector representation;

[0023] The triplet loss is constructed by utilizing the intermediate triplet losses corresponding to the plurality of watermarked audio data.

[0024] The step of extracting a third latent vector representation of the other audio data includes:

[0025] Extracting a third latent vector representation of the other audio data before the watermark data is added, or extracting a third latent vector representation of the other audio data after the watermark data is added.

[0026] The audio watermark generation method further includes:

[0027] Inputting the watermark data into the audio watermark extraction network of the audio watermark generation network to obtain a first watermark information code;

[0028] Selecting a simulation attack method for the current batch of audio data sets;

[0029] Performing attack simulation on the watermarked audio data according to the simulation attack mode to generate attack audio data;

[0030] Inputting the attack audio data into the audio watermark extraction network to extract the second watermark information code;

[0031] Constructing a difference loss using the first watermark information encoding and the second watermark information encoding;

[0032] The updating of the audio watermark generation network by using the triplet loss includes:

[0033] Utilizing the triplet loss and the difference loss to form a global loss;

[0034] The audio watermark generation network is updated using the global loss.

[0035] Among them, the simulation attack method of the audio data set of other batches is different from the simulation attack method of the audio data set of the current batch.

[0036] The step of inputting the target audio data into an audio watermark embedding network of an audio watermark generation network to add watermark data to obtain watermarked audio data comprises:

[0037] Inputting the target audio data into the audio watermark embedding network, and extracting the amplitude spectrum features and phase spectrum features of the target audio data;

[0038] Select one of the watermark information codes from the watermark codebook library, perform frame-level replication according to the number of frames of the amplitude spectrum feature, and obtain watermark data;

[0039] Performing frame-level splicing on the amplitude spectrum feature and the watermark data to obtain the watermark amplitude spectrum feature;

[0040] Extracting a watermark modulation weight matrix based on the watermark amplitude spectrum feature;

[0041] Fusing the amplitude spectrum feature with the watermark modulation weight matrix to obtain a modulation amplitude spectrum feature;

[0042] The modulation amplitude spectrum feature and the phase spectrum feature are combined and inverse Fourier transformed to obtain the watermark audio data.

[0043] In order to solve the above technical problems, the present application also proposes an audio watermark generation device, which includes: an acquisition module, an extraction module, a generation module, and an update module; wherein,

[0044] The acquisition module is configured to acquire a current batch of audio data sets, wherein the audio data sets include target audio data and other audio data;

[0045] The extraction module is configured to extract a first latent vector representation of the target audio data;

[0046] The generating module is configured to input the target audio data into an audio watermark embedding network of an audio watermark generating network to add watermark data, thereby obtaining watermarked audio data;

[0047] The extraction module is used to extract a second latent vector representation of the watermarked audio data;

[0048] The extraction module is configured to extract a third latent vector representation of the other audio data;

[0049] The updating module is configured to construct a triplet loss using the first latent vector representation, the second latent vector representation, and the third latent vector representation;

[0050] The updating module is used to update the audio watermark generation network using the triplet loss, wherein the updated audio watermark embedding network is used to generate an audio watermark.

[0051] In order to solve the above technical problems, the present application also proposes an audio watermark generation device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the audio watermark generation method as described above.

[0052] In order to solve the above technical problems, the present application also proposes a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the above audio watermark generation method.

[0053] Compared with the prior art, the beneficial effects of the present application are as follows: the audio watermark generation device obtains the current batch of audio data sets, wherein the audio data sets include target audio data and other audio data; extracts the first latent vector representation of the target audio data; inputs the target audio data into the audio watermark embedding network of the audio watermark generation network to add watermark data to obtain watermarked audio data; extracts the second latent vector representation of the watermarked audio data; extracts the third latent vector representation of the other audio data; constructs a triplet loss using the first latent vector representation, the second latent vector representation, and the third latent vector representation; and uses the triplet loss to update the audio watermark generation network, wherein the updated audio watermark embedding network is used to generate an audio watermark. Through the above-mentioned audio watermark generation method, the audio features before and after the watermark is added are aligned in the latent space to update the network parameters of the audio watermark embedding network, thereby improving the audio watermark generation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0055] Figure 1 This is a flowchart of the first embodiment of the audio watermark generation method provided by this application;

[0056] Figure 2 This is a schematic diagram of the architecture of the audio watermark based on neural network and random alternating simulation attack provided by this application;

[0057] Figure 3 yes Figure 1 The specific flow chart of step S13 of the audio watermark generation method is shown;

[0058] Figure 4 This is a flowchart of the second embodiment of the audio watermark generation method provided by this application;

[0059] Figure 5 This is a structural diagram of an embodiment of an audio watermark generation device provided by this application;

[0060] Figure 6 This is a structural diagram of an embodiment of an audio watermark generation device provided by this application;

[0061] Figure 7 It is a structural diagram of an embodiment of a computer storage medium provided by this application. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0063] The terms "first," "second," "third," "fourth," and the like (if any) in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product, or apparatus.

[0064] With the increasing popularity of digital multimedia (including but not limited to audio, images, and video) and the profound changes in its dissemination methods, copyright protection and authenticity verification of this content have become increasingly important. Low-cost tampering, forgery, and splicing methods can wholesale modify or regenerate information carriers distributed on the mobile internet, posing cognitive risks to the general public. Audio, as a widely used information carrier in digital media, presents corresponding security challenges due to its rapid dissemination and ease of copying and modification. Therefore, ensuring the integrity and copyright protection of audio has become a key research area.

[0065] Traditional digital multimedia information protection measures, similar to the encoding-then-decoding method used in communication systems, encrypt the multimedia information carrier, rendering it garbled and incoherent. This information remains invisible to the outside world during transmission. Even if intercepted, unauthorized users cannot access the valid information carried by the multimedia information carrier without the corresponding decoding algorithm, thereby protecting digital media and copyright. This protection method assumes that unauthorized users cannot obtain the corresponding decoding algorithm. However, technological advances have significantly reduced the difficulty of deciphering encoding and decoding algorithms. Unauthorized users can illegally decrypt encrypted information transmitted on the mobile internet, insert harmful information into it, and then re-post it back onto the mobile internet using the same encoding algorithm. Due to the lack of effective authentication, illegally tampered multimedia carriers cannot be identified, causing cognitive confusion for mobile internet users and creating the risk of public transmission vulnerabilities.

[0066] Digital watermarking technology provides a covert and effective means of indicating copyright ownership or verifying content authenticity by embedding imperceptible information into digital works. A key challenge lies in ensuring that the embedded watermark is not only transparent to the end user but also resilient to attacks from various signal processing operations. Digital watermarking technology is a crucial research area in the field of information hiding, effectively protecting information security without compromising its visibility. Unlike traditional encryption technology, digital watermarking protects content by leaving a unique copyright mark on digital media without restricting its normal use. Watermarks embedded in digital media maintain the readability of the content while enhancing its security. In the field of digital audio, in particular, the use of audio digital watermarking can achieve multiple functions, including copyright protection, content verification, and copy control, and has significant research significance for improving audio security.

[0067] However, the development of audio watermarking faces more complex challenges than image and video watermarking. Besides balancing imperceptibility and robustness, it must also account for the high sensitivity of the human auditory system, which places even stricter demands on audio quality. Therefore, in-depth research in audio digital watermarking technology is not only a technical requirement but also an inevitable trend in the pursuit of a high-quality audio experience.

[0068] The existing neural network-based watermark embedding network has the problems of fixed time-frequency resolution and spectrum leakage in STFT, and the loss function it provides cannot fully characterize the difference between the audio before and after adding the watermark.

[0069] Therefore, the audio watermark generation method provided in this application designs a latent vector representation consistency optimization network to align the audio before and after adding the watermark in the latent space, which can well characterize the difference between the audio before and after adding the watermark.

[0070] Please refer to the following for details: Figure 1 and Figure 2 , Figure 1 This is a flowchart of the first embodiment of the audio watermark generation method provided by this application. Figure 2 This is a schematic diagram of the architecture of the audio watermark based on neural network and random alternating simulation attack provided by this application.

[0071] The audio watermark generation method of the present application is applied to an audio watermark generation device, wherein the audio watermark generation device of the present application can be a server, a terminal device, or a system composed of a server and a terminal device. Accordingly, the various parts of the audio watermark generation device, such as the various units, subunits, modules, and submodules, can be all set in the server, all set in the terminal device, or separately set in the server and the terminal device.

[0072] Furthermore, the server described above may be either hardware or software. When the server is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it may be implemented as multiple software programs or software modules, such as software or software modules for providing a distributed server, or as a single software program or software module, without further limitation.

[0073] like Figure 1 As shown, the specific steps are as follows:

[0074] Step S11: Acquire the audio data set of the current batch, wherein the audio data set includes target audio data and other audio data.

[0075] In this embodiment of the present application, the current batch of audio data sets is used to Figure 2 The audio watermark generation network shown, or the audio watermark generation architecture, is used as a training dataset for network parameter update, and the current batch of audio datasets is used for network update in the current iteration stage.

[0076] The target audio data is the audio data currently used for audio watermark generation, and the other audio data is any audio data in the current batch of audio data sets except the target audio data.

[0077] Step S12: extracting a first latent vector representation of the target audio data.

[0078] In an embodiment of the present application, an audio watermark generation device extracts a first latent vector representation of the target audio data before adding a watermark, wherein extracting the latent vector representation of the audio is to convert the original audio signal into a low-dimensional, dense vector for use in machine learning tasks, such as the watermark generation task of the present application.

[0079] Specific as Figure 2 As shown, the audio watermark generation device inputs the target audio data before adding the watermark into the latent vector representation consistency optimization network to extract the relevant latent vector representation.

[0080] In a specific embodiment, the main network of the latent vector representation consistency optimization network provided by the present application can be a ViT basic network (Vision Transformer), which is used to directly extract the first latent vector representation of the target audio data. .

[0081] For the watermarked audio data after adding the watermark, the latent vector representation consistency optimization network extracts the relevant second latent vector representation For other audio data, the latent vector representation consistency optimization network extracts the relevant third latent vector representation .

[0082] Step S13: input the target audio data into the audio watermark embedding network of the audio watermark generation network to add watermark data to obtain watermarked audio data.

[0083] In the embodiment of the present application, the audio watermark generating device inputs the target audio data into Figure 2 The audio watermark embedding network shown is used to encode and fuse the watermark information in the watermark codebook library into the audio data to add relevant watermark data and generate watermarked audio data.

[0084] like Figure 2 As shown, the audio watermark generation network of the present application specifically includes an audio watermark embedding network, a latent vector representation consistency optimization network, and an audio watermark extraction network.

[0085] Regarding the watermarking principle of embedding audio watermarks into the network provided by this application, please continue to combine Figure 2 See Figure 3 , Figure 3 yes Figure 1 The specific flow chart of step S13 of the audio watermark generation method is shown.

[0086] like Figure 3 As shown, the specific steps are as follows:

[0087] Step S131: input the target audio data into the audio watermark embedding network, and extract the amplitude spectrum features and phase spectrum features of the target audio data.

[0088] In the embodiment of the present application, the audio watermark embedding network performs windowing and short-time Fourier transform on the audio data to obtain the amplitude spectrum feature and phase spectrum feature of STFT, and the feature sizes are both ,in Indicates the number of feature frames, Represents the feature dimension.

[0089] Step S132: Select one of the watermark information codes from the watermark codebook library, perform frame-level replication according to the number of frames of the amplitude spectrum feature, and obtain watermark data.

[0090] In the embodiment of the present application, the audio watermark embedding network will randomly select a watermark information code from the watermark code library, and perform frame-level replication according to the number of frames of the amplitude spectrum feature. The size of the watermark code after replication is ,in Indicates the number of feature frames, Indicates the number of bits used to encode the watermark information.

[0091] Step S133: perform frame-level splicing on the amplitude spectrum feature and the watermark data to obtain the watermark amplitude spectrum feature.

[0092] In the embodiment of the present application, the audio watermark embedding network performs frame-level splicing on the amplitude spectrum feature extracted in step S131 and the watermark data extracted in step S132 to obtain the amplitude spectrum feature integrated with the watermark information, i.e., the watermark amplitude spectrum feature, whose feature size is .

[0093] Step S134: extracting a watermark modulation weight matrix based on the watermark amplitude spectrum characteristics.

[0094] In the embodiment of the present application, the watermark modulation weight matrix is ​​an adaptive mask in deep watermark embedding, which is composed of a neural network (such as Figure 2 The network shown is generated by a Deep Residual Network (DRN) with a network depth of 50, which is mainly used to slightly adjust the amplitude spectrum to maintain auditory integrity; resist signal processing attacks; and dynamically optimize the embedding position and strength according to the audio content.

[0095] Compared with traditional methods, the watermark modulation weight matrix combined with deep learning can better balance imperceptibility and robustness.

[0096] Watermark modulation weight matrix is a weight matrix with the same size as the original amplitude spectrum, each element of which is express:

[0097] How to adjust the raw magnitude spectrum To embed watermarks while minimizing the damage to audio quality.

[0098] The value range is usually in, To modulate the intensity, for example 0.05, make sure to fine-tune the amplitude rather than completely replace it.

[0099] Step S135: Fusing the amplitude spectrum feature with the watermark modulation weight matrix to obtain the modulation amplitude spectrum feature.

[0100] In an embodiment of the present application, the audio watermark embedding network performs a point multiplication on the original amplitude spectrum feature and the watermark modulation weight matrix to obtain the amplitude spectrum feature modulated by the watermark information, that is, the modulated amplitude spectrum feature.

[0101] Step S136: Combine the modulation amplitude spectrum features and the phase spectrum features and perform inverse Fourier transformation to obtain watermarked audio data.

[0102] In the embodiment of the present application, the audio watermark embedding network combines the phase spectrum features with the modulation amplitude spectrum features generated in step S135, performs an inverse Fourier transform, and obtains the audio sample with the watermark information added, that is, the watermarked audio data.

[0103] Furthermore, in the above step S132, the audio watermark embedding network needs to select multiple watermark information from the watermark codebook library, so as to add watermarks to the same audio sample using different watermark information.

[0104] Specifically, during the watermark embedding training process, current watermark embedding techniques randomly add only one watermark to each training audio sample. This can easily lead to watermark information dependency. That is, for the same audio sample, choosing different watermark information to embed during the watermark embedding network training process will result in different network parameter update gradients.

[0105] Therefore, the audio watermark embedding network provided in this application chooses to adopt the method of adding multiple sets of watermarks together, and adds multiple sets of watermark information to the same training audio sample at the same time to solve the information dependence problem caused by the single watermark effect.

[0106] Specifically, the audio watermark generating device inputs the target audio data into the audio watermark embedding network and adds a number of different watermark data respectively to obtain a number of watermarked audio data, which are used to respectively calculate the triplet loss corresponding to each watermarked audio data.

[0107] Step S14: extracting a second latent vector representation of the watermarked audio data.

[0108] In the embodiment of the present application, the latent vector representation consistency optimization network extracts the second latent vector representation related to the watermarked audio data .

[0109] Step S15: extracting a third latent vector representation of other audio data.

[0110] In this embodiment of the present application, the latent vector representation consistency optimization network extracts a third latent vector representation related to other audio data .

[0111] Among them, the third latent vector represents The latent vector representation before or after the watermark data is added to the other audio data. The other audio data can be any audio data in the audio dataset except the target audio data.

[0112] In summary, in order to ensure that the audio samples before and after adding the watermark are basically the same, this application sends both the audio samples before and after adding the watermark to the ViT basic network (Vision Transformer), and extracts their corresponding latent vector representations respectively. and .

[0113] Step S16: Construct a triplet loss using the first latent vector representation, the second latent vector representation, and the third latent vector representation.

[0114] In the embodiment of the present application, after the current batch of training, the audio watermark generation device constructs a latent vector representation triplet ( , , ),in and They represent the latent vector representations of the current audio sample before and after the watermark information is added, respectively, extracted by the ViT-base model; Represents the potential vector representation of other audio samples in this training batch before or after adding watermarks.

[0115] For each audio sample, calculate the representation triplet loss function : ,in represents the cosine distance between two voiceprint models, is the cosine distance boundary (usually set to 0.2), 、 They represent the latent vector representation corresponding to the current audio sample and the latent vector representation of another randomly selected audio sample in the current training batch after the watermark information is added for the first time.

[0116] Furthermore, as shown in the above step S136, the present application selects to add multiple sets of watermark information to the same training audio sample at the same time. Therefore, the audio watermark generating device can generate multiple watermark audio data of the target audio data, and each watermark audio data corresponds to a second latent vector representation. .

[0117] Specifically, for the current audio data, the audio watermark generation device selects N-1 watermark information codes from the watermark code library and adds them to the audio data, and respectively sends them to the ViT basic network to extract the corresponding latent vector representation, and respectively calculates the corresponding triplet loss function ,..., .

[0118] Furthermore, the audio watermark generation device may also calculate the mean square error between the STFT features of the audio sample before adding the watermark and the STFT features of the audio sample after adding the watermark: ,in It indicates that the audio sample obtains the STFT feature of the audio after the watermark information is added for the first time. Therefore, the audio watermark generation device also needs to calculate the mean square error loss of N-1 STFT features. ,..., .

[0119] Step S17: using the triplet loss to update the audio watermark generation network, wherein the updated audio watermark embedding network is used to generate the audio watermark.

[0120] In the embodiment of the present application, the audio watermark generation device accumulates and averages all triplet loss functions in step S16 to obtain the overall loss function of the latent vector representation consistency optimization network. And the mean square error loss function of the audio watermark embedding network .

[0121] This application adds N watermark information embedding codes to the above sample at the same time and performs averaging processing on the loss function to weaken the dependence on the embedding code of a single watermark information.

[0122] In the present application, an audio watermark generation device obtains a current batch of audio data sets, wherein the audio data sets include target audio data and other audio data; extracts a first latent vector representation of the target audio data; inputs the target audio data into an audio watermark embedding network of an audio watermark generation network to add watermark data to obtain watermarked audio data; extracts a second latent vector representation of the watermarked audio data; extracts a third latent vector representation of the other audio data; constructs a triplet loss using the first latent vector representation, the second latent vector representation, and the third latent vector representation; and uses the triplet loss to update the audio watermark generation network, wherein the updated audio watermark embedding network is used to generate an audio watermark. Through the above-mentioned audio watermark generation method, the audio features before and after the watermark is added are aligned in the latent space to update the network parameters of the audio watermark embedding network, thereby improving the audio watermark generation effect.

[0123] Furthermore, attack simulations in existing systems randomly mix multiple attack methods together to attack training audio samples. The effects of these multiple attack methods may cancel each other out. Therefore, this application adopts a rotational attack method, using the same attack method for one training data batch and randomly applying a different attack method for the next training data batch. This ensures that only one simulated attack method is affected during a single network parameter update.

[0124] Please refer to the following for details: Figure 4 , Figure 4 This is a flow chart of the second embodiment of the audio watermark generation method provided by this application.

[0125] like Figure 4 As shown, the specific steps are as follows:

[0126] Step S21: input the watermark data into the audio watermark extraction network of the audio watermark generation network to obtain the first watermark information code.

[0127] Step S22: Select a simulation attack method for the current batch of audio data sets.

[0128] Step S23: Perform attack simulation on the watermark audio data according to the simulation attack method to generate attack audio data.

[0129] In an embodiment of the present application, the audio watermark generation device sends the audio strips after adding the watermark to each audio sample to the simulation attack simulation layer, randomly selects an attack method from the attack methods to simulate the attack on all audio samples in the training batch, and obtains the audio samples with watermark information after the attack, that is, the attack audio data.

[0130] Step S24: Input the attack audio data into the audio watermark extraction network to extract the second watermark information code.

[0131] In the embodiment of the present application, the audio watermark generating device inputs the watermark data added in the audio watermark generating process and the attack audio data generated in step S23 into the audio watermark generating device. Figure 2 The audio watermark extraction network shown extracts the first watermark information encoding and the second watermark information encoding.

[0132] Specifically, the audio watermark generation device first sends the above audio data into a deep residual network with a network depth of 34, and then passes it through an average pooling layer to finally extract the embedded watermark information.

[0133] Step S25: Utilize the first watermark information encoding and the second watermark information encoding to construct a difference loss.

[0134] In the embodiment of the present application, the audio watermark generation device performs a cosine distance measurement on the extracted watermark information code and the embedded watermark information code to obtain a difference loss function between the two watermark information codes:

[0135]

[0136] in, Encode the embedded watermark information, Encode the extracted watermark information.

[0137] Step S26: Use triplet loss and difference loss to form a global loss.

[0138] Step S27: Update the audio watermark generation network using the global loss.

[0139] In this embodiment of the present application, the audio watermark generation device adds all the loss functions constructed above to obtain the global loss function of the audio watermark generation network: The audio watermark generation device completes the network gradient update of the current training phase based on the global loss function constructed based on the current batch of audio data sets.

[0140] For the next batch of audio data sets and the training phase, the audio watermark generation device repeats the above steps. When the simulation attack method is selected in step S22, it is necessary to randomly switch to another attack method to simulate the attack on all audio samples in the training batch, so that the audio samples in the current training batch are subjected to the same attack simulation, and the audio samples in different training batches are subjected to different attack simulations. Ultimately, the audio watermark information embedding and extraction network is optimized.

[0141] The audio watermark generation method of the present application designs a latent vector representation consistency optimization network to align the audio before and after adding the watermark in the latent space.

[0142] The audio watermark generation method of the present application is designed to add multiple sets of watermarks together. For the same training audio sample, multiple sets of random watermark information are added at the same time to solve the dependency problem caused by adding a single watermark information to the current sample.

[0143] This application's audio watermark generation method proposes a method of alternating attack methods. For each training data batch, the same attack method is used, and for the next training data batch, a different attack method is randomly used. By alternating training with a single attack method, the interference of the attack method is uniformly limited to the current network space.

[0144] In order to implement the above audio watermark generation method, this application also proposes an audio watermark generation device, please refer to Figure 5 , Figure 5 It is a structural diagram of an embodiment of an audio watermark generation device provided by this application.

[0145] The audio watermark generating apparatus 500 of this embodiment includes: an acquisition module 51 , an extraction module 52 , a generation module 53 , and an update module 54 .

[0146] The acquisition module 51 is configured to acquire a current batch of audio data sets, wherein the audio data sets include target audio data and other audio data.

[0147] The extraction module 52 is configured to extract a first latent vector representation of the target audio data.

[0148] The generating module 53 is configured to input the target audio data into the audio watermark embedding network of the audio watermark generating network to add watermark data, thereby obtaining watermarked audio data.

[0149] The extraction module 52 is configured to extract a second latent vector representation of the watermarked audio data.

[0150] The extraction module 52 is configured to extract a third latent vector representation of the other audio data.

[0151] The updating module 54 is configured to construct a triplet loss using the first latent vector representation, the second latent vector representation, and the third latent vector representation.

[0152] The updating module 54 is configured to update the audio watermark generation network using the triplet loss, wherein the updated audio watermark embedding network is used to generate an audio watermark.

[0153] In order to implement the above audio watermark generation method, this application also proposes an audio watermark generation device, please refer to Figure 6 , Figure 6 It is a structural diagram of an embodiment of an audio watermark generation device provided by this application.

[0154] The audio watermark generating device 400 of this embodiment includes a processor 41 , a memory 42 , an input and output device 43 , and a bus 44 .

[0155] The processor 41 , memory 42 , and input / output device 43 are respectively connected to a bus 44 . The memory 42 stores program data, and the processor 41 is used to execute the program data to implement the audio watermark generation method described in the above embodiment.

[0156] In the embodiments of the present application, the processor 41 may also be referred to as a CPU (Central Processing Unit). The processor 41 may be an integrated circuit chip with signal processing capabilities. The processor 41 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor, or the processor 41 may be any conventional processor.

[0157] This application also provides a computer storage medium, please continue to refer to Figure 7 , Figure 7 6 is a schematic structural diagram of an embodiment of a computer storage medium provided in the present application. The computer storage medium 600 stores a computer program 61. When the computer program 61 is executed by a processor, it is used to implement the audio watermark generation method of the above embodiment.

[0158] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0159] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating an audio watermark, characterized in that: The audio watermark generation method comprises: obtaining a current batch of audio data sets, wherein the audio data sets include target audio data and other audio data; extracting a first latent vector representation of the target audio data; Inputting the target audio data into the audio watermark embedding network of the audio watermark generation network to add watermark data to obtain watermarked audio data; extracting a second latent vector representation of the watermarked audio data; extracting a third latent vector representation of the other audio data; constructing a triplet loss using the first latent vector representation, the second latent vector representation, and the third latent vector representation; updating the audio watermark generation network using the triplet loss, wherein the updated audio watermark embedding network is used to generate an audio watermark; The audio watermark generation network iteratively updates, based on the triplet loss, a direction in which the first latent vector representation is closer to the second latent vector representation, and a direction in which the distance between the third latent vector representation and the first latent vector representation is closer to a cosine distance boundary; The audio data in the current batch of audio data sets are attacked using the same attack method, and the audio data in the next batch of audio data sets are attacked using another attack method. The audio watermark generation network is iteratively updated by alternating training using different attack methods on different batches of audio data sets.

2. The audio watermark generation method according to claim 1, characterized in that: The audio watermark generation method further includes: extracting a first short-time Fourier transform feature of the target audio data; Extracting a second short-time Fourier transform feature of the watermarked audio data; constructing an error loss using the first short-time Fourier transform feature and the second short-time Fourier transform feature; The updating of the audio watermark generation network by using the triplet loss includes: updating the audio watermark generation network by using the triplet loss and the error loss.

3. The audio watermark generation method according to claim 1 or 2, characterized in that: The step of inputting the target audio data into an audio watermark embedding network of an audio watermark generation network to add watermark data to obtain watermarked audio data comprises: inputting the target audio data into the audio watermark embedding network to add a plurality of different watermark data respectively to obtain a plurality of watermarked audio data; The constructing of a triplet loss using the first latent vector representation, the second latent vector representation, and the third latent vector representation comprises: constructing an intermediate triplet loss using the second latent vector representation of each watermarked audio data, the first latent vector representation, and the third latent vector representation, respectively; The triplet loss is constructed by utilizing the intermediate triplet losses corresponding to the plurality of watermarked audio data.

4. The audio watermark generation method according to claim 1, characterized in that: The extracting the third latent vector representation of the other audio data includes: extracting the third latent vector representation of the other audio data before the watermark data is added, or extracting the third latent vector representation of the other audio data after the watermark data is added.

5. The audio watermark generation method according to claim 1, characterized in that: The audio watermark generation method further includes: inputting the watermark data into an audio watermark extraction network of the audio watermark generation network to obtain a first watermark information code; Selecting a simulation attack method for the current batch of audio data sets; Performing attack simulation on the watermarked audio data according to the simulation attack mode to generate attack audio data; Inputting the attack audio data into the audio watermark extraction network to extract the second watermark information code; Constructing a difference loss using the first watermark information encoding and the second watermark information encoding; The updating of the audio watermark generation network by using the triplet loss includes: using the triplet loss and the difference loss to form a global loss; The audio watermark generation network is updated using the global loss.

6. The audio watermark generation method according to claim 5, characterized in that: The simulation attack method for audio datasets of other batches is different from the simulation attack method for audio datasets of the current batch.

7. The audio watermark generation method according to claim 1, characterized in that: The step of inputting the target audio data into an audio watermark embedding network of an audio watermark generation network to add watermark data to obtain watermarked audio data comprises: inputting the target audio data into the audio watermark embedding network, extracting amplitude spectrum features and phase spectrum features of the target audio data; Select one of the watermark information codes from the watermark codebook library, perform frame-level replication according to the number of frames of the amplitude spectrum feature, and obtain watermark data; Performing frame-level splicing on the amplitude spectrum feature and the watermark data to obtain the watermark amplitude spectrum feature; Extracting a watermark modulation weight matrix based on the watermark amplitude spectrum feature; Fusing the amplitude spectrum feature with the watermark modulation weight matrix to obtain a modulation amplitude spectrum feature; The modulation amplitude spectrum feature and the phase spectrum feature are combined and inverse Fourier transformed to obtain the watermark audio data.

8. An audio watermark generation device, characterized in that: The audio watermark generating device includes: an acquisition module, an extraction module, a generation module, and an update module; The acquisition module is configured to acquire a current batch of audio data sets, wherein the audio data sets include target audio data and other audio data; The extraction module is configured to extract a first latent vector representation of the target audio data; The generating module is configured to input the target audio data into an audio watermark embedding network of an audio watermark generating network to add watermark data, thereby obtaining watermarked audio data; The extraction module is used to extract a second latent vector representation of the watermarked audio data; The extraction module is configured to extract a third latent vector representation of the other audio data; The updating module is configured to construct a triplet loss using the first latent vector representation, the second latent vector representation, and the third latent vector representation; The updating module is configured to update the audio watermark generation network using the triplet loss, wherein the updated audio watermark embedding network is used to generate an audio watermark; The audio watermark generation network iteratively updates, based on the triplet loss, a direction in which the first latent vector representation is closer to the second latent vector representation, and a direction in which the distance between the third latent vector representation and the first latent vector representation is closer to a cosine distance boundary; The audio data in the current batch of audio data sets are attacked using the same attack method, and the audio data in the next batch of audio data sets are attacked using another attack method. The audio watermark generation network is iteratively updated by alternating training using different attack methods on different batches of audio data sets.

9. An audio watermark generation device, characterized in that: The audio watermark generating device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the audio watermark generation method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the audio watermark generation method according to any one of claims 1 to 7.

Images