An audio watermark generation method and related apparatus

Abstract

The application discloses an audio watermark generation method and related devices, and relates to the technical field of artificial intelligence, and comprises the following steps: acquiring input audio, performing acoustic feature block and hidden space coding on the input audio to obtain hidden space representation vectors corresponding to a plurality of acoustic feature blocks of the input audio, respectively, fusing the hidden space representation vectors corresponding to the plurality of acoustic feature blocks with watermark information coding vectors to obtain watermark embedding vectors corresponding to the plurality of acoustic feature blocks, respectively, and decoding and reconstructing watermark embedding audio containing watermark information according to the watermark embedding vectors corresponding to the plurality of acoustic feature blocks. The hidden space coding can remove a large amount of audio redundant information in the acoustic feature blocks, and only keep more critical and more representative features of the nature of the audio. The watermark information coding vectors are embedded on the features, the influence of the watermark on the overall characteristics of the audio can be reduced as much as possible, so that the watermark is not easy to be perceived, and the listening experience is better.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an audio watermark generation method and related device. BACKGROUND

[0002] The audio watermark embedding method based on a neural network is to embed watermark information in the original audio signal frame by frame. Specifically, first, the original audio signal is subjected to a short-time Fourier transform (STFT) to separate the amplitude spectrum features and the phase spectrum features of each audio frame. Then, the watermark information code is spliced with the amplitude spectrum features of each audio frame to construct a joint feature tensor of each audio frame as the input of a convolutional neural network, and then the watermark embedding weight matrix corresponding to each frequency point of the amplitude spectrum is learned. After the weight matrix of each audio frame is multiplied element by element with the amplitude spectrum features of the corresponding audio frame, the watermark embedding audio containing the watermark information is generated by inverse short-time Fourier transform (iSTFT) in combination with the phase spectrum features of the corresponding audio frame.

[0003] However, the watermark information embedding method frame by frame will introduce perceptible noise in the original audio signal, reducing the user's listening experience. SUMMARY

[0004] In view of the above problems, the present application provides an audio watermark generation method and related device to achieve the purpose of embedding watermark in the audio without feeling. The specific scheme is as follows:

[0005] The first aspect of the present application provides an audio watermark generation method, comprising:

[0006] obtaining an input audio;

[0007] performing acoustic feature blocking and hidden space coding on the input audio to obtain hidden space representation vectors corresponding to a plurality of acoustic feature blocks of the input audio respectively;

[0008] fusing the hidden space representation vectors corresponding to the plurality of acoustic feature blocks respectively with watermark information code vectors to obtain watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively;

[0009] decoding and reconstructing watermark embedding audio containing watermark information according to the watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively.

[0010] In one possible implementation, the decoding and reconstructing watermark embedding audio containing watermark information according to the watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively comprises:

[0011] decoding the watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively to obtain reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively;

[0012] determine whether the reconstruction quality of the reconstruction feature blocks respectively corresponding to the plurality of acoustic feature blocks satisfies a preset quality requirement;

[0013] take the reconstruction feature block that does not satisfy the quality requirement as a target reconstruction feature block, perform feature compensation replacement on the target reconstruction feature block according to the acoustic feature block corresponding to the target reconstruction feature block, and obtain an updated reconstruction feature block;

[0014] generate the watermark embedding audio according to the reconstruction feature blocks or the updated reconstruction feature blocks respectively corresponding to the plurality of acoustic feature blocks.

[0015] In a possible implementation, the determining whether the reconstruction quality of the reconstruction feature blocks respectively corresponding to the plurality of acoustic feature blocks satisfies a preset quality requirement comprises:

[0016] calculating the energy of the reconstruction feature blocks respectively corresponding to the plurality of acoustic feature blocks as the reconstruction energy respectively corresponding to the plurality of acoustic feature blocks;

[0017] determining whether the reconstruction quality of the reconstruction feature blocks respectively corresponding to the plurality of acoustic feature blocks satisfies the quality requirement according to whether the reconstruction energy respectively corresponding to the plurality of acoustic feature blocks is greater than or equal to a preset energy threshold value;

[0018] and / or,

[0019] for each acoustic feature block in the plurality of acoustic feature blocks, calculating the gap between the acoustic feature block and the reconstruction feature block corresponding to the acoustic feature block as the reconstruction gap corresponding to the acoustic feature block;

[0020] determining whether the reconstruction quality of the reconstruction feature blocks respectively corresponding to the plurality of acoustic feature blocks satisfies the quality requirement according to whether the reconstruction gap respectively corresponding to the plurality of acoustic feature blocks is greater than or equal to a preset gap threshold value.

[0021] In a possible implementation, the calculating the gap between the acoustic feature block and the reconstruction feature block corresponding to the acoustic feature block comprises:

[0022] calculating the mean square error between the acoustic feature block and the reconstruction feature block corresponding to the acoustic feature block.

[0023] In a possible implementation, the performing feature compensation replacement on the target reconstruction feature block according to the acoustic feature block corresponding to the target reconstruction feature block comprises:

[0024] for each acoustic feature block in the plurality of acoustic feature blocks:

[0025] if the reconstruction gap corresponding to the acoustic feature block is less than the gap threshold value, performing feature compensation on the reconstruction feature block corresponding to the acoustic feature block according to the acoustic feature block;

[0026] if the reconstruction energy corresponding to the acoustic feature block is less than the energy threshold value, replacing the reconstruction feature block corresponding to the acoustic feature block with the acoustic feature block.

[0027] In a possible implementation, the feature compensation on the reconstruction feature block corresponding to the acoustic feature block according to the acoustic feature block includes:

[0028] performing weighted summation on the acoustic feature block and the reconstruction feature block corresponding to the acoustic feature block to obtain a fusion feature block corresponding to the acoustic feature block as an updated reconstruction feature block corresponding to the acoustic feature block.

[0029] In a possible implementation, the acoustic feature blocking and the latent space coding of the input audio to obtain the latent space representation vectors corresponding to the plurality of acoustic feature blocks respectively, fusing the latent space representation vectors corresponding to the plurality of acoustic feature blocks respectively with the watermark information coding vectors to obtain the watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively, and decoding the watermark embedding audio containing the watermark information according to the watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively include:

[0030] sending the input audio into a pre-trained audio watermark embedding model to obtain the watermark embedding audio output by the model.

[0031] In a possible implementation, the audio watermark embedding model and the watermark extraction model are jointly trained.

[0032] The training process of the joint training includes:

[0033] obtaining an original audio sample;

[0034] sending the original audio sample into an audio watermark embedding model to obtain a watermark audio sample containing watermark information;

[0035] calculating a first loss between acoustic features of the original audio sample and the watermark audio sample respectively;

[0036] sending the watermark audio sample into an attack simulation layer to perform attack simulation to obtain an attacked audio sample;

[0037] extracting the watermark from the attacked audio sample through a watermark extraction model to obtain a watermark codebook recovery vector;

[0038] compute a second loss between the watermark codebook recovery vector and a watermark codebook vector used to encode the watermark information encoding vector;

[0039] train the audio watermark embedding model and the watermark extraction model according to the first loss and the second loss.

[0040] The second aspect of the present application provides a computer program product, comprising computer readable instructions, which, when executed on an electronic device, cause the electronic device to implement the audio watermark generation method of the first aspect or any implementation manner of the first aspect.

[0041] The third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0042] The memory is configured to store a computer program;

[0043] The processor is configured to execute the computer program to enable the electronic device to implement the audio watermark generation method of the first aspect or any implementation manner of the first aspect.

[0044] The fourth aspect of the present application provides a computer storage medium, which carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the audio watermark generation method of the first aspect or any implementation manner of the first aspect.

[0045] By means of the above technical solution, the audio watermark generation method provided by the present application considers the way of embedding watermark on the amplitude spectrum frame by frame. The amplitude spectrum contains a lot of frequency points corresponding to audio redundant information. Embedding watermark on these frequency points may interfere with the normal frequency distribution of the audio, resulting in the watermark being easily perceived. In order to avoid embedding watermark on the audio redundant information as much as possible, the present application obtains the input audio, performs acoustic feature blocking and hidden space coding on the input audio to obtain hidden space representation vectors corresponding to a plurality of acoustic feature blocks of the input audio respectively, fuses the hidden space representation vectors corresponding to the plurality of acoustic feature blocks respectively with watermark information encoding vectors to obtain watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively, and decodes and reconstructs watermark embedding audio containing watermark information according to the watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively. The present application can remove a large amount of audio redundant information in the acoustic feature blocks through hidden space coding, and only keep more critical and more representative features of the audio essence. Embedding watermark information encoding vectors on these features can minimize the impact of watermark on the overall characteristics of the audio, so that the watermark is not easily perceived and the listening experience is better.

[0046] Further, the latent space representation vector is composed of more core features with strong correlation and integrity, and the watermark is embedded in the features with strong correlation and integrity. Due to the mutual restriction and coordination between the features, the watermark information can be better integrated into the overall characteristics of the input audio, without causing excessive damage to a single feature, thereby maintaining the naturalness and continuity of the watermark-embedded audio and improving the listening experience of the watermark-embedded audio. BRIEF DESCRIPTION OF DRAWINGS

[0047] The above and other features, advantages, and aspects of the present disclosure will become more apparent with reference to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals are used to refer to the same or similar elements. It should be understood that the drawings are schematic and elements and features do not necessarily appear to scale.

[0048] Figure 1 A structural schematic diagram of a system architecture provided by the present application is shown in the figure;

[0049] Figure 2 A flowchart of an audio watermark generation method provided by the present application is shown in the figure;

[0050] Figure 3 A structural schematic diagram of an audio watermark embedding model provided by the present application is shown in the figure;

[0051] Figure 4 A calculation process diagram of a model loss provided by the present application is shown in the figure;

[0052] Figure 5 A structural schematic diagram of an audio watermark generation device provided by the present application is shown in the figure;

[0053] Figure 6 A structural schematic diagram of an electronic device provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0054] The embodiments of the present application are described below in conjunction with the accompanying drawings. The terms used in the embodiment part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0055] The embodiments of the present application are described below in conjunction with the accompanying drawings. It is known to those skilled in the art that as technology develops and new scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0056] The terms "first", "second", and the like in the description and in the claims of the present application and above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the terms used in this way can be interchanged, as appropriate, and are merely a way of distinguishing between objects of the same attribute used in the description of the embodiments of the present application. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus containing a series of units includes not only those units, but also other units not clearly listed or inherent to the process, method, product, or apparatus.

[0057] The present application provides an audio watermark generation method and related apparatus, which can be applied to a scenario in which a watermark needs to be embedded in audio, for example, in a copyright protection scenario of a music work and a video work, a watermark can be embedded in a music work and a video work to be protected, so as to explicitly identify the copyright ownership information of the music; in an important audio recording scenario such as a conference recording and an interview recording, a watermark is embedded on a recording audio, so as to detect whether the recording audio is tampered with or verify the authenticity of the recording audio; in an advertisement playing scenario, a watermark is embedded on an advertisement audio to be played, so as to track the playing situation of the advertisement audio in real time; in a covert communication scenario, confidential information is embedded as a watermark in ordinary audio data, is transmitted through a public audio transmission channel, and covert communication is realized.

[0058] It should be noted that the above scenarios are only examples and do not limit the present application.

[0059] Optionally, the audio watermark generation method provided by the present application can be applied to a system architecture as shown in Figure 1 The system can include a terminal 100 and a server 200. The server 200 can include one or more servers (for example, one server is taken as an example in Figure 1 ).

[0060] The terminal 100 or the server 200 can be used alone to execute the audio watermark generation method provided by the embodiments of the present application. In addition, the terminal 100 and the server 200 can also be used cooperatively to execute the audio watermark generation method provided by the embodiments of the present application.

[0061] Next, the product form of the terminal 100 in Figure 1 will be described;

[0062] The terminal 100 in the embodiments of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and the like, and the embodiments of the present application do not make any limitation in this regard.

[0063] In order for those skilled in the art to better understand the present application, the audio watermark generation method of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0064] Referring to Figure 2 , Figure 2 A flowchart of an audio watermark generation method provided by the embodiments of the present application is shown in FIG. 2, which can include the following steps. Figure 2

[0065] Step S201, obtaining input audio.

[0066] Here, the input audio refers to audio data to be embedded with a watermark.

[0067] For example, in the copyright protection scenario of music works and video works, the input audio refers to audio data in the music works or video works; in the important audio recording scenario such as conference recording and interview recording, the input audio refers to recording audio; in the advertisement delivery scenario, the input audio refers to advertisement audio; and in the covert communication scenario, the input audio refers to ordinary audio data.

[0068] Step S202, performing acoustic feature blocking and latent space coding on the input audio to obtain latent space representation vectors corresponding to the plurality of acoustic feature blocks of the input audio.

[0069] The embodiments can first extract acoustic features of the input audio to obtain a feature spectrogram of the input audio, then perform blocking processing on the feature spectrogram to obtain a plurality of acoustic feature blocks, and finally perform latent space coding on each acoustic feature block to obtain a latent space representation vector corresponding to each acoustic feature block.

[0070] Optionally, the process of "extracting acoustic features of the input audio" can include performing windowing processing on the input audio and then extracting acoustic features FilterBank. Here, FilterBank refers to a filter bank, which is a kind of acoustic feature simulating the hearing characteristics of human ears.

[0071] ​The embodiment can smooth the signal boundary of the input audio through windowing processing, avoid high-frequency noise caused by the discontinuity of the input audio, and divide the long audio data into a plurality of short audio data, so that the audio in a local range is approximately stationary, which is conducive to more accurately extracting acoustic features.

[0072] Meanwhile, the FilterBank is used as the extracted acoustic feature. Since the FilterBank is an acoustic feature simulating the hearing characteristics of human ears, the subsequent watermark embedding and hidden space encoding based on the FilterBank can make the watermark embedding only affect a specific frequency band with low auditory sensitivity, and the concealment of the watermark is significantly improved.

[0073] The size of the feature spectrogram obtained through the above processing can be denoted as fxd, where f represents the number of feature frames, and d represents the feature dimension. The feature spectrogram is divided into blocks to obtain a plurality of acoustic feature blocks, and the n th acoustic feature block is denoted as , n = 1, 2, …, N.

[0074] Optionally, the size of each acoustic feature block can be fixed at 16x16.

[0075] Of course, other block sizes can also be set in the application, or the block size can be selected and adapted according to the audio characteristics of the input audio, and the like, which are not limited here.

[0076] Optionally, the encoder of the pre-trained model can be used to encode the hidden space of each acoustic feature block to obtain the hidden space representation vector corresponding to each acoustic feature block.

[0077] Preferably, the pre-trained model can be an AudioMAE model, which is an audio self-supervised pre-training model based on Masked Autoencoder (Masked Autoencoder) technology. The core goal is to mask part of the audio data and reconstruct the missing part, learn the general representation of the audio, and thus improve the understanding ability of the model to the audio features, and support various audio processing tasks (such as classification, generation, identification, etc.).

[0078] Of course, the pre-trained model can also be other models capable of realizing hidden space encoding, such as GPT series large models, etc., which are not limited here.

[0079] In step S203, the hidden space representation vectors corresponding to the plurality of acoustic feature blocks are respectively fused with the watermark information encoding vectors to obtain watermark embedding vectors corresponding to the plurality of acoustic feature blocks.

[0080] Here, the watermark information encoding vector can be a watermark codebook vector selected from a watermark codebook library The vector obtained by encoding.

[0081] Optionally, the watermark information encoder can be pre-trained to encode the watermark codebook vector, wherein the watermark information encoder is trained by using a training watermark codebook vector labeled with a watermark information encoding vector label.

[0082] Optionally, the watermark information encoder can be a network architecture composed of 3 layers of DNN (Deep Neural Network), and of course, the watermark information encoder can also use other network architectures, which are not limited by the present application.

[0083] Preferably, the watermark information encoding vector can be the same dimension as the latent space representation vector, i.e., if the dimension of the latent space representation vector is , then the label dimension used when training the watermark information encoder is also .

[0084] When the watermark information encoding vector and the latent space representation vector are the same dimension, an optional fusion method is to add the latent space representation vector corresponding to each acoustic feature block to the watermark information encoding vector to obtain a watermark embedding vector corresponding to each acoustic feature block, i.e. , wherein represents the latent space representation vector, represents the watermark information encoding vector, represents the watermark embedding vector.

[0085] In addition to the above addition fusion method, there can be other fusion methods, such as direct splicing, weighted addition, etc., and the appropriate fusion method can be selected according to the actual application scenario, which is not limited by the present application.

[0086] It should be further pointed out that the watermark information encoding vector fused with the latent space representation vector corresponding to different acoustic feature blocks can be the same or different.

[0087] Step S204, according to the watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively, the watermark embedding audio containing watermark information is reconstructed by decoding.

[0088] In the present embodiment, the watermark embedding vector corresponding to each acoustic feature block can be decoded to obtain a reconstructed feature block corresponding to each acoustic feature block, for example, the nth reconstructed feature block is denoted as . Next, the watermark embedding vector containing the watermark information can be obtained according to all the reconstructed feature blocks.

[0089] Optionally, the decoder of the pre-trained model can be used to decode the watermark embedding vector corresponding to each acoustic feature block to obtain a reconstructed feature block corresponding to each acoustic feature block.

[0090] The audio watermark generation method provided in the application considers the watermark embedding method frame by frame on the amplitude spectrum. The amplitude spectrum contains a lot of frequency points corresponding to audio redundant information. Embedding watermark on these frequency points may interfere with the normal frequency distribution of the audio, resulting in the watermark being easily perceived. In order to avoid embedding watermark on the audio redundant information as much as possible, the application obtains input audio, performs acoustic feature blocking and latent space coding on the input audio to obtain latent space representation vectors corresponding to a plurality of acoustic feature blocks of the input audio respectively, fuses the latent space representation vectors corresponding to the plurality of acoustic feature blocks respectively with watermark information coding vectors to obtain watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively, and decodes the watermark embedding audio containing watermark information according to the watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively. The latent space coding can remove a large amount of audio redundant information in the acoustic feature block, and only keep the more critical and more representative features of the audio essence. Embedding the watermark information coding vector on these features can minimize the impact of the watermark on the overall characteristics of the audio, so that the watermark is not easily perceived and the listening experience is better.

[0091] Further, the latent space representation vector is composed of more core features with strong correlation and integrity. Embedding watermark in such features with strong correlation and integrity, due to the mutual restriction and coordination between the features, the watermark information can be better integrated into the overall characteristics of the input audio, without causing excessive damage to individual features, thereby maintaining the naturalness and continuity of the watermark embedding audio and improving the listening experience of the watermark embedding audio.

[0092] In some embodiments of the application, the process of the foregoing step S204 "decoding and reconstructing the watermark embedding audio containing watermark information according to the watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively" is introduced.

[0093] In one possible implementation, the embodiment can decode the watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively to obtain reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively, and then directly generate the watermark embedding audio according to the reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively.

[0094] Specifically, the reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively can be spliced into a reconstructed feature spectrogram, and then audio recovery can be performed based on the reconstructed feature spectrogram to obtain the watermark embedding audio.

[0095] In another possible implementation, in consideration of the fact that the reconstructed feature blocks can be slightly perceptible due to low energy, in order to avoid the influence of the reconstructed feature blocks that can be slightly perceptible on the quality of the final watermark-embedded audio, the embodiment can calculate the reconstruction quality of the reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively, and then determine whether the reconstruction quality of the reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively meets the preset quality requirement.

[0096] Here, the reconstruction quality can be used to quantitatively evaluate the perceptible degree of the watermark embedded in the reconstructed feature blocks. If the reconstruction quality of a reconstructed feature block meets the quality requirement, it indicates that the watermark embedded in the reconstructed feature block is almost imperceptible, and then the audio can be directly recovered according to the reconstructed feature block. Conversely, if the reconstruction quality of a reconstructed feature block does not meet the quality requirement, it indicates that the watermark embedded in the reconstructed feature block can be slightly perceptible, and then the original acoustic feature block can be used to replace the reconstructed feature block by feature compensation.

[0097] That is, the embodiment can find a reconstructed feature block that does not meet the quality requirement, denoted as a target reconstructed feature block, replace the target reconstructed feature block by feature compensation according to the acoustic feature block corresponding to the target reconstructed feature block, obtain an updated reconstructed feature block, and then generate a watermark-embedded audio according to the reconstructed feature blocks or the updated reconstructed feature blocks corresponding to the plurality of acoustic feature blocks after the feature compensation. Specifically, generate the watermark-embedded audio according to the feature compensation results (updated reconstructed feature blocks) of the reconstructed feature blocks that meet the quality requirement and the reconstructed feature blocks that do not meet the quality requirement.

[0098] The above "determining whether the reconstruction quality of the reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively meets the preset quality requirement" has multiple implementation manners, and the following two manners are provided herein but are not limited thereto.

[0099] The first manner: calculate the energy of the reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively as the reconstructed energy of the acoustic feature blocks respectively, and determine whether the reconstruction quality of the reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively meets the quality requirement according to whether the reconstructed energy of the acoustic feature blocks respectively is greater than or equal to a preset energy threshold value.

[0100] If the reconstructed energy of an acoustic feature block is greater than or equal to the energy threshold value, it is determined that the reconstruction quality of the reconstructed feature block corresponding to the acoustic feature block meets the quality requirement. If the reconstructed energy of an acoustic feature block is less than the energy threshold value, it is determined that the reconstruction quality of the reconstructed feature block corresponding to the acoustic feature block does not meet the quality requirement.

[0101] Optionally, the calculation formula of the reconstructed energy of an acoustic feature block is as follows: wherein, This represents the reconstructed feature block corresponding to the nth acoustic feature block. The i-th feature amplitude value in the matrix, i=1,2,…,256; This represents the energy of the reconstructed feature block corresponding to the nth acoustic feature block, which is also the reconstruction energy corresponding to the nth acoustic feature block.

[0102] Optionally, for each of the multiple acoustic feature blocks, if the reconstruction energy corresponding to the acoustic feature block is less than the energy threshold, then "performing feature compensation replacement of the target reconstruction feature block according to the acoustic feature block corresponding to the target reconstruction feature block" may include: replacing the reconstruction feature block corresponding to the acoustic feature block with the acoustic feature block.

[0103] Specifically, if the reconstructed energy corresponding to the nth acoustic feature block Below the energy threshold This indicates that the reconstructed feature block corresponding to the nth acoustic feature block has low energy. Therefore, using this low-energy reconstructed feature block to recover the audio may introduce noise interference into the recovered audio, affecting the listening experience. Thus, the nth acoustic feature block can be used instead. Replace the reconstructed feature block corresponding to the nth acoustic feature block The newly reconstructed feature block after compensation, obtained by feature block replacement, is denoted as... .

[0104] It is understandable that the low energy of the reconstructed feature block may be caused by the embedded watermark. In order to avoid poor audio restoration due to the watermark, this application provides a method of replacing it with the original acoustic feature block, which can avoid the audio listening experience degradation caused by the watermark to a certain extent and improve the listening experience of the watermark embedded audio to a certain extent.

[0105] The second method is to calculate the difference between each acoustic feature block and the corresponding reconstructed feature block for each of the multiple acoustic feature blocks. This difference is used as the reconstruction difference for that acoustic feature block. Based on whether the reconstruction differences corresponding to the multiple acoustic feature blocks are greater than or equal to a preset difference threshold, it is determined whether the reconstruction quality of the reconstructed feature blocks corresponding to the multiple acoustic feature blocks meets the quality requirements.

[0106] Optionally, the above-mentioned "calculating the difference between the acoustic feature block and the reconstructed feature block corresponding to the acoustic feature block" may include: calculating the mean square error between the acoustic feature block and the reconstructed feature block corresponding to the acoustic feature block, i.e. ,in, This represents the mean square error between the nth acoustic feature block and its corresponding reconstructed feature block.

[0107] If the reconstruction gap (such as the mean square error mentioned above) corresponding to an acoustic feature block is greater than or equal to the gap threshold, then the reconstruction quality of the reconstructed feature block corresponding to that acoustic feature block is determined to meet the quality requirements; if the reconstruction gap corresponding to an acoustic feature block is less than the gap threshold, then the reconstruction quality of the reconstructed feature block corresponding to that acoustic feature block is determined to not meet the quality requirements.

[0108] Of course, the above-mentioned gap can also be other indicator gaps, such as Frobenius norm, Chebyshev distance, similarity, etc. This embodiment does not make specific limitations.

[0109] Optionally, for each of the multiple acoustic feature blocks, if the reconstruction gap corresponding to the acoustic feature block is less than the gap threshold, then "performing feature compensation and replacement of the target reconstruction feature block according to the acoustic feature block corresponding to the target reconstruction feature block" may include: performing feature compensation on the reconstruction feature block corresponding to the acoustic feature block according to the acoustic feature block.

[0110] Specifically, if the reconstruction gap corresponding to the nth acoustic feature block is as follows: Below the gap threshold This indicates that the reconstruction effect of the reconstructed feature block corresponding to the nth acoustic feature block is poor. This means that the original audio has been significantly modified due to factors such as watermark embedding. Therefore, using this poorly reconstructed feature block to recover the audio will likely result in a poor listening experience. To avoid this problem, we can use the nth acoustic feature block... For the reconstructed feature block corresponding to the nth acoustic feature block Feature compensation is performed, and the newly reconstructed feature blocks obtained by feature block compensation are denoted as follows: .

[0111] Optionally, the process of "performing feature compensation for the reconstructed feature block corresponding to the acoustic feature block" may include: performing a weighted summation of the acoustic feature block and the reconstructed feature block corresponding to the acoustic feature block, for example, The fused feature block corresponding to the acoustic feature block is obtained and used as the updated reconstructed feature block corresponding to the acoustic feature block. Here, 'a' represents... The corresponding weight, b represents The corresponding weight is a+b=1.

[0112] The values ​​of a and b can be determined based on the actual scenario. For example, in one possible scenario, a = 0.15 and b = 0.85.

[0113] It can be understood that the difference between the reconstructed feature block and the original acoustic feature block is likely to be caused by the embedded watermark. In order to avoid poor audio recovery caused by the watermark, the application provides a compensation method using the original acoustic feature block, which can avoid the problem of audio listening feeling caused by the watermark to a certain extent, and improve the listening feeling of the watermark embedded audio to a certain extent.

[0114] It should be further pointed out that the above feature compensation method can be replaced by a feature replacement method, which is not limited by the application.

[0115] There can be an acoustic feature block corresponding to a reconstructed energy less than an energy threshold value, and a reconstructed difference less than a difference threshold value. At this time, the above-mentioned feature replacement method can be used for overall replacement of the feature block, and the above-mentioned feature compensation method can be used for feature compensation, which is not limited by the application.

[0116] After the above feature compensation replacement, the updated reconstructed feature block corresponding to each of the plurality of acoustic feature blocks is obtained or the reconstructed feature block (no updated reconstructed feature block is selected, and the updated reconstructed feature block is selected when there is an updated reconstructed feature block) to generate a watermark embedded audio.

[0117] In summary, the embodiment can correct the reconstructed feature block through feature compensation replacement when the reconstructed quality of the reconstructed feature block does not meet the requirements, so that the updated reconstructed feature block obtained after compensation has higher reconstructed quality, and the listening feeling of the recovered watermark embedded audio is better.

[0118] In some embodiments of the application, an audio watermark embedding model can be pre-trained, and the processes of steps S201-S204 in the foregoing are implemented through the audio watermark embedding model. Therefore, the input audio can be sent into the pre-trained audio watermark embedding model to obtain the watermark embedded audio output by the model.

[0119] Referring to Figure 3 , a structure diagram of an audio watermark embedding model provided by the application is shown.

[0120] As Figure 3 , the audio watermark embedding model can include an acoustic feature blocking module 310, a latent space encoder 320, a watermark information encoder 330, a vector fusion module 340, a decoder 350, and an audio recovery module 360. Optionally, the latent space encoder 320 can be an encoder in the pre-trained model provided in the foregoing, and the decoder 350 can be a decoder in the pre-trained model.

[0121] The acoustic feature chunking module 310 chunks acoustic features of the input audio to obtain a plurality of acoustic feature chunks corresponding to the input audio; the latent space encoder 320 encodes the plurality of acoustic feature chunks in a latent space to obtain a plurality of latent space representation vectors respectively corresponding to the plurality of acoustic feature chunks; the watermark information encoder 330 encodes a watermark codebook vector obtained from a watermark codebook vector library into a watermark information encoding vector; the vector fusion module 340 fuses the plurality of latent space representation vectors respectively corresponding to the plurality of acoustic feature chunks with the watermark information encoding vector to obtain a plurality of watermark embedding vectors respectively corresponding to the plurality of acoustic feature chunks; and the decoder 350 decodes the plurality of watermark embedding vectors respectively corresponding to the plurality of acoustic feature chunks to obtain a plurality of reconstructed feature chunks respectively corresponding to the plurality of acoustic feature chunks. The audio recovery module 360 generates a watermark-embedded audio according to the plurality of reconstructed feature chunks or the updated reconstructed feature chunks respectively corresponding to the plurality of acoustic feature chunks.

[0122] In the case where feature compensation replacement can be implemented, the audio watermark embedding model can further include a feature compensation replacement module 370 configured to determine whether the reconstructed quality of the reconstructed feature chunks meets a preset quality requirement, take a reconstructed feature chunk that does not meet the quality requirement as a target reconstructed feature chunk, and perform feature compensation replacement on the target reconstructed feature chunk according to the acoustic feature chunk corresponding to the target reconstructed feature chunk to obtain an updated reconstructed feature chunk. Accordingly, the audio recovery module 360 generates a watermark-embedded audio according to the plurality of reconstructed feature chunks or the updated reconstructed feature chunks respectively corresponding to the plurality of acoustic feature chunks.

[0123] In one possible implementation, the present embodiment further provides a watermark extraction model, which is jointly trained with the audio watermark embedding model. The following describes the watermark extraction model in combination with Figure 4 The training process of the joint training is described.

[0124] As Figure 4 In the present embodiment, an original audio sample can be obtained, and the original audio sample is input into the audio watermark embedding model to obtain a watermark-embedded audio sample containing watermark information.

[0125] Optionally, the processing of the original audio sample in the audio watermark embedding model can be the same as described above, and details can be referred to the description above, which will not be repeated here.

[0126] To further train the watermark-unaware embedding capability of the model, optionally, for the original audio sample, each acoustic feature chunk of the original audio sample can be masked before being input into the latent space encoder 320, so that the audio watermark embedding model can achieve watermark-unaware embedding in the absence of part of the audio information.

[0127] Further optionally, in order to avoid overfitting of the audio watermark embedding model training, a certain random noise can be added after the hidden space representation vector and the watermark information encoding vector are added, that is, , represents a Gaussian noise with a mean of 0 and a variance of 1.

[0128] Of course, the above noise form can also be other, which is not limited here.

[0129] After obtaining the watermark audio sample containing the watermark information, the embodiment can calculate the first loss between the acoustic features of the original audio sample and the watermark audio sample.

[0130] Taking the first loss as the mean square error loss of the Short-Time Fourier Transform (STFT) features of the original audio sample and the watermark audio sample as an example, the first loss can be: wherein, represents the first loss, represents the norm, represents the STFT feature of the original audio sample, represents the STFT feature of the watermark audio sample.

[0131] In order to simulate various processing and destruction that the audio may encounter in actual application, the application can send the watermark audio sample into the attack simulation layer to simulate the attack, and obtain the attacked audio sample.

[0132] For example, one attack mode can be randomly selected from the attack modes to simulate the attack on all watermark audio samples in the same training batch, and the attacked audio sample corresponding to each watermark audio sample is obtained.

[0133] Of course, different attack modes can also be used to simulate the attack on different watermark audio samples in the same training batch, which is not limited here.

[0134] Optionally, the attack mode can include but is not limited to environmental noise, packet loss and various time domain signal attacks.

[0135] Further, the watermark extraction is performed on the attacked audio sample to obtain a watermark codebook recovery vector.

[0136] As Figure 4 Optionally, the watermark extraction model 410 can be preset, and the watermark codebook recovery vector is extracted from the attacked audio sample by the watermark extraction model 410.

[0137] Optionally, the watermark extraction model 410 can be composed of a deep residual network with a network depth of 34 and an average pooling layer. Then, the attacked audio sample can be sent into the deep residual network and then the average pooling layer, and finally the watermark codebook recovery vector is extracted.

[0138] Next, the difference loss between the watermark codebook recovery vector and the watermark codebook vector is calculated as the second loss, where the watermark codebook vector refers to the vector used to encode the watermark information encoding vector extracted from the watermark codebook vector library.

[0139] Optionally, the calculation formula of the second loss is: wherein, represents the cosine distance between the watermark codebook recovery vector and the watermark codebook vector , represents the second loss.

[0140] Finally, the audio watermark embedding model and the watermark extraction model 410 are trained according to the first loss and the second loss. For example, the first loss and the second loss are added, and the sum is taken as the global loss, i.e. , represents the global loss. Then, the network gradient update of the data in the training batch can be completed by using the global loss .

[0141] The above process is repeated several times, and the optimized audio watermark embedding model and watermark extraction model after training can be obtained.

[0142] Through the above training process, not only the audio watermark embedding model capable of inaudibly embedding the watermark can be trained, but also the corresponding watermark extraction model can be trained, so that the present application not only can inaudibly embed the watermark in the audio, but also can accurately extract the watermark from the audio, which has higher practical value.

[0143] The above introduces an audio watermark generation method provided by an embodiment of the present application. The following will introduce a device for executing the above audio watermark generation method.

[0144] Please refer to Figure 5 , Figure 5 for a structural schematic diagram of an audio watermark generation device provided by an embodiment of the present application. As shown in Figure 5 , the audio watermark generation device can include:

[0145] A data input unit 501 configured to obtain input audio.

[0146] The data processing unit 502 is used to perform acoustic feature segmentation and latent space encoding on the input audio to obtain latent space representation vectors corresponding to multiple acoustic feature blocks of the input audio. The latent space representation vectors corresponding to multiple acoustic feature blocks are fused with the watermark information encoding vector to obtain watermark embedding vectors corresponding to multiple acoustic feature blocks. The watermark embedded audio containing watermark information is decoded and reconstructed based on the watermark embedding vectors corresponding to multiple acoustic feature blocks.

[0147] Each module in the aforementioned audio watermark generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0148] This application also provides an electronic device, which may include at least one processor and a memory connected to the processor, wherein:

[0149] Memory is used to store computer programs;

[0150] The processor is used to execute computer programs to enable electronic devices to implement any of the audio watermark generation methods provided in the embodiments of this application.

[0151] refer to Figure 6 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0152] like Figure 6 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0153] In general, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 608 including, for example, a memory card, a hard disk, and the like; and communication devices 609. The communication devices 609 can allow the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device having various devices is shown, but it is understood that all of the shown devices are not required to be implemented or present. More or less devices can alternatively be implemented or present.

[0154] The embodiment of the present application further provides a computer program product comprising computer readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the audio watermark generation methods provided by the embodiments of the present application.

[0155] The embodiment of the present application further provides a computer readable storage medium, which carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement any of the audio watermark generation methods provided by the embodiments of the present application.

[0156] In addition, it should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e. they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. In addition, the connection relationship between the modules in the device embodiment provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0157] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, and of course can also be implemented by means of dedicated hardware including special integrated circuit, special CPU, special memory, special component, etc. Generally, any function completed by computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuit, digital circuit or special circuit, etc. However, for the application, software program implementation is a better embodiment. Based on such understanding, the technical solution of the application or the part of the application which makes contribution to the prior art can be embodied in the form of software product, which is stored in readable storage medium, such as computer floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a plurality of instructions for making a computer device (which can be personal computer, training device or network device, etc.) execute the method described in various embodiments of the application.

[0158] In the above embodiments, the implementation can be achieved by software, hardware, firmware or any combination thereof, entirely or partially. When implemented by software, the implementation can be achieved in the form of a computer program product, entirely or partially.

[0159] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function described in the embodiments of the application is generated entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be stored by a computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

Claims

1. An audio watermark generation method characterized by, The method comprises: acquiring input audio; performing acoustic feature blocking and hidden space coding on the input audio to obtain hidden space representation vectors corresponding to a plurality of acoustic feature blocks of the input audio respectively; fusing the hidden space representation vectors corresponding to the plurality of acoustic feature blocks respectively with watermark information coding vectors to obtain watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively; decoding and reconstructing watermark embedding audio containing watermark information according to the watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively; wherein the decoding and reconstructing of the watermark embedding audio containing watermark information according to the watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively comprises: decoding the watermark embedding vectors corresponding to the plurality of acoustic feature blocks respectively to obtain reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively; determining whether the reconstructed quality of the reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively meets a preset quality requirement; taking a reconstructed feature block that does not meet the quality requirement as a target reconstructed feature block, performing feature compensation replacement on the target reconstructed feature block according to the acoustic feature block corresponding to the target reconstructed feature block to obtain an updated reconstructed feature block; generating the watermark embedding audio according to the reconstructed feature blocks or the updated reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively.

2. The audio watermark generation method of claim 1, wherein, The determination of whether the reconstructed quality of the reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively meets the preset quality requirement comprises: calculating the energy of the reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively as reconstructed energy corresponding to the plurality of acoustic feature blocks respectively; determining whether the reconstructed quality of the reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively meets the quality requirement according to whether the reconstructed energy corresponding to the plurality of acoustic feature blocks respectively is greater than or equal to a preset energy threshold value; and / or, for each acoustic feature block in the plurality of acoustic feature blocks, calculating the distance between the acoustic feature block and the reconstructed feature block corresponding to the acoustic feature block as a reconstructed distance corresponding to the acoustic feature block; determining whether the reconstructed quality of the reconstructed feature blocks corresponding to the plurality of acoustic feature blocks respectively meets the quality requirement according to whether the reconstructed distance corresponding to the plurality of acoustic feature blocks respectively is greater than or equal to a preset distance threshold value.

3. The audio watermark generation method of claim 2, wherein, The calculation of the distance between the acoustic feature block and the reconstructed feature block corresponding to the acoustic feature block comprises: calculating the mean square error between the acoustic feature block and the reconstructed feature block corresponding to the acoustic feature block.

4. The audio watermark generation method of claim 2, wherein, The feature compensation replacement of the target reconstructed feature block according to the acoustic feature block corresponding to the target reconstructed feature block comprises: for each acoustic feature block in the plurality of acoustic feature blocks: if the reconstructed distance corresponding to the acoustic feature block is less than the distance threshold value, performing feature compensation on the reconstructed feature block corresponding to the acoustic feature block according to the acoustic feature block; if the reconstructed energy corresponding to the acoustic feature block is less than the energy threshold value, replacing the reconstructed feature block corresponding to the acoustic feature block with the acoustic feature block.

5. The audio watermark generation method of claim 4, wherein, The feature compensation of the reconstructed feature block corresponding to the acoustic feature block according to the acoustic feature block comprises: The acoustic feature block and the reconstructed feature block corresponding to the acoustic feature block are weighted and summed to obtain a fusion feature block corresponding to the acoustic feature block as an updated reconstructed feature block corresponding to the acoustic feature block.

6. The audio watermark generation method of any of claims 1-5, wherein, The input audio is subjected to acoustic feature blocking and latent space coding to obtain latent space representation vectors corresponding to a plurality of acoustic feature blocks of the input audio, respectively, and the latent space representation vectors corresponding to the plurality of acoustic feature blocks, respectively, are fused with watermark information coding vectors to obtain watermark embedding vectors corresponding to the plurality of acoustic feature blocks, respectively, and a watermark embedding audio containing watermark information is decoded from the watermark embedding vectors corresponding to the plurality of acoustic feature blocks, respectively, including: The input audio is input into a pre-trained audio watermark embedding model to obtain the watermark embedding audio output by the model.

7. The audio watermark generation method of claim 6, wherein, The audio watermark embedding model and the watermark extraction model are jointly trained; The training process of the joint training includes: obtaining an original audio sample; inputting the original audio sample into an audio watermark embedding model to obtain a watermark audio sample containing watermark information; calculating a first loss between the acoustic features of the original audio sample and the watermark audio sample, respectively; inputting the watermark audio sample into an attack simulation layer to simulate attacks to obtain an attacked audio sample; extracting the watermark from the attacked audio sample through a watermark extraction model to obtain a watermark codebook recovery vector; calculating a second loss between the watermark codebook recovery vector and a watermark codebook vector, the watermark codebook vector being used to encode the watermark information coding vector; training the audio watermark embedding model and the watermark extraction model according to the first loss and the second loss.

8. A computer program product, characterised in that, The computer readable instructions, when executed on an electronic device, cause the electronic device to implement the audio watermark generation method of any one of claims 1 to 7.

9. An electronic device, comprising: The memory is configured to store computer programs; The processor is configured to execute the computer programs to enable the electronic device to implement the audio watermark generation method of any one of claims 1 to 7. The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the audio watermark generation method of any one of claims 1 to 7.

10. A computer storage medium, characterized in that ​

Images