Watermark generation, information processing, audio watermark generation model training method and device

Abstract

The present disclosure provides a watermark generation method and device, an information processing method and device, a multimodal information retrieval method, and an audio watermark generation model training method and device, which relate to the field of artificial intelligence technology, specifically the technical fields of deep learning, large models, and generation models. The specific implementation scheme of the watermark generation method is: obtaining the material to be processed; processing the material to be processed to obtain the text to be processed; inputting the text to be processed into the audio watermark generation model to obtain the audio watermark; the audio watermark generation model is used to characterize the correspondence between the input text and the high-frequency, silent audio watermark. This implementation improves the generation effect of the audio watermark.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as deep learning, large models, and generative models, and in particular to a watermark generation method and device, an information processing method and device, an audio watermark generation model training method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of artificial intelligence, more and more generative products have emerged, but there are also many issues involving copyright, law, etc.

[0003] Existing technologies all directly obtain audio watermarks, and add watermarks at special watermark positions based on the original audio. Although audio watermark encryption and decryption can be performed directly on the audio, this audio watermark cannot directly correspond one-to-one with the relevant material, and the interpretability of the audio watermark is low. Summary of the invention

[0004] Provided are a watermark generation method and device, an information processing method and device, an audio watermark generation model training method and device, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to the first aspect, a watermark generation method is provided, the method comprising: obtaining a material to be processed; processing the material to be processed to obtain a text to be processed; inputting the text to be processed into an audio watermark generation model to obtain an audio watermark; the audio watermark generation model is used to characterize the correspondence between the input text and a high-frequency, silent audio watermark.

[0006] According to the second aspect, an information processing method is provided, which includes: obtaining multimodal materials to be stored and audio watermarks generated by the method described in any implementation method of the first aspect; based on the materials to be stored and the audio watermarks, obtaining retrieval vectors of audio watermarks with similar meanings and various modal materials to be stored; and storing the retrieval vectors in a vector retrieval library.

[0007] According to a third aspect, a method for training an audio watermark generation model is provided, the method comprising: obtaining a preset sample data pair set, the sample data pair set comprising at least one sample data pair, the sample data pair comprising: text and high-frequency, silent audio corresponding to the text; obtaining a diffusion model, the diffusion model comprising: an audio encoder, a text encoder, a sampler, and a decoder connected to the sampler, the sampler being respectively connected to the audio encoder and the text encoder, and the sampler and the text encoder having a residual connection; performing the following training steps: selecting a sample data pair from the sample data pair set, inputting the text in the selected sample data pair into the text encoder of the diffusion model; inputting the audio in the selected sample data pair into the audio encoder to obtain predicted audio output by the decoder; in response to the diffusion model satisfying a training completion condition, obtaining a trained audio watermark generation model.

[0008] According to a fourth aspect, a watermark generation device is provided, which includes: a to-be-processed acquisition unit, configured to acquire a to-be-processed material; a text acquisition unit, configured to process the to-be-processed material to obtain a to-be-processed text; a watermark acquisition unit, configured to input the to-be-processed text into an audio watermark generation model to obtain an audio watermark; the audio watermark generation model is used to characterize the correspondence between the input text and a high-frequency, silent audio watermark.

[0009] According to the fifth aspect, an information processing device is provided, which includes: a to-be-stored acquisition unit, configured to acquire multimodal to-be-stored materials and audio watermarks generated by a device as described in any implementation method of the fourth aspect; a vector acquisition unit, configured to obtain audio watermarks with similar meanings and retrieval vectors of various modal materials to be stored based on the to-be-stored materials and the audio watermarks; and a storage unit, configured to store the retrieval vectors into a vector retrieval library.

[0010] According to the sixth aspect, there is provided an audio watermark generation model training device, which includes: a sample acquisition unit, configured to acquire a preset sample data pair set, the sample data pair set includes at least one sample data pair, the sample data pair includes: text and high-frequency, silent audio corresponding to the text; a model acquisition unit, configured to acquire a diffusion model, the diffusion model includes: an audio encoder, a text encoder, a sampler, and a decoder connected to the sampler, the sampler is respectively connected to the audio encoder and the text encoder, and the sampler and the text encoder have a residual connection; a training unit, configured to select a sample data pair from the sample data pair set, input the text in the selected sample data pair into the text encoder of the diffusion model; input the audio in the selected sample data pair into the audio encoder to obtain the predicted audio output by the decoder; a model acquisition unit, configured to obtain a trained audio watermark generation model in response to the diffusion model satisfying a training completion condition.

[0011] According to the seventh aspect, an electronic device is provided, which includes: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any implementation of the first aspect, the second aspect, or the third aspect.

[0012] According to an eighth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect, the second aspect, or the third aspect.

[0013] According to a ninth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method described in any one of the implementations of the first aspect, the second aspect, or the third aspect.

[0014] The watermark generation method and device provided by the embodiments of the present disclosure first obtain the material to be processed; secondly, process the material to be processed to obtain the text to be processed; finally, input the text to be processed into the audio watermark generation model to obtain the audio watermark. Thus, by inputting the processed text to the audio watermark generation model to obtain the audio watermark output by the audio watermark generation model, a reliable implementation method is provided for obtaining the audio watermark, and the reliability of obtaining the audio watermark is improved.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0017] Figure 1 is a flow chart of an embodiment of a watermark generation method according to the present disclosure;

[0018] Figure 2 is a flow chart of an embodiment of an information processing method according to the present disclosure;

[0019] Figure 3 is a flow chart of an embodiment of a method for training an audio watermark generation model according to the present disclosure;

[0020] Figure 4 is a schematic structural diagram of an embodiment of a watermark generating device according to the present disclosure;

[0021] Figure 5is a schematic structural diagram of an embodiment of an information processing device according to the present disclosure;

[0022] Figure 6 is a structural schematic diagram of an embodiment of an audio watermark generation model training device according to the present disclosure;

[0023] Figure 7 It is a block diagram of an electronic device used to implement the watermark generation method, information processing method, and audio watermark generation model training method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] In this embodiment, "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0026] The present invention provides a watermark generation method, through which high-frequency audio processing is no longer required for audio, and a high-frequency, silent audio watermark can be conveniently and quickly obtained directly through the to-be-processed text converted from audio. Figure 1 A process 100 according to an embodiment of a watermark generation method of the present disclosure is shown. The watermark generation method comprises the following steps:

[0027] Step 101, obtaining the material to be processed.

[0028] In this embodiment, the material to be processed is a material that can generate text data. The material to be processed can be a type of multimodal data such as image, text, audio, video, etc. By extracting the text data of the material to be processed, text data corresponding to the text to be processed can be obtained.

[0029] Optionally, the above step 101 includes: acquiring initial materials, detecting whether the initial materials contain materials related to the text; and in response to the initial materials containing materials related to the text, taking the materials related to the text as materials to be processed.

[0030] Step 102, processing the material to be processed to obtain the text to be processed.

[0031] In this embodiment, the processing of the material to be processed refers to: converting the material to be processed into text. Based on the different modalities of the material to be processed, the processing of the material to be processed is different. When the material to be processed includes an image, the processing of the material to be processed includes: converting the image to text of the material to be processed to obtain the text to be processed; when the material to be processed includes audio, the processing of the material to be processed includes: converting the audio to text of the material to be processed to obtain the text to be processed.

[0032] In this embodiment, when the material to be processed is a text, processing the material to be processed may further include: performing word segmentation, removing stop words, removing punctuation marks, etc. on the material to be processed to obtain the text to be processed.

[0033] Step 103: input the text to be processed into the audio watermark generation model to obtain the audio watermark.

[0034] In this embodiment, the audio watermark generation model is used to characterize the correspondence between the input text and the high-frequency, silent audio watermark. The audio watermark is a high-frequency, silent audio that can be placed in the original audio to characterize the copyright of the original audio. The audio watermark can also be placed in the original audio to achieve the effect of encrypting the original audio.

[0035] In this embodiment, the audio watermark generation model is a generation model, and different types of generation models can be applied to the present disclosure to achieve the conversion of text into high-frequency, silent audio.

[0036] Optionally, when the material to be processed is audio data, the above method also includes: performing discrete wavelet transform on the audio data to obtain low-frequency area and high-frequency area information with audio information, and characterizing the high-frequency area information to obtain extracted features; performing feature extraction on the audio watermark to obtain watermark features; comparing the extracted features with the watermark features, and in response to the similarity between the extracted features and the watermark features being greater than a similarity threshold (e.g., 85%), determining that the audio watermark is generated accurately.

[0037] Optionally, the method further includes: obtaining the audio to be encrypted, performing high-frequency area analysis on the encrypted audio to obtain the high-frequency area of ​​the audio to be encrypted, placing the audio watermark in the high-frequency area, and obtaining the encrypted audio.

[0038] Optionally, the method further comprises: performing high-pass filtering on the generated audio watermark. High-frequency filtering can ensure the high-frequency effect of the audio watermark and ensure that the final generated audio watermark has more obvious high-frequency properties.

[0039] The watermark generation method provided by the embodiment of the present disclosure first obtains the material to be processed; secondly, processes the material to be processed to obtain the text to be processed; finally, inputs the text to be processed into the audio watermark generation model to obtain the audio watermark. Thus, by inputting the processed text to be processed into the audio watermark generation model to obtain the audio watermark output by the audio watermark generation model, a reliable implementation method is provided for obtaining the audio watermark, and the reliability of obtaining the audio watermark is improved.

[0040] In some optional implementations of this embodiment, the audio watermark generation model is a diffusion model, and the audio watermark generation model includes: an encoder, an upsampler, a downsampler, and a decoder connected in sequence.

[0041] In this optional implementation, the diffusion model learns the information attenuation caused by noise, and then uses the learned pattern to generate a high-frequency, silent audio watermark. In the process of learning the pattern, the diffusion model introduces noise information to the text data based on the learned audio noise distribution, and attempts to generate an audio watermark through denoising. Through multiple iterations over a period of time, the diffusion model learns to generate a new audio watermark each time given some noise input.

[0042] In traditional technology, diffusion models are generally used to generate images. The present disclosure adopts diffusion models to generate audio watermarks, completing the transfer of diffusion models from the image field to the audio field.

[0043] In this optional implementation, the diffusion model mainly includes a forward process and a backward process. The forward process is also a diffusion process. The diffusion process continuously adds noise to the input data. Adding noise is actually a process of building a label. The time step approaches positive infinity and finally becomes pure noise. The backward process is a process of denoising and continuous restoration, that is, the process of generating an audio watermark.

[0044] The audio watermark generation model provided by this optional implementation adopts a diffusion model to realize the extraction of audio watermarks, and based on the advantages of the diffusion model, improves the reliability of audio watermark generation.

[0045] In some optional implementations of this embodiment, the above-mentioned encoder includes: a text encoder, and the audio watermark generation model also includes: a first residual connection between the text encoder and the upsampler; and a second residual connection between the text encoder and the downsampler.

[0046] In this embodiment, the first residual connection is a non-linear change superposition of the output of the text encoder and the output of the upsampler, that is, the first residual connection directly adds the output of the text encoder to the output of the upsampler, thereby providing a path to bypass the non-linear change of the upsampler; the second residual connection is a non-threaded change superposition of the output of the text encoder and the output of the downsampler, that is, the second residual connection directly adds the output of the text encoder to the output of the downsampler, thereby providing a path to bypass the non-linear change of the downsampler.

[0047] In this embodiment, the first residual connection and the second residual connection can be used to perform a robust operation on the consistency between text and audio in the audio watermark generation model during the training process. The first residual connection and the second residual connection can also solve the problem of gradient vanishing and gradient exploding in the audio watermark model during the training process, and can also help the audio watermark generation model to converge faster, thereby ensuring the training effect of the audio watermark generation model.

[0048] In some optional implementations of this embodiment, the audio watermark generation model further includes: a denoising module, which is used to remove noise from the audio watermark.

[0049] In this embodiment, the denoising module is a denoising autoencoder, which can effectively remove the noise of the audio generated by the model, thereby improving the purity of the audio watermark.

[0050] The audio watermark generation model provided by this optional implementation method adds a denoising module to the audio watermark generation model, which can effectively remove the noise of the audio watermark and improve the effect of the generated audio watermark.

[0051] In some optional implementations of this embodiment, the audio watermark generation model is a generative adversarial network.

[0052] In the process of training the generative adversarial network, the generator of the generative adversarial network first uses random text vectors to generate high-frequency, silent audio as the input of the discriminator of the generative adversarial network. The training discriminator of the generative adversarial network uses the label data corresponding to the text vector and the data of the generator of the generative adversarial network to judge whether it is correct, until the generator of the generative adversarial network and the discriminator of the generative adversarial network reach a balance. Therefore, a generative adversarial network suitable for extracting audio watermarks is obtained.

[0053] The audio watermark generation model provided in this embodiment adopts the generation and confrontation principle of the generative adversarial network to realize the generation of the audio watermark, thereby improving the accuracy of the generation of the audio watermark.

[0054] Optionally, the above audio watermark generation model may also adopt a VAE (Variational Autoencoder) model.

[0055] In view of the above audio watermark generation method, the present disclosure also provides an information processing method. Figure 2 A process 200 according to an embodiment of the information processing method of the present disclosure is shown. The information processing method includes the following steps:

[0056] Step 201, obtaining multi-modal materials to be stored and audio watermarks.

[0057] In this embodiment, the audio watermark may be an audio watermark generated by the above-mentioned audio generation method.

[0058] In this embodiment, the multimodal materials to be stored include materials of at least one modality, and each material is material related to the audio watermark. For example, the multimodal materials to be stored are audio data and text data, wherein the audio data is data that interprets the subject to which the audio watermark belongs by audio means, and the text data is data that interprets the subject to which the audio watermark belongs by text means.

[0059] Optionally, the above step 201 includes: obtaining multi-modal initial materials and interpretation information of audio watermarks; detecting whether the initial materials of each modality are related; if the initial materials of all modalities are related, based on the interpretation information, detecting in turn whether the initial materials of each modality are related to the audio watermark; if at least one modality of initial material is related to the audio watermark, the initial material related to the audio watermark is used as material to be stored in the database.

[0060] Step 202, based on the material to be stored and the audio watermark, obtain the search vectors of the audio watermark with similar meaning and the material to be stored in various modalities.

[0061] In this embodiment, the above step 202 includes: inputting the material to be stored and the audio watermark into the embedded model, and obtaining the audio watermark with the same meaning and the search vectors of the materials to be stored in various modalities output by the embedded model. The audio watermark with the same meaning and the search vectors of the materials to be stored in various modalities mean that the search vector of the audio watermark has the same meaning as the search vector of the materials to be stored in various modalities, and has a corresponding relationship. The generated search vector can unify the consistency between the audio watermark and the various modal information.

[0062] Optionally, the above step 202 includes: extracting features of the material to be stored and the audio watermark at the same time, screening the extracted features, and obtaining features with the same meaning for the material to be stored and the audio watermark; based on the features with the same meaning, performing vector transformation on the material to be stored and the audio watermark, and obtaining retrieval vectors of audio watermarks with similar meanings and various modal materials to be stored. In this embodiment, the above features with the same meaning, performing vector transformation on the material to be stored and the audio watermark, and obtaining retrieval vectors of audio watermarks with similar meanings and various modal materials to be stored include: respectively determining unique features whose similarity between the features of the material to be stored and the audio watermark and the features with the same meaning is greater than a similarity threshold, restoring the material to be stored based on the unique features of the material to be stored, obtaining the restored material, performing vector transformation on the restored material, and obtaining the retrieval vector of the material to be stored; restoring the audio watermark based on the unique features of the audio watermark, obtaining the restored watermark, and performing vector transformation on the restored watermark, and obtaining the retrieval vector of the audio watermark.

[0063] Step 203: store the search vector into a vector search library.

[0064] In this embodiment, the vector retrieval library is a database that stores retrieval vectors of audio watermarks with similar meanings and various modal materials to be stored. When searching for a retrieval vector of any modal information in the vector retrieval library, the retrieval vector of the audio watermark with the same meaning can be found accordingly.

[0065] In this embodiment, the vector retrieval library includes: retrieval vectors of audio watermarks with similar meanings stored in different areas, and retrieval vectors of various modal materials to be stored.

[0066] Optionally, the vector retrieval library may also include: materials to be stored and audio watermarks stored corresponding to the retrieval vectors of audio watermarks with similar meanings and various modal materials to be stored. In this embodiment, the materials to be stored and audio watermarks corresponding to the retrieval vectors of audio watermarks with similar meanings and various modal materials to be stored are stored in the vector retrieval library. After the vectors are matched, the audio watermark and the materials to be stored related to the audio watermark can be conveniently and quickly determined, thereby improving the interpretability of the audio watermark.

[0067] The information processing method provided in this embodiment first obtains multimodal materials to be stored and audio watermarks; secondly, based on the materials to be stored and the audio watermarks, retrieval vectors of audio watermarks with similar meanings and materials to be stored in various modalities are obtained; finally, the retrieval vectors of audio watermarks with similar meanings and materials to be stored in various modalities are stored in a vector retrieval library, which can facilitate the retrieval of information related to the retrieval vector of the audio watermark and improve the interpretability of the audio watermark.

[0068] In some optional implementations of the present disclosure, the above-mentioned obtaining of audio watermarks with similar meanings and retrieval vectors of various modal materials to be stored in the library based on the materials to be stored and the audio watermarks includes: obtaining a cross-modal macro model corresponding to the modality of the materials to be stored and the audio watermarks, wherein the cross-modal macro model is used to characterize the correspondence between different modal information with similar meanings and retrieval vectors of various modal information; inputting the materials to be stored and the audio watermarks into the cross-modal macro model at the same time to obtain the audio watermarks with similar meanings and retrieval vectors of various modal materials to be stored in the library.

[0069] In this embodiment, the cross-modal big model is a model that can perform vectorized transformation on multiple modal data (such as images, text, audio, and video). When the cross-modal big model is a model that can process audio data, the cross-modal big model is a model corresponding to the audio watermark. Furthermore, when the cross-modal big model can still process the information of the model corresponding to the material to be stored, the cross-modal big model is a cross-modal big model corresponding to the modality of the material to be stored. For example, the material to be stored includes: text and images, and the cross-modal big model is a model that processes text, images, and audio. Then the cross-modal big model is a cross-modal big model corresponding to the modality and audio watermark of the material to be stored.

[0070] In this embodiment, the cross-modal big model can be, for example, an ImageBind big model, which can integrate six types of data, including vision (presented in the form of images and videos), heat (infrared images), text, audio, depth information, and motion readings generated by IMU (inertial measurement unit).

[0071] The method provided in this embodiment for obtaining audio watermarks with similar meanings and retrieval vectors of various modal materials to be stored provides retrieval vectors with similar meanings in multiple modal information through a cross-modal large model, thereby improving the efficiency of generating retrieval vectors.

[0072] In some embodiments of the present disclosure, the above-mentioned information processing method also includes: obtaining encrypted encrypted audio; decrypting the encrypted audio to obtain the watermark to be processed; performing vector processing on the watermark to be processed to obtain an audio vector; based on the audio vector and the vector retrieval library, obtaining a retrieval vector corresponding to the audio vector; based on the retrieval vector, obtaining a multimodal material corresponding to the watermark to be processed.

[0073] In this embodiment, the encrypted audio is obtained by adding the watermark to be processed to ordinary audio. The above-mentioned decryption of the encrypted audio to obtain the watermark to be processed includes: performing discrete wavelet transform on the encrypted audio to obtain high-frequency area information of the audio, performing feature extraction on the high-frequency area information to obtain features of the watermark to be processed, and using the audio corresponding to the features of the watermark to be processed in the high-frequency area information as the watermark to be processed.

[0074] In this embodiment, based on the audio vector and the vector retrieval library, obtaining a retrieval vector corresponding to the audio vector includes: calculating the similarity between the audio vector and the retrieval vector in the vector retrieval library; in response to the similarity between the audio vector and the retrieval vector being greater than a similarity threshold (for example, 89%), determining that the retrieval vector is the retrieval vector corresponding to the audio vector.

[0075] Optionally, the method further includes: in response to a situation where the similarity between the audio vector and any search vector in the vector search library is less than a similarity threshold, adding the audio vector to the vector search library.

[0076] In this embodiment, each search vector in the vector search library can correspond to a corresponding material and a watermark to be processed. After the search vector corresponding to the audio vector is obtained, the material and the watermark to be processed corresponding to the search vector can be found accordingly, and the material corresponding to the search vector is the multimodal material corresponding to the watermark to be processed.

[0077] In this embodiment, the multimodal material corresponding to the watermark to be processed is material related to the watermark to be processed in the encrypted audio. The watermark to be processed in the encrypted audio can be effectively interpreted through the multimodal material corresponding to the watermark to be processed, providing reliable technical support for the interpretability of the watermark to be processed.

[0078] The information processing method provided in this embodiment decrypts the encrypted audio to obtain the watermark to be processed; performs vector processing on the watermark to be processed to obtain an audio vector, retrieves the retrieval vector related to the audio vector through a retrieval library, and further finds multimodal materials related to the retrieval vector, thereby providing effective technical support for the interpretability of the watermark to be processed.

[0079] In some optional implementations of the present embodiment, the above-mentioned vector processing of the watermark to be processed to obtain the audio vector includes: inputting the watermark to be processed into a cross-modal large model to obtain the audio vector output by the cross-modal large model.

[0080] In this optional implementation, the cross-modal big model can be a model of the retrieval vector of the generated vector retrieval library. By inputting the watermark to be processed into the cross-modal big model that is the same as the retrieval vector used to generate the retrieval vector library, the uniformity of watermark processing can be ensured, thereby improving the processing efficiency of the watermark to be processed.

[0081] In some optional implementations of the present embodiment, the above-mentioned obtaining a retrieval vector corresponding to the audio vector based on the audio vector and the vector retrieval library includes: matching the audio vector with a retrieval vector in the vector retrieval library; in response to a successful match between the audio vector and the retrieval vector in the vector retrieval library, obtaining all retrieval vectors related to the retrieval vector; and using the retrieval vector and all retrieval vectors related to the retrieval vector as the retrieval vector corresponding to the audio vector.

[0082] In this embodiment, the vector retrieval library can store information based on its meaning, and multimodal retrieval vectors with the same meaning have the same meaning label. Matching the audio vector with the retrieval vector includes: comparing the audio vector with the retrieval vectors under each meaning label for similarity; in response to the similarity between the audio vector and the retrieval vector under any one or more meaning labels being greater than a similarity threshold, comparing the audio vector with the retrieval vector corresponding to the audio watermark under the meaning label for similarity; if the similarity is greater than the similarity threshold, determining that the audio vector matches the retrieval vector corresponding to the audio watermark under the meaning label successfully.

[0083] The method for obtaining a retrieval vector corresponding to an audio vector provided by this optional implementation improves the reliability of audio vector retrieval by matching the audio vector with the retrieval vector to determine the retrieval vector corresponding to the audio vector.

[0084] In view of the above watermark generation method, the present disclosure also provides a training method for an audio watermark generation model. Figure 3 A process 300 of an embodiment of a method for training an audio watermark generation model according to the present disclosure is shown. The method for training an audio watermark generation model comprises the following steps:

[0085] Step 301: Obtain a preset sample data pair set.

[0086] In this embodiment, the sample data pair set includes at least one sample data pair, and the sample data pair includes: text and high-frequency, silent audio corresponding to the text.

[0087] In this embodiment, in the early stage of training the audio watermark generation model, a large number of data pairs of text and high-frequency, silent audio need to be prepared in advance, and the data pairs are sample data pairs in the sample data pair set. The text in the sample data pair and the high-frequency, silent audio corresponding to the text may have a certain relationship.

[0088] Optionally, the generation process of the sample data pair set is as follows: obtain a sample speech segment, divide the sample speech segment into multiple sub-speech segments, perform discrete wavelet transform on each sub-speech segment to obtain high-frequency area information, and use the silent audio with a specific frequency in the high-frequency area information as the sample audio; perform audio-to-text transformation on each sub-speech segment to obtain sample text, and use the sample audio and sample text as the sample data pair in the sample data pair set.

[0089] Step 302: Obtain a diffusion model.

[0090] In this embodiment, the diffusion model includes: an audio encoder, a text encoder, a sampler, and a decoder connected to the sampler. The sampler is connected to the audio encoder and the text encoder respectively, and the sampler and the text encoder have a residual connection.

[0091] In this embodiment, the audio encoder and the text encoder in the diffusion model are distributed in parallel, and each is used to encode the audio and text to obtain audio encoding and text encoding; the audio encoding and the text encoding are input into the sampler together. Since the sampler and the text encoder have a residual connection, the text information from the text encoder can be transmitted to the sampler, so that the sampler can understand the input text information more clearly, thereby improving the robustness of text information processing during training.

[0092] Optionally, the encoder includes: a text encoder, the sampler includes: an upsampler and a downsampler, and the residual connection includes: a first residual connection between the text encoder and the upsampler; and a second residual connection between the text encoder and the downsampler. In this embodiment, the first residual connection and the second residual connection are used, so that both the upsampler and the downsampler can obtain text information, thereby improving the robustness of the training of the entire diffusion model.

[0093] Step 303: Select a sample data pair from the sample data pair set, and input the text in the selected sample data pair into the text encoder of the diffusion model.

[0094] In this embodiment, the execution subject may select a sample data pair from the sample data pair set obtained in step 301, and perform the training steps of steps 303 to 305, wherein the method and number of sample selection are not limited in the present disclosure. For example, at least one sample data pair may be randomly selected, or a sample data pair with clear textual expression may be selected from the sample data pair set, each sample data pair is a text and audio pair, wherein the audio in the text and audio pair is a high-frequency, silent audio.

[0095] In this embodiment, the text in the selected sample data pair is input into a text encoder, so that the text in the sample data pair can be converted into a meaning that can be understood by the diffusion model, so that the diffusion model can refer to the information of the text to predict the speech.

[0096] Step 304: input the audio in the selected sample data pair into an audio encoder to obtain the predicted audio output by the decoder.

[0097] In this embodiment, the audio in the selected sample data pair is input into an audio encoder, which can convert the audio in the sample data pair into a meaning that can be understood by the diffusion model, so that the forward process, reverse process and loss function calculation are performed in the diffusion model to obtain the predicted speech.

[0098] Step 305: In response to the diffusion model satisfying the training completion condition, a trained audio watermark generation model is obtained.

[0099] In this embodiment, the training completion conditions are at least one of the following: the number of training iterations of the diffusion model reaches an iteration threshold, and the loss value calculated by the loss function of the diffusion model is less than a predetermined loss value threshold, wherein the iteration threshold and the predetermined loss value threshold can be determined based on the training requirements of the diffusion model.

[0100] In this embodiment, the diffusion model can use a contrastive loss function to perform comparative calculations. The contrastive loss function can enable the model to determine in real time whether the predicted audio is similar to the audio of the input sample, thereby ultimately obtaining the final loss value.

[0101] Optionally, when the diffusion model does not meet the training completion condition, relevant parameters in the diffusion model are adjusted, and steps 303 to 305 are performed again, so that the loss value of the diffusion model converges until the diffusion model meets the training completion condition.

[0102] In the audio watermark generation model training method provided in this embodiment, during the training process of the audio watermark generation model, each text data and audio data in the sample data pair will be respectively passed through an encoder to generate a corresponding vector, and the diffusion model is used for model training, and the contrast loss function is used for contrast training, thereby improving the accuracy of the generated audio watermark generation model. Furthermore, a residual connection is set between the sampler and the text encoder, so that the sampler can obtain the information of the input text in real time, thereby improving the consistency of the text and audio and the robustness of various information during the training process.

[0103] Further references Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a watermark generation device. Figure 1Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0104] like Figure 4 As shown, the watermark generation device 400 provided in this embodiment includes: a to-be-processed acquisition unit 401, a text acquisition unit 402, and a watermark acquisition unit 403. Among them, the to-be-processed acquisition unit 401 can be configured to acquire the to-be-processed material. The text acquisition unit 402 can be configured to process the to-be-processed material to obtain the to-be-processed text. The watermark acquisition unit 403 can be configured to input the to-be-processed text into an audio watermark generation model to obtain an audio watermark; the audio watermark generation model is used to characterize the correspondence between the input text and the high-frequency, silent audio watermark.

[0105] In this embodiment, in the watermark generating device 400, the specific processing of the to-be-processed obtaining unit 401, the text obtaining unit 402, and the watermark obtaining unit 403 and the technical effects thereof can be referred to in Figure 1 The relevant descriptions of step 101, step 102, and step 103 in the corresponding embodiment are not repeated here.

[0106] In some optional implementations of this embodiment, the audio watermark generation model is a diffusion model, and the audio watermark generation model includes: an encoder, an upsampler, a downsampler, and a decoder connected in sequence.

[0107] In some optional implementations of this embodiment, the above-mentioned encoder includes: a text encoder, and the audio watermark generation model also includes: a first residual connection between the text encoder and the upsampler; and a second residual connection between the text encoder and the downsampler.

[0108] In some optional implementations of this embodiment, the audio watermark generation model further includes: a denoising module, which is used to remove noise from the audio watermark.

[0109] In some optional implementations of the present disclosure, the above-mentioned audio watermark generation model is a generative adversarial network.

[0110] The watermark generation device provided by the embodiment of the present disclosure, first, the to-be-processed acquisition unit 401 acquires the to-be-processed material; secondly, the text acquisition unit 402 processes the to-be-processed material to obtain the to-be-processed text; finally, the watermark acquisition unit 403 inputs the to-be-processed text into the audio watermark generation model to obtain the audio watermark. Thus, by inputting the to-be-processed text obtained after the processing into the audio watermark generation model, the audio watermark output by the audio watermark generation model is obtained, which provides a reliable implementation method for obtaining the audio watermark and improves the reliability of obtaining the audio watermark.

[0111] Continue to see Figure 5 , as a response to the above Figure 2 To implement the method shown in the figure, the present application provides an embodiment of an information processing device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0112] like Figure 5 As shown, the information processing device 500 provided in this embodiment includes: a to-be-stored acquisition unit 501, a vector acquisition unit 502, and a storage unit 503. Among them, the to-be-stored acquisition unit 501 can be configured to acquire multi-modal to-be-stored materials and audio watermarks generated by a watermark generation device. The vector acquisition unit 502 can be configured to obtain audio watermarks with similar meanings and retrieval vectors of various modal to-be-stored materials based on the to-be-stored materials and audio watermarks. The storage unit 503 can be configured to store the retrieval vector in a vector retrieval library.

[0113] In this embodiment, in the information processing device 500, the specific processing of the storage acquisition unit 501, the vector acquisition unit 502, and the storage unit 503 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of step 201, step 202, and step 203 in the corresponding embodiment are not repeated here.

[0114] In some optional implementations of the present embodiment, the vector obtaining unit 502 is further configured to: obtain a cross-modal big model corresponding to the modality and audio watermark of the material to be stored, wherein the cross-modal big model is used to characterize the correspondence between different modal information with similar meanings and the retrieval vectors of various modal information; input the material to be stored and the audio watermark into the cross-modal big model at the same time to obtain a cross-modal big model with audio watermarks with similar meanings and retrieval vectors of various modal materials to be stored.

[0115] In some optional implementations of the present embodiment, the above-mentioned device 500 also includes: an audio acquisition unit (not shown in the figure), a decryption unit (not shown in the figure), a processing unit (not shown in the figure), a retrieval unit (not shown in the figure), and a material acquisition unit (not shown in the figure). The above-mentioned audio acquisition unit can be configured to acquire encrypted audio. The above-mentioned decryption unit can be configured to decrypt the encrypted audio to obtain the watermark to be processed. The above-mentioned processing unit can be configured to perform vector processing on the watermark to be processed to obtain an audio vector. The above-mentioned retrieval unit can be configured to obtain a retrieval vector corresponding to the audio vector based on the audio vector and the vector retrieval library. The above-mentioned material acquisition unit is configured to obtain multimodal material corresponding to the watermark to be processed based on the retrieval vector.

[0116] In some optional implementations of this embodiment, the processing unit is configured to: input the watermark to be processed into the cross-modal large model to obtain the audio vector output by the cross-modal large model.

[0117] In some optional implementations of the present embodiment, the retrieval unit is further configured to: match the audio vector with a retrieval vector in a vector retrieval library; in response to a successful match between the audio vector and the retrieval vector in the vector retrieval library, obtain all retrieval vectors related to the retrieval vector; and use the retrieval vector and all retrieval vectors related to the retrieval vector as the retrieval vector corresponding to the audio vector.

[0118] The information processing device provided by the embodiment of the present disclosure firstly obtains the multimodal materials to be stored and audio watermarks by the to-be-stored acquisition unit 501; secondly, the vector acquisition unit 502 obtains the audio watermarks with similar meanings and the retrieval vectors of the materials to be stored in various modalities based on the materials to be stored and the audio watermarks; finally, the storage unit 503 stores the audio watermarks with similar meanings and the retrieval vectors of the materials to be stored in various modalities into the vector retrieval library, which can facilitate the retrieval of the information related to the retrieval vector of the audio watermark and improve the interpretability of the audio watermark.

[0119] Continue to see Figure 6 , as a response to the above Figure 3 The present application provides an embodiment of an audio watermark generation model training device. Figure 3 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0120] like Figure 6 As shown, the audio watermark generation model training device 600 provided in this embodiment includes: a sample acquisition unit 601, a model acquisition unit 602, a training unit 603, and a model acquisition unit 604. The sample acquisition unit 601 can be configured to acquire a preset sample data pair set, the sample data pair set includes at least one sample data pair, and the sample data pair includes: text and high-frequency, silent audio corresponding to the text. The model acquisition unit 602 can be configured to acquire a diffusion model, the diffusion model includes: an audio encoder, a text encoder, a sampler, and a decoder connected to the sampler, the sampler is respectively connected to the audio encoder and the text encoder, and the sampler and the text encoder have a residual connection. The training unit 603 can be configured to select a sample data pair from the sample data pair set, input the text in the selected sample data pair into the text encoder of the diffusion model; input the audio in the selected sample data pair into the audio encoder, and obtain the predicted audio output by the decoder. The model acquisition unit 604 can be configured to obtain a trained audio watermark generation model in response to the diffusion model meeting the training completion condition.

[0121] In this embodiment, in the audio watermark generation model training device 600, the specific processing of the sample acquisition unit 601, the model acquisition unit 602, the training unit 603, and the model acquisition unit 604 and the technical effects thereof can be referred to respectively. Figure 3 The relevant descriptions of step 301, step 302, step 303, and step 304 in the corresponding embodiment are not repeated here.

[0122] The audio watermark generation model training device provided by the embodiment of the present disclosure generates a corresponding vector for each text data and audio data in the sample data pair through an encoder during the training process of the audio watermark generation model, and uses a diffusion model for model training, and uses a contrast loss function for contrast training, thereby improving the accuracy of the generated audio watermark generation model. Furthermore, a residual connection is set between the sampler and the text encoder, so that the sampler can obtain the information of the input text in real time, thereby improving the consistency of the text and audio and the robustness of various information during the training process.

[0123] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0124] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0125] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0126] like Figure 7As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0127] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0128] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a watermark generation method or an information processing method or an audio watermark generation model training method. For example, in some embodiments, the watermark generation method or the information processing method or the audio watermark generation model training method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the watermark generation method or the information processing method or the audio watermark generation model training method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the watermark generation method or the information processing method or the audio watermark generation model training method in any other suitable manner (eg, by means of firmware).

[0129] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0130] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable watermark generation device or information processing device or audio watermark generation model training device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0131] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0132] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0133] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0134] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0135] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0136] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A watermark generation method, the method include: Get the material to be processed; Processing the material to be processed to obtain a text to be processed; The text to be processed is input into an audio watermark generation model to obtain an audio watermark; the audio watermark generation model is used to characterize the corresponding relationship between the input text and a high-frequency, silent audio watermark.

2. The method according to claim 1, in, The audio watermark generation model is a diffusion model, and the audio watermark generation model includes: an encoder, an upsampler, a downsampler, and a decoder connected in sequence.

3. The method according to claim 2, in, The encoder comprises: a text encoder, and the audio watermark generation model further comprises: a first residual connection between the text encoder and the upsampler; and a second residual connection between the text encoder and the downsampler.

4. The method according to claim 3, in, The audio watermark generation model further includes: a denoising module, which is used to remove the noise of the audio watermark.

5. The method according to claim 1, in, The audio watermark generation model is a generative adversarial network.

6. An information processing method, the method include: Obtain multimodal materials to be stored and the audio watermark generated by the method according to any one of claims 1 to 5; Based on the material to be stored and the audio watermark, obtaining a search vector of the audio watermark with similar meaning and various modal materials to be stored; The search vector is stored in a vector search library.

7. The method according to claim 6, in, The step of obtaining the audio watermark and the retrieval vectors of various modal materials to be stored in the library with similar meanings based on the materials to be stored and the audio watermark includes: Acquire a cross-modal macromodel corresponding to the modality of the material to be stored and the audio watermark, wherein the cross-modal macromodel is used to characterize the correspondence between different modal information with similar meanings and retrieval vectors of various modal information; The material to be stored and the audio watermark are simultaneously input into the cross-modal large model to obtain retrieval vectors of audio watermarks with similar meanings and materials to be stored in various modalities.

8. The method according to claim 6, wherein include: Get the encrypted audio; Decrypting the encrypted audio to obtain a watermark to be processed; Performing vector processing on the watermark to be processed to obtain an audio vector; Based on the audio vector and the vector retrieval library, obtaining a retrieval vector corresponding to the audio vector; Based on the retrieval vector, a multimodal material corresponding to the watermark to be processed is obtained.

9. The method according to claim 8, in, The performing vector processing on the watermark to be processed to obtain an audio vector comprises: The watermark to be processed is input into a cross-modal large model to obtain an audio vector output by the cross-modal large model, wherein the cross-modal large model is a model for generating a retrieval vector of the vector retrieval library.

10. The method according to claim 8, in, The obtaining of a search vector corresponding to the audio vector based on the audio vector and the vector search library comprises: Matching the audio vector with a search vector in the vector search library; In response to the audio vector successfully matching the search vector in the vector search library, obtaining all search vectors related to the search vector; The retrieval vector and all retrieval vectors related to the retrieval vector are used as retrieval vectors corresponding to the audio vector.

11. A method for training an audio watermark generation model, the method include: Acquire a preset sample data pair set, the sample data pair set comprising at least one sample data pair, the sample data pair comprising: a text and a high-frequency, silent audio corresponding to the text; Acquire a diffusion model, the diffusion model comprising: an audio encoder, a text encoder, a sampler, and a decoder connected to the sampler, the sampler being connected to the audio encoder and the text encoder respectively, and the sampler and the text encoder having a residual connection; Perform the following training steps: Selecting a sample data pair from the sample data pair set, inputting the text in the selected sample data pair into a text encoder of a diffusion model; inputting the audio in the selected sample data pair into the audio encoder, and obtaining the predicted audio output by the decoder; In response to the diffusion model satisfying the training completion condition, a trained audio watermark generation model is obtained, and the audio watermark generation model is used in the method according to any one of claims 1-5.

12. A watermark generating device, the device include: A to-be-processed acquisition unit, configured to acquire the to-be-processed material; A text obtaining unit is configured to process the material to be processed to obtain a text to be processed; The watermark obtaining unit is configured to input the text to be processed into an audio watermark generation model to obtain an audio watermark; the audio watermark generation model is used to characterize the corresponding relationship between the input text and a high-frequency, silent audio watermark.

13. The device according to claim 12, in, The audio watermark generation model is a diffusion model, and the audio watermark generation model includes: an encoder, an upsampler, a downsampler, and a decoder connected in sequence.

14. The device according to claim 13, in, The encoder comprises: a text encoder, and the audio watermark generation model further comprises: a first residual connection between the text encoder and the upsampler; and a second residual connection between the text encoder and the downsampler.

15. The device according to claim 14, in, The audio watermark generation model further includes: a denoising module, which is used to remove the noise of the audio watermark.

16. The device according to claim 12, in, The audio watermark generation model is a generative adversarial network.

17. An information processing device, the device include: A to-be-stored acquisition unit, configured to acquire multimodal to-be-stored materials and an audio watermark generated by the device according to any one of claims 12 to 16; A vector obtaining unit is configured to obtain, based on the material to be stored and the audio watermark, a search vector of the audio watermark with similar meaning and various modal materials to be stored; The storage unit is configured to store the search vector into a vector search library.

18. The device according to claim 17, in, The vector obtaining unit is further configured to: obtain a cross-modal large model corresponding to the modality of the material to be stored and the audio watermark, wherein the cross-modal large model is used to characterize the correspondence between different modal information with similar meanings and retrieval vectors of various modal information; input the material to be stored and the audio watermark into the cross-modal large model at the same time to obtain a cross-modal large model of retrieval vectors of audio watermarks with similar meanings and various modal materials to be stored.

19. The device according to claim 17, further comprising: include: An audio acquisition unit, configured to acquire the encrypted audio; A decryption unit, configured to decrypt the encrypted audio to obtain a watermark to be processed; A processing unit, configured to perform vector processing on the watermark to be processed to obtain an audio vector; A retrieval unit configured to obtain a retrieval vector corresponding to the audio vector based on the audio vector and the vector retrieval library; The material obtaining unit is configured to obtain the multimodal material corresponding to the watermark to be processed based on the retrieval vector.

20. The device according to claim 19, in, The processing unit is configured to: input the watermark to be processed into a cross-modal large model to obtain an audio vector output by the cross-modal large model, wherein the cross-modal large model is a model for generating a retrieval vector of the vector retrieval library.

21. The device according to claim 19, in, The retrieval unit is further configured to: match the audio vector with a retrieval vector in the vector retrieval library; in response to a successful match between the audio vector and the retrieval vector in the vector retrieval library, obtain all retrieval vectors related to the retrieval vector; and use the retrieval vector and all retrieval vectors related to the retrieval vector as the retrieval vector corresponding to the audio vector.

22. An audio watermark generation model training device, the device include: A sample acquisition unit is configured to acquire a preset sample data pair set, wherein the sample data pair set includes at least one sample data pair, wherein the sample data pair includes: a text and a high-frequency, silent audio corresponding to the text; A model acquisition unit is configured to acquire a diffusion model, wherein the diffusion model includes: an audio encoder, a text encoder, a sampler, and a decoder connected to the sampler, wherein the sampler is connected to the audio encoder and the text encoder respectively, and the sampler and the text encoder have a residual connection; A training unit is configured to select a sample data pair from the sample data pair set, input the text in the selected sample data pair into a text encoder of the diffusion model; input the audio in the selected sample data pair into the audio encoder, and obtain the predicted audio output by the decoder; The model obtaining unit is configured to obtain a trained audio watermark generation model in response to the diffusion model satisfying a training completion condition, wherein the audio watermark generation model is used in the method described in any one of claims 1-5.

23. An electronic device, It is characterized in that include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, It is characterized in that The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 11.

25. A computer program product, comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 11.

Images