Audio processing method and device and related equipment

Abstract

The invention discloses an audio processing method and device and related equipment, and belongs to the technical field of audio processing. The audio processing method provided by the embodiment of the invention comprises the steps that N first spectrum features and N watermark features which are in one-to-one correspondence are acquired, the first spectrum features are spectrum features of audio segments in audio, the watermark features are features of watermark information, and N is a positive integer; inputting the N first frequency spectrum features and the N watermark features into a watermark processing model, and embedding each watermark feature into the corresponding first frequency spectrum feature through a generator of the watermark processing model to obtain N second frequency spectrum features; and generating a watermark audio based on the N second spectrum features and the audio.

Description

Technical Field

[0001] The present application belongs to the field of audio processing technology, and specifically relates to an audio processing method, device and related equipment. Background Art

[0002] At present, electronic devices can use sound replication technology to determine the timbre, rhythm and style of a speaker based on a small amount of real audio of the speaker, and use the timbre, rhythm and style of the speaker to perform audio synthesis to obtain a replica audio that matches the timbre, rhythm and style of the speaker. Usually, in order to avoid the abuse of sound replication technology, after obtaining the replica audio, the electronic device can perform a watermark embedding operation to add audio watermark information to the replica audio, so that after the electronic device transmits the replica audio with the audio watermark information to other electronic devices, the other electronic devices can determine that the replica audio is not real audio when detecting the audio watermark information.

[0003] However, during the transmission of the reproduced audio with the audio watermark information added, the reproduced audio may be subjected to signal processing, such as filtering, resampling, etc. This may cause the audio watermark information to be interfered with or destroyed. Therefore, the reliability of the electronic device in performing the watermark embedding operation is low. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an audio processing method, apparatus and related equipment, which can improve the reliability of watermark embedding operations performed by electronic devices.

[0005] In a first aspect, an embodiment of the present application provides an audio processing method, the method comprising: obtaining N first spectral features and N watermark features corresponding to each other, the first spectral features being spectral features of an audio segment in the audio, the watermark features being features of watermark information, and N being a positive integer; inputting the N first spectral features and the N watermark features into a watermark processing model, and embedding each watermark feature into the corresponding first spectral feature through a generator of the watermark processing model to obtain N second spectral features; and generating watermarked audio based on the N second spectral features and the audio.

[0006] In a second aspect, an embodiment of the present application provides an audio processing device, which includes: an acquisition module and a processing module. The acquisition module is used to acquire N first spectral features and N watermark features that correspond to each other, wherein the first spectral features are spectral features of an audio segment in the audio, and the watermark features are features of watermark information, and N is a positive integer. The processing module is used to input the N first spectral features and N watermark features acquired by the acquisition module into a watermark processing model, and embed each watermark feature into the corresponding first spectral feature through a generator of the watermark processing model to obtain N second spectral features; and generate watermarked audio based on the N second spectral features and the audio.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0011] In an embodiment of the present application, the electronic device can obtain N first spectral features and N watermark features that correspond to each other, each first spectral feature is a spectral feature of an audio segment in the audio, and each watermark feature is a feature of the watermark information, and the N first spectral features and the N watermark features are input into the watermark processing model, and each watermark feature is embedded into the corresponding first spectral feature through the generator of the watermark processing model to obtain N second spectral features, so that the electronic device can generate watermark audio based on the N second spectral features and the audio. Since the electronic device can obtain N first spectral features of N audio segments in the audio and N watermark features of N watermark information, and embed each watermark feature into the corresponding first spectral feature through the generator of the trained watermark processing model to obtain N second spectral features, instead of directly adding N watermark information to N audio segments, therefore, on the one hand, because the first spectral features and the watermark features are not easily interfered or destroyed due to signal processing, even if the obtained watermark audio is subjected to signal processing, the watermark features will not be interfered or destroyed. On the other hand, because the watermark processing model is a trained model, after N first spectral features and N watermark features are input into the watermark processing model, the generator of the watermark processing model can accurately embed each watermark feature into the corresponding first spectral feature, thereby improving the reliability of the watermarked audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is one of the flowcharts of the audio processing method provided in the embodiment of the present application;

[0013] Figure 2 This is the second flowchart of the audio processing method provided in the embodiment of the present application;

[0014] Figure 3 is a schematic diagram of the amount of audio data in the audio processing method provided in an embodiment of the present application;

[0015] Figure 4 This is the third flowchart of the audio processing method provided in the embodiment of the present application;

[0016] Figure 5 This is a fourth flowchart of the audio processing method provided in an embodiment of the present application;

[0017] Figure 6 is a schematic diagram of a reversible block in the audio processing method provided in an embodiment of the present application;

[0018] Figure 7 This is the fifth flowchart of the audio processing method provided in the embodiment of the present application;

[0019] Figure 8This is the sixth flowchart of the audio processing method provided in the embodiment of the present application;

[0020] Fig. 9 This is the seventh flowchart of the audio processing method provided in the embodiment of the present application;

[0021] Fig.10 This is the eighth flowchart of the audio processing method provided in the embodiment of the present application;

[0022] Fig.11 is a schematic diagram of the amount of audio data in the audio processing method provided in an embodiment of the present application;

[0023] Fig.12 This is a ninth flowchart of the audio processing method provided in an embodiment of the present application;

[0024] Fig.13 is a schematic diagram of a reversible block in the audio processing method provided in an embodiment of the present application;

[0025] Fig.14 This is a tenth flowchart of the audio processing method provided in an embodiment of the present application;

[0026] Fig.15 This is the eleventh flowchart of the audio processing method provided in the embodiment of the present application;

[0027] Fig.16 is a structural diagram of an audio processing device provided in an embodiment of the present application;

[0028] Fig.17 This is one of the hardware structure diagrams of the electronic device provided in the embodiment of the present application;

[0029] Fig.18 This is the second schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0031] The professional terms involved in the embodiments of the present application are explained below.

[0032] 1. Sound reproduction technology

[0033] Voice replication technology is a text-to-speech (TTS) technology that has developed rapidly in recent years. Among them, the electronic device can collect the voice of the speaker and extract the voice features of the voice. So when the voice of the speaker needs to be replicated, the electronic device can directly perform audio synthesis based on the audio and the voice features to obtain a replicated audio, the content of the replicated audio is the same as the text or audio, and the timbre, rhythm and style of the content of the replicated audio are the same as the timbre, rhythm and style of the speaker.

[0034] 2. Audio watermark information

[0035] Audio watermark information is special information hidden in the audio signal, such as copyright information or other metadata. Its purpose is to provide the copyright, source or other relevant information of the audio without affecting the listening experience.

[0036] Audio watermarking is a technology that uses redundant data and randomness in digital audio to embed audio watermark information into audio signals. These audio watermark information are imperceptible to the human ear and are not easy to remove.

[0037] 3. Generative Adversarial Networks

[0038] Generative adversarial networks mainly consist of two parts, the generator and the discriminator. The basic principle is to learn through the mutual confrontation between the two parts. The task of the generator is to generate artificial samples that are as close to the real data distribution as possible, and the goal of the discriminator is to distinguish between real data and generated data as accurately as possible.

[0039] 4. Sliding Window

[0040] A sliding window is a fixed-size or dynamically adjustable window that can slide over a data stream or data set to process the data.

[0041] 5. Data Augmentation

[0042] Data augmentation is a commonly used technique in machine learning and deep learning. It generates new training samples by performing various transformations and processing on existing training data, thereby increasing the diversity and quantity of data sets to improve the generalization ability and robustness of the model.

[0043] 6. Other terms

[0044] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0045] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0046] The audio processing method, apparatus and related equipment provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0047] The audio processing method provided in the embodiment of the present application can be executed by an audio processing device, a terminal, or a functional module or entity in the terminal. The audio processing method provided in the embodiment of the present application is described by taking an electronic device executing the audio processing method as an example.

[0048] Figure 1 FIG. 1 is a flow chart of an audio processing method provided by an embodiment of the present application. Figure 1 As shown, an audio processing method provided in an embodiment of the present application may include the following steps 101 to 103.

[0049] Step 101: The electronic device obtains N first spectrum features and N watermark features in one-to-one correspondence.

[0050] In the embodiment of the present application, each of the N first spectral features is a spectral feature of an audio segment in the audio, and N is a positive integer.

[0051] In some embodiments of the present application, when the electronic device obtains the above-mentioned audio through an audio processing application, the electronic device can directly obtain N first spectrum features and N watermark features.

[0052] In some embodiments of the present application, the above audio may be a replica audio, i.e., a replica audio generated using a sound replica technology. The replica audio may be a voice audio or a music audio, etc. Of course, the audio may also be other audio, which is not limited in the embodiments of the present application.

[0053] In some embodiments of the present application, the audio segment in the above audio can be understood as the audio segment to which a watermark is to be added in the audio.

[0054] In some embodiments of the present application, the N audio segments at least partially overlap, or the N audio segments do not overlap. The durations corresponding to the N audio segments may be at least partially the same, or the durations corresponding to the N audio segments are all different.

[0055] In a possible implementation of the present application, the N audio segments may be predetermined audio segments in the audio. For example, N predetermined time periods may be pre-stored in the electronic device, so that the electronic device may determine an audio segment of the audio in a first predetermined time period as the first audio segment, and determine an audio segment of the audio in a second predetermined time period as the second audio segment, and so on, to determine N audio segments.

[0056] In another possible implementation of the present application, the N audio segments may be audio segments determined according to a preset duration and a preset time interval. Figure 1 ,like Figure 2 As shown, before the above step 101, the audio processing method provided in the embodiment of the present application may further include the following step 201.

[0057] Step 201: The electronic device determines an audio segment from the audio at a preset time interval.

[0058] In the embodiment of the present application, the duration corresponding to an audio segment matches the preset duration.

[0059] In some embodiments of the present application, the preset duration may be any value within a first range, the lower limit of the first range may be 1 second, and the upper limit of the first range may be 1.5 seconds. The preset time interval may be any value within a second range, the lower limit of the second range may be 0.5 seconds, and the upper limit of the second range may be 1.5 seconds.

[0060] In some embodiments of the present application, the electronic device may first determine an audio start position from the audio, and then determine the position after a preset time length from the audio start position as an audio end position, and determine the audio segment between the audio start position and the audio end position as an audio segment. Then, the electronic device may first determine the position after a preset time interval from the audio end position as another audio start position, and then determine the position after a preset time length from the other audio start position as another audio end position, and determine the audio segment between the other audio start position and the other audio end position as another audio segment. And so on, until N audio segments are determined.

[0061] For example, Figure 3 FIG. 1 is a schematic diagram showing the data flow of audio. Figure 3 As shown, the electronic device can determine an audio segment whose corresponding duration matches the preset duration from the audio at every preset time interval to determine audio segment 1 and audio segment 2, wherein audio segment 1 is located in a dotted frame 10, and audio segment 2 is located in a dotted frame 11. Assuming that the preset time interval is 1s and the preset duration is 1.2s, the duration corresponding to audio segment 1 is 1s, the duration corresponding to audio segment 2 is 1s, and the interval between the audio end position of audio segment 1 and the audio start position of audio segment 2 is 1.2s.

[0062] It can be seen that since the electronic device can determine an audio segment whose duration matches the preset duration from the audio at every preset time interval, that is, the electronic device can determine the audio segment in a determined manner, that is, the audio segment with watermark information added, therefore, other electronic devices can also accurately determine the audio segment with watermark information added according to the determined manner, without the need for the electronic device to additionally indicate the location of the audio segment with watermark information added, thereby simplifying the complexity of parsing the watermark information.

[0063] In some embodiments of the present application, the first spectrum feature may include at least one of the following: a mel spectrum feature, a spectrum slope feature, a spectrum centroid feature, a spectrum smoothness feature, a spectrum sharpness feature, etc. Of course, the first spectrum feature may also include other features, which are not limited in the embodiments of the present application.

[0064] The mel spectrum feature is a two-dimensional spectrogram that contains the time domain information and frequency domain information of the audio.

[0065] In some examples, when the first spectrum feature includes a mel spectrum feature, when the electronic device determines N audio segments, the electronic device may first preprocess the N audio segments, and then perform a short-time Fourier transform operation on the processed N audio segments to obtain N short-time amplitude spectra, and finally input the N short-time amplitude spectra into a mel filter bank to obtain N mel spectrum features, the feature dimension may be [T×D], T and D are both positive integers. The above preprocessing may include at least one of the following: sampling rate conversion, pre-emphasis processing, frame processing, and windowing processing.

[0066] For example, assuming that the duration of an audio segment is 1 second, the dimension of the mel spectrum feature corresponding to the audio segment is [100×128].

[0067] In the embodiment of the present application, each of the above-mentioned N watermark features is a feature of the watermark information.

[0068] In some embodiments of the present application, the N watermark features may be features of the same watermark information. It is understandable that the N watermark features may be the same.

[0069] In some embodiments of the present application, the watermark information may be a watermark code, and the watermark code may be a binary code.

[0070] In some embodiments of the present application, the electronic device may first obtain a first identifier, determine the watermark content information based on the first identifier, and then determine the watermark information based on the watermark content information. The first identifier includes at least one of the following: the device ID of the electronic device, the user ID logged in by the above-mentioned audio processing application, and the user name logged in by the above-mentioned audio processing application. Among them, at least one first corresponding relationship is stored in the electronic device, and each first corresponding relationship is a corresponding relationship between an audio watermark code and an identifier, so that the electronic device can determine a matching identifier from at least one identifier based on the above-mentioned first identifier, and determine an audio watermark code corresponding to the identifier as the above-mentioned watermark content information.

[0071] It should be noted that the number of bits of the above audio watermark encoding can be greater than or equal to the preset number of bits, so that the number of bits of the watermark content information can also be greater than or equal to the preset number of bits, so that the watermark content information can cover more identifiers. Among them, the preset number of bits can be any integer from 32 to 40. Of course, the preset number of bits can also be other numbers, which is not limited in the embodiments of the present application.

[0072] In some examples, the electronic device may directly determine the watermark content information as watermark information.

[0073] In other examples, the electronic device may first obtain a tag information and watermark content information, and determine the watermark information according to the tag information and the watermark content information. Figure 1 ,like Figure 4 As shown, before the above step 101, the audio processing method provided in the embodiment of the present application may further include the following steps 202 and 203.

[0074] Step 202: The electronic device obtains first tag information and watermark content information.

[0075] In the embodiment of the present application, the first marking information is used to locate the watermark information.

[0076] In some embodiments of the present application, the first tag information is stored in the electronic device, so that the electronic device can directly obtain the first tag information. It can be understood that the first tag information can be preset tag information.

[0077] In some embodiments of the present application, the first tag information may be a tag code of a predetermined number of bits, and the tag code may be a binary code. The predetermined number of bits may be 8, and of course the predetermined number of bits may also be other numbers, which is not limited in the embodiments of the present application. The tag code may be an all-zero code, such as 00000000; or, it may be an all-one code, such as 11111111. Of course, the tag code may also be other fixed codes, which are not limited in the embodiments of the present application.

[0078] In some embodiments of the present application, at least one first correspondence is stored in the electronic device, and each first correspondence is a correspondence between an audio watermark code and an identifier, so that the electronic device can first obtain the first identifier, and determine a matching identifier from at least one identifier based on the first identifier, and determine an audio watermark code corresponding to the identifier as the above-mentioned watermark content information. The first identifier includes at least one of the following: the device ID of the electronic device, the user ID logged in by the above-mentioned audio processing application, and the user name logged in by the above-mentioned audio processing application.

[0079] Step 203: The electronic device generates watermark information according to the first tag information and the watermark content information.

[0080] In some embodiments of the present application, the electronic device may concatenate the first marking information and the watermark content information to obtain the watermark information.

[0081] In one example, the electronic device may obtain the watermark information by adding the watermark content information after the first marking information, or the electronic device may obtain the watermark information by using the watermark content information before the first marking information.

[0082] For example, assuming that the first marking information is a marking code, the marking code is 01010101, and the watermark content information is 000000....00001, the electronic device can splice the watermark content information 000000....00001 after the marking code 01010101 to obtain the watermark information 01010101000000....00001.

[0083] It can be seen that since the electronic device can obtain the first mark information used to locate the watermark information while obtaining the watermark content information, when parsing the watermark information, the position of the watermark information can be accurately determined based on the first mark information, and the watermark content information can be accurately parsed based on the position of the watermark information. Therefore, the situation where the watermark information cannot be parsed can be reduced, thereby improving the reliability of the electronic device in performing the watermark embedding operation.

[0084] In some embodiments of the present application, after generating watermark information, the electronic device may first input the watermark information into the linear layer of the watermark processing model, and map the watermark information into a vector consistent with the waveform length of each audio segment through the linear layer to obtain N vectors. Then the electronic device may first pre-process the N vectors, and then perform a short-time Fourier transform operation on the processed N vectors to obtain N short-time amplitude spectra, and finally input the N short-time amplitude spectra into the Mel filter group to obtain N watermark features. The above pre-processing may include at least one of the following: sampling rate conversion, pre-emphasis processing, frame processing, and windowing processing.

[0085] It can be understood that the N watermark features may be mel spectrum features.

[0086] Step 102: The electronic device inputs N first spectrum features and N watermark features into a watermark processing model, and embeds each watermark feature into a corresponding first spectrum feature through a generator of the watermark processing model to obtain N second spectrum features.

[0087] In some embodiments of the present application, the watermark processing model may be a reversible adversarial generative neural network. The watermark processing model may include a generator, which is used to embed the watermark feature into the corresponding first spectrum feature, or separate the watermark feature from the first spectrum feature. Of course, the watermark processing model may also include other layers or structures, which are not limited in the embodiments of the present application.

[0088] It can be understood that, in this example, the generator is used to embed the watermark feature into the corresponding first spectrum feature.

[0089] In some embodiments of the present application, the above-mentioned generator may include at least one reversible block, each reversible block includes at least two complementary affine coupling layers, and its input and output dimensions are consistent, so that the generator can input N first spectral features and N watermark features into the at least one reversible block, and embed each watermark feature into the corresponding first spectral feature through the at least one reversible block.

[0090] Each reversible block may adopt a ConvNeXt structure. Of course, each reversible block may also adopt other structures, which are not limited in the embodiments of the present application.

[0091] In some embodiments of the present application, the generator includes M reversible blocks connected in sequence, where M is a positive integer. Figure 1 ,like Figure 5 As shown, the above step 102 can be specifically implemented through the following steps 102a and 102b.

[0092] Step 102a: The electronic device inputs the N first spectrum features and the N watermark features into the first reversible block of the M reversible blocks, and embeds at least part of each watermark feature into the corresponding first spectrum feature through the first reversible block.

[0093] In some embodiments of the present application, each reversible block includes at least two complementary affine coupling layers, and each affine coupling layer includes at least two operation functions, so that at least part of the features of each watermark feature can be embedded into the corresponding first spectral feature through the at least two complementary affine coupling layers.

[0094] For example, Figure 6 Schematic diagram of any reversible block is shown. Assume that any reversible block is the first reversible block 12, the first reversible block includes two complementary affine coupling layers, for example, affine coupling layer 1 and affine coupling layer 2, the affine coupling layer 1 includes operation function C1 and operation function N1, the affine coupling layer 2 includes operation function C2 and operation function N2, as shown in FIG. Figure 6 As shown, the electronic device can input the first spectrum feature 1 and the corresponding watermark feature 1 into the first reversible block 12. At this time, the operation function C1 and the operation function N1 can extract the features of the watermark feature 1 and embed the extracted partial features into the first spectrum feature 1, and the operation function C2 and the operation function N2 can embed the partial noise features in the extracted partial features into the watermark feature 1, so that the first reversible block 12 can output the processed first spectrum feature 2 and the processed watermark feature 2. It can be understood that the first spectrum feature 2 includes some features in the watermark feature 1.

[0095] In some embodiments of the present application, at least two complementary affine coupling layers included in each reversible block may use a first preset algorithm to embed at least part of each watermark feature into the corresponding first spectrum feature.

[0096] The first preset algorithm may include:

[0097] v i+1 =v i ·exp(C1(u i ))+N1(u i );

[0098] u i+1 =u i ·exp(C2(v i+1 ))+N2(v i+1 );

[0099] Among them, v i is the first spectral feature of the first reversible block, v i is the first spectral feature output by the first reversible block, u i is the watermark feature of the first reversible block, u i+1 It is the watermark feature output by the first reversible block, and C1, N1, C2 and N2 are all operation functions.

[0100] Step 102b: The electronic device inputs the N first spectrum features and N watermark features output by the first reversible block into the i-th reversible block of the M reversible blocks, and embeds at least part of each watermark feature into the corresponding first spectrum feature through the i-th reversible block.

[0101] In the embodiment of the present application, i is a positive integer greater than 1.

[0102] It should be noted that, for the description of embedding at least part of the features of each watermark feature into the corresponding first spectral feature through the i-th reversible block, reference may be made to the specific description of embedding at least part of the features of each watermark feature into the corresponding first spectral feature through the first reversible block in the above embodiment, and the embodiments of the present application will not be repeated here.

[0103] It can be seen that since the generator can include M reversible blocks connected in sequence, the generator can extract N watermark features in sequence through the M reversible blocks, and embed the extracted features into the corresponding first spectrum features. Therefore, the situation of incomplete feature extraction of N watermark features can be reduced, thereby reducing the situation of inaccurate N second spectrum features.

[0104] Step 103: The electronic device generates watermark audio based on the N second frequency spectrum features and the audio.

[0105] In one example, the electronic device may first generate a replacement audio segment based on each second spectrum feature to obtain N replacement audio segments, and then use each replacement audio segment to replace the corresponding audio segment in the audio to obtain the watermarked audio.

[0106] In another example, the electronic device may replace the corresponding first spectrum feature in the spectrum feature of the audio based on each second spectrum feature, and generate the watermarked audio based on the replaced spectrum feature. Figure 1 ,like Figure 7 As shown, the above step 103 can be specifically implemented through the following steps 103a and 103b.

[0107] Step 103a: The electronic device uses each second frequency spectrum feature to replace the corresponding first frequency spectrum feature in the frequency spectrum feature of the audio, so as to obtain the replaced frequency spectrum feature of the audio.

[0108] In some embodiments of the present application, the electronic device may first delete the N first spectrum features from the spectrum features of the audio, and then insert the corresponding second spectrum feature at the position of each first spectrum feature to obtain the replaced spectrum features of the audio.

[0109] Step 103b: The electronic device generates watermark audio based on the replaced frequency spectrum features.

[0110] In some embodiments of the present application, the electronic device may input the replaced spectral features into a vocoder, and the vocoder may perform audio reconstruction according to the replaced spectral features to obtain watermarked audio in a predetermined format.

[0111] The above-mentioned vocoder may include at least one of the following: HiFiGan, LPCnet. Of course, the vocoder may also include other vocoders, which is not limited in the embodiment of the present application. The above-mentioned predetermined format may specifically be a wav format, and of course the predetermined format may also be other formats, which is not limited in the embodiment of the present application.

[0112] It can be seen that since the electronic device can use each second spectral feature to replace the corresponding first spectral feature in the spectral features of the audio, without processing other spectral features in the spectral features of the audio, the amount of calculation for obtaining the watermarked audio can be simplified, thereby reducing the power consumption of generating the watermarked audio.

[0113] The embodiment of the present application provides an audio processing method, where an electronic device can obtain N first spectral features and N watermark features that correspond to each other, each first spectral feature is a spectral feature of an audio segment in the audio, and each watermark feature is a feature of watermark information, and the N first spectral features and N watermark features are input into a watermark processing model, and each watermark feature is embedded into the corresponding first spectral feature through a generator of the watermark processing model to obtain N second spectral features, so that the electronic device can generate watermark audio based on the N second spectral features and the audio. Since the electronic device can obtain N first spectral features of N audio segments in the audio and N watermark features of N watermark information, and embed each watermark feature into the corresponding first spectral feature through a generator of a trained watermark processing model to obtain N second spectral features, instead of directly adding N watermark information to N audio segments, therefore, on the one hand, because the first spectral features and watermark features are not easily interfered or destroyed due to signal processing, even if the obtained watermark audio is subjected to signal processing, the watermark feature will not be interfered or destroyed. On the other hand, because the watermark processing model is a trained model, after N first spectral features and N watermark features are input into the watermark processing model, the generator of the watermark processing model can accurately embed each watermark feature into the corresponding first spectral feature, thereby improving the reliability of the watermarked audio.

[0114] Furthermore, since the electronic device does not use any of the least significant bit editing method, transform domain algorithm, echo hiding method, spread spectrum watermark technology, and echo modulation techniques to perform watermark embedding operations, there is no need to require a lower number of bits for the watermark information, thereby increasing the number of user IDs that can be covered by the watermark information.

[0115] Furthermore, since the watermark processing model is a trained model, the difference between the N second spectral features generated by the watermark processing model and the above-mentioned N first spectral features can be smaller. Therefore, the degradation of the sound quality of the watermarked audio can be avoided, thereby improving the sound quality of the replica watermarked audio obtained by performing the watermark embedding operation, and further improving the reliability of the electronic device performing the watermark embedding operation.

[0116] The following example illustrates a specific scheme for training the generator to be trained in the watermark processing model.

[0117] It should be noted that the training watermark processing model can be obtained by training an electronic device or other electronic devices (such as a server, etc.). In the embodiment of the present application, the training watermark processing model of an electronic device is used as an example for illustration.

[0118] In some embodiments of the present application, Figure 1 ,like Figure 8As shown, before the above step 101, the audio processing method provided in the embodiment of the present application may further include the following steps 204 to 207.

[0119] Step 204: The electronic device obtains T first sample spectrum features and T sample watermark features that correspond to each other.

[0120] In the embodiment of the present application, the first sample spectrum feature is a spectrum feature of a sample audio segment in the sample audio, the sample watermark feature is a feature of the sample watermark information, and T is a positive integer.

[0121] It should be noted that, for the description of the first sample spectrum characteristics, sample watermark characteristics, sample audio, sample audio segment and sample watermark information, reference may be made to the specific description of the first spectrum characteristics, watermark characteristics, audio, audio segment and watermark information in the above embodiments, and the embodiments of the present application will not go into details therein.

[0122] In some embodiments of the present application, the electronic device may select sample audio from a training sample set and determine T sample audio segments from the sample audio, so that the electronic device may obtain T first sample spectrum features of the T sample audio segments.

[0123] It should be noted that, for the description of the electronic device determining T sample audio segments from the sample audio, reference may be made to the specific description of the electronic device determining N audio segments from the audio in the above embodiment, and the embodiments of the present application will not be repeated here. For the description of the electronic device obtaining T first sample spectral features of T sample audio segments, reference may be made to the specific description of the electronic device obtaining N first spectral features of N audio segments in the above embodiment, and the embodiments of the present application will not be repeated here.

[0124] In some embodiments of the present application, the electronic device may generate a binary code with a number of bits greater than or equal to the above-mentioned preset number of bits, and determine the binary code as sample watermark information, so that the electronic device may determine T sample watermark features according to the sample watermark information. Among them, the preset number of bits may be 40, and of course the preset number of bits may also be other values, which are not limited in the embodiments of the present application.

[0125] Among them, the electronic device can use algorithms such as a linear congruential generator or a Mersenne twister algorithm to generate T binary codes with a bit number greater than or equal to the above-mentioned preset bit number to obtain T sample watermark features.

[0126] It should be noted that, for the description of the electronic device determining T sample watermark features according to the sample watermark information, reference may be made to the specific description of the electronic device determining N watermark features according to the watermark information in the above embodiment, which will not be repeated herein in the embodiment of the present application.

[0127] Step 205: The electronic device inputs T first sample spectrum features and T sample watermark features into the watermark processing model, and embeds each sample watermark feature into the corresponding first sample spectrum feature through the generator to be trained of the watermark processing model to obtain N second sample spectrum features.

[0128] It should be noted that, for the description of the electronic device obtaining N second sample spectrum characteristics, reference may be made to the specific description of the electronic device obtaining N second spectrum characteristics in the above embodiment, which will not be repeated herein in the embodiment of the present application.

[0129] Step 206: The electronic device generates a sample watermark audio based on the N second sample frequency spectrum features and the sample audio.

[0130] It should be noted that, for the description of the electronic device generating sample watermark audio based on N second sample spectral features and sample audio, reference can be made to the specific description of the electronic device generating watermark audio based on N second spectral features and audio in the above embodiment, and the embodiments of the present application will not be repeated here.

[0131] Step 207: The electronic device determines a first loss function based on the sample watermark audio and the sample audio, and uses the first loss function to train the generator to be trained to obtain a generator.

[0132] In the embodiment of the present application, the first loss function is used to characterize the audio type difference between the sample watermark audio and the sample audio.

[0133] It can be understood that since the first loss function is used to characterize the audio type difference between the sample watermark audio and the sample audio, after the generator is trained using the first loss function, the difference between the second spectral features generated by the generator and the spectral features of the audio is small, thereby ensuring that the audio type difference between the generated watermark audio and the audio is small, that is, the user cannot distinguish the watermark audio and the audio from the auditory sense.

[0134] In some embodiments of the present application, before the generator is trained, the electronic device may repeatedly perform steps 204 to 207 to repeatedly train the generator to be trained.

[0135] It can be seen that after the electronic device determines N second spectral features based on T first sample spectral features and T sample watermark features, and generates sample watermark audio based on the N second spectral features, it can use the first loss function used to characterize the audio type difference between the sample watermark audio and the sample audio to train the generator to be trained. In this way, the difference between the second spectral features generated by the trained generator and the spectral features of the audio is small, thereby ensuring that the audio type difference between the generated watermark audio and the audio is small, that is, the user cannot distinguish the watermark audio and the audio by hearing.

[0136] In some embodiments of the present application, the above step 207 can be specifically implemented through the following steps 207a and 207b.

[0137] Step 207a: The electronic device inputs the N second sample frequency spectrum features into the discriminator of the watermark processing model, and determines the audio type of the sample watermark audio through the discriminator.

[0138] In some embodiments of the present application, the above-mentioned audio type can be any one of the following: replica audio, real audio.

[0139] In some embodiments of the present application, the electronic device may perform binary classification on the N second sample frequency spectrum features through a discriminator to obtain an output result, namely, the audio type of the sample watermark audio.

[0140] Step 207b: The electronic device determines a first loss function according to the audio type of the sample watermark audio and the audio type of the sample audio.

[0141] In the embodiment of the present application, the audio type of the above-mentioned sample audio is real audio.

[0142] In some embodiments of the present application, the electronic device may construct a first loss function by using a first algorithm. The first algorithm is:

[0143] L g =log(1-D(f′));

[0144] Among them, L g is the first loss function, D(f') is the audio type of the sample watermarked audio, and 1 is used to indicate the audio type of the sample audio.

[0145] It can be seen that since the electronic device can accurately determine the audio type of the sample watermark audio through the discriminator, the electronic device can accurately determine the first loss function based on the accurate audio type of the sample watermark audio and the audio type of the sample audio, so that after the generator to be trained is trained using the first loss function, the difference between the second spectral features generated by the trained generator and the spectral features of the audio is small, thereby ensuring that the audio type difference between the generated watermark audio and the audio is small, that is, the user cannot distinguish the watermark audio and the audio from the auditory sense.

[0146] In some embodiments of the present application, before the above step 207a, the audio processing method provided by the embodiment of the present application may further include the following steps 301 to 303.

[0147] Step 301: The electronic device inputs N second sample spectral features and spectral features of the sample watermark audio into the to-be-trained discriminator of the watermark processing model, and determines the training audio type of the sample watermark audio and the training audio type of the sample audio through the to-be-trained discriminator.

[0148] It can be understood that the above training audio type can be an audio type determined by the discriminator to be trained. The training audio type may be different from the real audio type.

[0149] In some embodiments of the present application, the above-mentioned discriminator to be trained can perform binary classification based on the N second sample spectral features to determine the training audio type of the sample watermark audio, and perform binary classification based on the spectral features of the sample watermark audio to determine the training audio type of the sample audio.

[0150] Step 302: The electronic device determines a third loss function according to the training audio type of the sample watermark audio, and determines a fourth loss function according to the training audio type of the sample audio.

[0151] In an embodiment of the present application, the third loss function is used to characterize the difference between the training audio type of the sample watermark audio and the audio type of the sample watermark audio, and the fourth loss function is used to characterize the difference between the training audio type of the sample audio and the audio type of the sample audio.

[0152] In some embodiments of the present application, the electronic device may use the second algorithm to construct the third loss function, and use the third algorithm to construct the fourth loss function. The second algorithm is:

[0153] L d1 =log(D(f′));

[0154] Among them, L d1is the third loss function, f' is the second sample spectrum feature, and D(f') is used to indicate the training audio type of the sample watermark audio.

[0155] The third algorithm is:

[0156] L d1 =log(1-D(f));

[0157] Among them, L d2 is the fourth loss function, f is the spectral feature of the sample watermark audio, and D(f) is used to indicate the training audio type of the sample audio.

[0158] Step 303: The electronic device trains the discriminator to be trained using the third loss function and the fourth loss function to obtain the discriminator.

[0159] In some embodiments of the present application, the electronic device may determine the sum of the third loss function and the fourth loss function as the loss function used for training, and use the loss function used for training to train the discriminator to be trained.

[0160] It can be understood that the electronic device can first determine the loss function L used in training d =log(1-D(f))+log(D(f')), using the loss function L used in the training d Train the discriminator to be trained.

[0161] Thus, it can be known that the electronic device can also determine the training audio type of the sample watermark audio based on the N second sample spectral features, and determine the third loss function based on the training audio type of the sample watermark audio, and determine the training audio type of the sample audio based on the spectral features of the sample watermark audio, and determine the fourth loss function based on the training audio type of the sample audio. In this way, after the discriminator to be trained is trained using the third loss function and the fourth loss function, the trained discriminator can accurately determine the audio type of the audio corresponding to the input spectral features, so that the electronic device can determine a more accurate first loss function in the subsequent steps. That is to say, the electronic device can perform adversarial training on the generator to be trained and the discriminator to be trained, so that the difference between the second spectral features generated by the trained generator and the spectral features of the audio can be further reduced, thereby further reducing the audio type difference between the generated watermark audio and the audio, and then the sound quality of the watermark audio obtained by performing the watermark embedding operation can be improved, so that the reliability of the electronic device performing the watermark embedding operation can be improved.

[0162] Of course, in order to further ensure the difference between the second spectral features generated by the trained generator and the spectral features of the audio, the electronic device can also construct multiple loss functions to train the generator to be trained through the first loss function and the multiple loss functions, which will be illustrated by examples below.

[0163] In some embodiments of the present application, before the above step 207, the audio processing method provided in the embodiment of the present application may further include the following step 304, and the above step 207 may be specifically implemented by the following step 207c.

[0164] Step 304: The electronic device determines a second loss function according to the T first sample spectrum features and the T sample watermark features.

[0165] In the embodiment of the present application, the second loss function is used to characterize the difference between T first sample spectrum features and T sample watermark features.

[0166] In some embodiments of the present application, the electronic device may determine the second loss function according to the gap between each first sample spectrum feature and the corresponding sample watermark feature.

[0167] The electronic device may use a fourth algorithm to construct the second loss function. The fourth algorithm is:

[0168]

[0169] Among them, L f is the second loss function, f is the first sample spectrum feature, and f' is the sample watermark feature.

[0170] Step 207c: The electronic device determines a first loss function according to the sample watermark audio and the sample audio, and uses the first loss function and the second loss function to train the generator to be trained.

[0171] In some embodiments of the present application, the electronic device may determine the sum of the first loss function and the second loss function as the loss function used for training, and train the generator to be trained based on the loss function used for training.

[0172] It can be seen that since the electronic device can also determine the second loss function based on T first sample spectral features and T sample watermark features, after the generator to be trained is trained using the first loss function and the second loss function, the difference between the second spectral features generated by the trained generator and the spectral features of the audio is smaller, thereby ensuring that the audio type difference between the generated watermark audio and the audio is smaller, that is, the user cannot distinguish the watermark audio and the audio from the auditory sense. Therefore, the sound quality of the watermark audio can be further improved, and thus, the reliability of the electronic device in performing the watermark embedding operation can be improved.

[0173] In some embodiments of the present application, before "using the first loss function and the second loss function to train the generator to be trained" in the above step 207c, the electronic device may also perform watermark analysis on the sample watermark audio to obtain analyzed watermark information, so that the electronic device may also determine a loss function (such as the fifth loss function in the following embodiment) based on the analyzed watermark information and the sample watermark information, and train the generator to be trained based on the first loss function, the second loss function and the one loss function to obtain the generator.

[0174] It can be understood that since the above-mentioned generator can also be used to separate the watermark feature from the first spectral feature, the electronic device can determine the above-mentioned loss function, and train the generator to be trained according to the first loss function, the second loss function and the one loss function, so that the trained generator can accurately embed the watermark feature into the corresponding first spectral feature, and can accurately separate the watermark feature from the first spectral feature.

[0175] It should be noted that the description of watermark parsing of sample watermark audio by an electronic device will be specifically described in the following embodiments, and the embodiments of the present application will not be repeated here.

[0176] In some examples, the electronic device may use a fifth algorithm to construct the above loss function. The fifth algorithm may be:

[0177]

[0178] Among them, L m is the loss function, m is the sample watermark information, To analyze watermark information.

[0179] In some examples, the electronic device may determine the sum of the first loss function, the second loss function, and the one loss function as the loss function used for training, and train the generator to be trained based on the loss function used for training.

[0180] Among them, the loss function L used in this training can be

[0181] L=λ f L f +λ g L g +L m ;

[0182] Among them, L g is the first loss function, L f is the second loss function, λ f is the weight of the first loss function, λg is the weight of the second loss function.

[0183] It can be understood that by controlling λ f and λ g The size of λ is used to control the balance between the stability of the watermark information and not affecting the auditory experience. Here, λ f The value can be 10000, g The value can be 10.

[0184] In some embodiments of the present application, Figure 1 ,like Fig. 9 As shown, after the above step 103, an audio processing method provided by an embodiment of the present application may include the following steps 401 to 403.

[0185] Step 401: The electronic device obtains L third frequency spectrum features.

[0186] In the embodiment of the present application, each of the L third spectral features is a spectral feature corresponding to an audio segment in the watermarked audio, and L is a positive integer.

[0187] In some embodiments of the present application, when an electronic device receives watermarked audio from other electronic devices, the electronic device can directly obtain L third spectrum features.

[0188] In some embodiments of the present application, the L third spectrum features are the N second spectrum features in the above embodiment, or the L third spectrum features are spectrum features of an audio segment determined by the electronic device.

[0189] It is understandable that, since the electronic device may not be able to accurately determine the position of the audio segment to which the watermark information has been added, the electronic device may determine L audio segments from the watermarked audio to determine L third spectral features. The L audio segments may include the above-mentioned N audio segments, or the L audio segments may be at least part of the above-mentioned N audio segments.

[0190] The following is an example of a specific solution for an electronic device to determine L third frequency spectrum features from watermarked audio.

[0191] In some embodiments of the present application, Fig. 9 ,like Fig.10 As shown, before the above step 401, the audio processing method provided in the embodiment of the present application may further include the following steps 501 and 502.

[0192] Step 501: The electronic device determines at least one sliding window.

[0193] In the embodiment of the present application, the duration corresponding to each sliding window in the at least one sliding window matches the preset duration, and the interval between two adjacent sliding windows is the preset time interval.

[0194] In some embodiments of the present application, the number of sliding windows, preset duration, and preset time interval can be pre-configured in the electronic device, so that the electronic device can determine at least one sliding window based on the number of sliding windows, preset duration, and preset time interval.

[0195] The number of the sliding windows may be the same as the number N of audio segments in the audio in the above embodiment. It can be understood that since the electronic device adds watermark information to the N audio segments in the audio according to the preset duration and the preset time interval, the electronic device may determine at least one sliding window according to the number N of sliding windows, the preset duration and the preset time interval, so as to determine the audio segment to which the watermark information is added through the at least one sliding window.

[0196] In some embodiments of the present application, an audio start position can be pre-configured in an electronic device, and then the electronic device can determine the position after a preset time length from the audio start position as an audio end position, and determine the window corresponding to the audio start position and the audio end position as a sliding window. Next, the electronic device can first determine the position after a preset time interval from the audio end position as another audio start position, and then determine the position after a preset time length from the other audio start position as another audio end position, and determine the window corresponding to the other audio start position and the other audio end position as another sliding window. And so on, to determine at least one sliding window, that is, N sliding windows.

[0197] Step 502: The electronic device controls at least one sliding window to slide on the audio segment of the watermarked audio according to a first preset step length, and determines at least one audio segment located in the at least one sliding window after each sliding as at least one audio segment, so as to determine L audio segments.

[0198] In some embodiments of the present application, the first preset step length may be determined by a preset duration. The first preset step length may be Y times the preset duration, where Y is a positive number. For example, the first preset step length may be 0.1 times the preset duration.

[0199] In some embodiments of the present application, the electronic device may first control at least one sliding window to slide at least once in the first direction on the audio segment of the watermarked audio according to the first preset step length, and determine at least one audio segment located in the at least one sliding window after each sliding as at least one audio segment 1; and then control at least one sliding window to slide at least once in the second direction on the audio segment of the watermarked audio according to the first preset step length, and determine at least one audio segment located in the at least one sliding window after each sliding as at least one audio segment 2; thereby, the electronic device may determine at least one audio segment 1 and at least one audio segment 2 as L audio segments. The second direction is opposite to the first direction.

[0200] For example, assuming that the preset duration is 1s, the preset time interval is 1s, then the duration of each sliding window in at least one is 1s, the first preset step is 0.1s, the duration of the watermarked audio is 3.5s, and the number of audio segments in the watermarked audio is 2. Fig.11 As shown, the electronic device may first control the two sliding windows to slide at least once in the first direction at 0.1s, for example, control the sliding window 13 and the sliding window 14 to slide to the right 5 times, and determine the two audio segments located in the two sliding windows after each sliding as audio segment 1; then the electronic device may control the two sliding windows to slide at least once in the second direction at 0.1s, for example, control the sliding window 13 and the sliding window 14 to slide to the left 5 times, and determine the two audio segments located in the two sliding windows after each sliding as audio segment 2. Thus, the electronic device may determine L audio segments, namely, audio segment 1 and audio segment 2.

[0201] It should be noted that in Fig.11 In the figure, a plurality of dotted-line frames are used to illustrate the sliding window 13 and the sliding window 14, and each dotted-line frame can be understood as the position of the sliding window 13 or the sliding window 14 after the sliding window 13 or the sliding window 14 moves.

[0202] It can be seen that since the electronic device can determine at least one sliding window based on the preset duration and the preset time interval, and slide through the at least one sliding window multiple times to determine L audio segments, that is, determine the audio segments including the watermark information added, without the electronic device indicating the specific location of the audio segments with the watermark information added, the complexity of parsing the watermark information can be simplified.

[0203] Step 402: The electronic device inputs L third frequency spectrum features into a watermark processing model, and determines a watermark feature in each third frequency spectrum feature through a generator of the watermark processing model.

[0204] In the embodiment of the present application, each watermark feature is a feature of the watermark information.

[0205] In some embodiments of the present application, the generator includes M reversible blocks connected in sequence, where M is a positive integer. Fig. 9 ,like Fig.12 As shown, the above step 402 can be specifically implemented through the following steps 402a and 402b.

[0206] Step 402a: the electronic device inputs the corresponding L third spectrum features and L noise features into the Mth reversible block among the M reversible blocks, and embeds at least part of the watermark feature in each third spectrum feature into the corresponding noise feature through the Mth reversible block.

[0207] In some embodiments of the present application, the above-mentioned noise feature may be random noise generated by an electronic device.

[0208] In some embodiments of the present application, each reversible block includes at least two complementary affine coupling layers, and each affine coupling layer includes at least two operation functions, so that at least part of the watermark features in each third spectral feature can be embedded into the corresponding noise feature through the at least two complementary affine coupling layers.

[0209] For example, Fig.13 Schematic diagram of any reversible block is shown. Assume that any reversible block is the Mth reversible block 15, the Mth reversible block 15 includes two complementary affine coupling layers, for example, affine coupling layer 1 and affine coupling layer 2, the affine coupling layer 1 includes operation function C1 and operation function N1, the affine coupling layer 2 includes operation function C2 and operation function N2, as shown in FIG. Fig.13 As shown, the electronic device can input the third spectrum feature 1 and the corresponding noise feature 1 into the Mth reversible block 15. At this time, the operation function C2 and the operation function N2 can embed part of the features of a watermark feature in the third spectrum feature 13 into the noise feature 1, and the operation function C2 and the operation function N2 can embed part of the noise in the extracted part of the features into the third spectrum feature 1, so that the Mth reversible block 15 can output the processed third spectrum feature 2 and the processed noise feature 2. It can be understood that the noise feature 2 includes part of the features of a watermark feature.

[0210] In some embodiments of the present application, at least two complementary affine coupling layers included in each reversible block may use a second preset algorithm to embed at least part of the watermark features in each third spectral feature into the corresponding noise feature.

[0211] The second preset algorithm may include:

[0212] u i =(u i+1 -N2(vi+1 ))·exp(-C2(v i+1 ));

[0213] v i =(v i+1 -N1(u i ))·exp(-C1(u i ));

[0214] Among them, v i+1 is the third spectral feature of the first reversible block, v i is the third spectral feature output by the Mth reversible block, u i+1 is the watermark feature of the input Mth reversible block, u i is the watermark feature output by the Mth reversible block, C1, N1, C2 and N2 are all operation functions.

[0215] Step 402b: The electronic device inputs the L third spectrum features and L noise features output by the Mth reversible block into the Mjth reversible block among the M reversible blocks, and embeds at least part of the watermark feature in each third spectrum feature into the corresponding noise feature through the Mjth reversible block.

[0216] In the embodiment of the present application, j is a positive integer less than M.

[0217] It can be seen that since the generator can include M reversible blocks connected in sequence, the generator can use the M reversible blocks to extract features of the L third spectrum features in sequence, and embed at least part of the watermark features in each third spectrum feature into the corresponding noise feature to obtain accurate watermark features. Therefore, the situation of incomplete feature extraction of the L third spectrum features can be reduced, thereby reducing the situation of inaccurate L watermark features.

[0218] Step 403: The electronic device determines watermark information based on L watermark features.

[0219] In a possible implementation of the present application, the electronic device may perform restoration processing according to each watermark feature respectively to obtain L watermark information, and then determine the watermark information with the highest number of repetitions among the L watermark information as the above watermark information.

[0220] It can be understood that since some of the determined L audio segments may be inaccurate, that is, the some audio segments may be completely different from any of the above-mentioned N audio segments, this may cause some of the L third spectral features to be inaccurate, thereby causing some of the L watermark features to be inaccurate. Therefore, the electronic device can first determine the L watermark information, and then determine the watermark information with the highest number of repetitions among the L watermark information, that is, the watermark information that is most likely to be correct, as the above-mentioned watermark information.

[0221] In another possible implementation of the present application, the electronic device can perform restoration processing according to each watermark feature to obtain L watermark information, and then determine the most likely correct watermark information based on the marking information in each watermark information, so as to determine the most likely correct watermark information as the above watermark information. Fig. 9 ,like Fig.14 As shown, the above step 403 can be specifically implemented through the following steps 403a to 403c.

[0222] Step 403a: The electronic device generates a candidate watermark information based on each watermark feature.

[0223] In the embodiment of the present application, each candidate watermark information includes second marking information and watermark content information.

[0224] It should be noted that, for the description of the second tag information, reference can be made to the specific description of the first tag information in the above embodiment, and the present embodiment will not be repeated here. For the description of the watermark content information, reference can be made to the specific description in the above embodiment, and the present embodiment will not be repeated here.

[0225] In some embodiments of the present application, the electronic device may perform restoration processing based on each watermark feature to obtain each candidate watermark information, that is, L candidate watermark information.

[0226] It can be understood that each candidate watermark information can be binary coded.

[0227] Step 403b: The electronic device determines at least one candidate watermark information including second marking information matching the first marking information from the L candidate watermark information.

[0228] In the embodiment of the present application, the first marking information is used to locate the watermark information.

[0229] Step 403c: The electronic device determines watermark information according to the mean value of at least one candidate watermark information.

[0230] In some embodiments of the present application, the electronic device may sum and average each digit of at least one candidate watermark information. If the average result of a certain digit is greater than 0.5, the average of the certain digit is 1, otherwise the average of the certain digit is 0, thereby obtaining the watermark information.

[0231] Exemplarily, assuming that at least one candidate watermark information includes: candidate watermark information 00000100, candidate watermark information 00001010 and candidate watermark information 00001100, the electronic device can first sum up the first digit of each candidate watermark information and then take the average, the mean of the first digit is 0, then sum up the second digit of each candidate watermark information and then take the average, the mean of the second digit is 0, and so on and so forth, and finally the watermark information 00001100 can be obtained.

[0232] It can be seen that since the electronic device can first determine L candidate watermark information and accurately determine at least one candidate watermark information that may be the watermark information based on the second label information of the L candidate watermark information, the electronic device can accurately determine the watermark information based on the mean of the at least one candidate watermark information that may be the watermark information.

[0233] In summary, the electronic device can obtain L third spectral features, each third spectral feature is a spectral feature of an audio segment in the watermark audio, L is a positive integer, and the L third spectral features are input into the watermark processing model, and the watermark feature in each third spectral feature is determined by the generator of the watermark processing model, and the watermark feature is a feature of the watermark information, so that the electronic device can determine the watermark information based on the L watermark features. Since the electronic device can obtain the L third spectral features corresponding to the audio segment in the watermark audio, and determine the watermark feature in each third spectral feature by the generator of the trained watermark processing model, therefore, on the one hand, because the third spectral features and the watermark features are not easily interfered or destroyed due to signal processing, even if the watermark audio is subjected to signal processing, the electronic device can still obtain accurate L third spectral features, and accurately determine the watermark feature in each third spectral feature by the generator to obtain accurate L watermark features, so that the electronic device can accurately determine the watermark information based on the accurate L watermark features, and can accurately determine that the watermark audio is a replica audio, and will not mistakenly determine that the watermark audio is a real audio. On the other hand, because the watermark processing model is a trained model, after the L third spectral features are input into the watermark processing model, the generator of the watermark processing model can accurately determine the watermark features in each third spectral feature, thereby reducing the situation where the electronic device determines the wrong watermark features, and then the electronic device can accurately determine the watermark information based on the accurate L watermark features, and can accurately determine that the watermarked audio is a replica audio, and will not mistakenly determine that the watermarked audio is real audio. In this way, the reliability of the electronic device in distinguishing between replica audio and real audio can be improved.

[0234] Furthermore, the electronic device can accurately determine the watermark information based on the L third frequency spectrum features without the need for electronic device instructions, thereby simplifying the complexity of parsing the watermark information.

[0235] The following example illustrates a specific scheme for training the generator to be trained in the watermark processing model.

[0236] It should be noted that the training watermark processing model can be obtained by training an electronic device or other electronic devices (such as a server, etc.). In the embodiment of the present application, the training watermark processing model of an electronic device is used as an example for illustration.

[0237] In some embodiments of the present application, Fig. 9 ,like Fig.15 As shown, before the above step 401, the audio processing method provided in the embodiment of the present application may further include the following steps 503 to 506.

[0238] Step 503: The electronic device obtains R third sample frequency spectrum features.

[0239] In the embodiment of the present application, each of the R third sample frequency spectrum features is a frequency spectrum feature of a sample audio segment in the sample watermark audio, and R is a positive integer.

[0240] In some embodiments of the present application, the electronic device may first obtain T first sample spectrum features and T sample watermark features in one-to-one correspondence, wherein the first sample spectrum features are spectrum features of a sample audio segment in the sample audio, and the sample watermark features are features of the sample watermark information, and T is a positive integer; then the T first sample spectrum features and the T sample watermark features are input into the watermark processing model, and each sample watermark feature is embedded into the corresponding sample spectrum feature through the generator to be trained of the watermark processing model to obtain N second sample spectrum features; thus, the electronic device may generate sample watermark audio based on the N second sample spectrum features and the sample audio. In this way, the electronic device may obtain R third sample spectrum features.

[0241] It should be noted that, for the description of the electronic device generating the sample watermark audio, reference may be made to the specific description of the electronic device generating the sample watermark audio in the above embodiment, which will not be repeated herein in the embodiment of the present application.

[0242] In some embodiments of the present application, the above step 503 can be specifically implemented through the following steps 503a and 503b.

[0243] Step 503a: The electronic device determines at least one training sliding window.

[0244] In the embodiment of the present application, the duration corresponding to each training sliding window in the at least one training sliding window matches the preset duration, and the interval between two adjacent training sliding windows is the preset time interval.

[0245] It should be noted that, for the description of determining at least one training sliding window for the electronic device, reference may be made to the specific description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0246] Step 503b: The electronic device controls at least one training sliding window to slide on the audio segment of the watermarked audio according to a second preset step length, and determines at least one audio segment located in the at least one training sliding window after each sliding as a sample audio segment to determine R sample audio segments.

[0247] It can be seen that since the electronic device can determine at least one training sliding window based on a preset duration and a preset time interval, the electronic device can slide through the at least one training sliding window multiple times to accurately determine R sample audio segments, thereby simplifying the complexity of training the watermark processing model.

[0248] Step 504: The electronic device inputs the R third sample spectrum features into the watermark processing model, and determines the training watermark feature in each third sample spectrum feature through the to-be-trained generator of the watermark processing model.

[0249] It can be understood that the above training watermark features are watermark features parsed by the generator to be trained, and the training watermark features may be inaccurate watermark features, that is, watermark features different from the watermark features of the sample watermark information.

[0250] Step 505: The electronic device determines a parsed watermark information based on each training watermark feature.

[0251] In some embodiments of the present application, the electronic device may perform restoration processing on each training watermark feature respectively to obtain a parsed watermark information.

[0252] It can be understood that the parsed watermark information is the watermark information parsed by the generator to be trained, and the parsed watermark information may be different from the sample watermark information.

[0253] Step 506: The electronic device determines a fifth loss function based on the R parsed watermark information and the sample watermark information, and uses the fifth loss function to train the generator to be trained to obtain a generator.

[0254] In the embodiment of the present application, the fifth loss function is used to characterize the difference between the R parsed watermark information and the sample watermark information.

[0255] In some embodiments of the present application, before "using the fifth loss function to train the generator to be trained" in the above step 506, the electronic device may also determine the first loss function and the second loss function, and determine the sum of the first loss function, the second loss function and the fifth loss function as the loss function used for training, and train the generator to be trained based on the loss function used for training.

[0256] It should be noted that for the description of how the electronic device determines the first loss function and the second loss function, reference may be made to the specific description in the above embodiments, and the embodiments of the present application will not be repeated here.

[0257] In some embodiments of the present application, before the generator is trained, the electronic device may repeatedly perform steps 503 to 506 to repeatedly train the generator to be trained.

[0258] It can be seen that since the electronic device can also determine the R training watermark features based on the R third sample spectral features, and determine the fifth loss function based on the R training watermark information and sample watermark information corresponding to the R training watermark features, and use the fifth loss function to train the generator to be trained, therefore, after using the fifth loss function to train the generator to be trained, the accuracy of the watermark features generated by the trained generator is higher, thereby improving the accuracy of determining the watermark information.

[0259] In some embodiments of the present application, when the electronic device repeatedly executes steps 503 to 506 to repeatedly train the generator to be trained, the electronic device can also obtain in real time the number of rounds of training the generator to be trained. The number of rounds can be understood as the number of times the electronic device executes steps 503 to 506, and when the number of rounds is greater than or equal to a preset threshold, the sample watermark audio is data enhanced, which will be illustrated by examples below.

[0260] In some embodiments of the present application, the audio processing method provided in the embodiments of the present application may further include the following steps 507 to 509.

[0261] Step 507: During the process of training the generator to be trained, the electronic device obtains the first round number of training the generator to be trained.

[0262] In some embodiments of the present application, a first counter may be provided in the electronic device, and each time the electronic device performs a round of training on the generator to be trained, the count value of the first counter may be controlled to increase by 1, so that the electronic device can determine the current count value of the first counter as the first round number when any round of training is completed on the generator to be trained.

[0263] Step 508: When the first round number is greater than or equal to a preset threshold, the electronic device processes the sample watermark audio using at least one processing method to obtain enhanced sample watermark audio.

[0264] In an embodiment of the present application, the at least one processing method includes at least one of the following: adding noise, adding reverberation, volume scaling, audio loss, and lossy compression.

[0265] In some embodiments of the present application, the above-mentioned adding noise may specifically be: adding uniformly distributed random white noise to the sample watermark audio, wherein the signal-to-noise ratio of the random white noise may be between 30 and 35 dB. Of course, the signal-to-noise ratio of the random white noise may also be between other ranges, which is not limited in the embodiments of the present application.

[0266] In some embodiments of the present application, the above-mentioned reverberation can be specifically: randomly attenuating the volume of the sample watermark audio to 0.1-0.3 times, randomly delaying 50-100 milliseconds ms, obtaining a new watermark audio, and then superimposing the new watermark audio and the sample watermark audio. Of course, the multiple corresponding to the random attenuation and the value of the random delay can be other values, which are not limited in the embodiments of the present application.

[0267] In some embodiments of the present application, the volume scaling may specifically be: randomly changing the volume of the sample watermark audio to 80% to 120% of the sample watermark audio. Of course, the value corresponding to the random change may also be other values, which are not limited in the embodiments of the present application.

[0268] In some embodiments of the present application, the above audio loss may specifically be: 5% to 10% of the sampling points in the sample watermark audio are randomly set to zero. Of course, the number of the sampling points may also be other values, which are not limited in the embodiments of the present application.

[0269] In some embodiments of the present application, the above lossy compression may specifically be: converting the sample watermark audio into a lossy mp3 format or m4a format, and then converting it back. Of course, the lossy format may also be other formats, which are not limited in the embodiments of the present application.

[0270] In some embodiments of the present application, the sample watermark audio may be in wav format, so that the electronic device may use the sox tool or ffmpeg to process the sample watermark audio using at least one of the above processing methods.

[0271] It can be understood that in the complex environment of actual audio transmission, the audio signal of the audio will face the combined influence of multiple transformations and disturbances at the same time, so in the training stage, at least one processing method can be combined to process the sample watermark audio to obtain enhanced sample watermark audio.

[0272] In the embodiment of the present application, the number of the at least one processing method mentioned above is positively correlated with the number of the first rounds.

[0273] It can be understood that since the increase in the number of processing methods will lead to an increase in the difficulty of training the generator to be trained, when the number of first rounds of the generator to be trained is small, fewer processing methods can be used to process the sample watermark audio; when the number of first rounds of the generator to be trained is large, more processing methods can be used to process the sample watermark audio.

[0274] Step 509: The electronic device trains the generator to be trained based on the enhanced sample watermark audio.

[0275] Exemplarily, the preset threshold can be 30% of the total number of training rounds. When the first round is less than 30% of the total number of training rounds, avoid introducing any form of data enhancement, that is, do not use any processing method to process the sample watermark audio, that is, directly use the sample watermark audio to train the generator to be trained, so that the generator to be trained can learn and master the basic characteristics of the sample watermark audio, laying a solid foundation for subsequent complex training. When the first round is greater than or equal to 30% of the total number of training rounds and less than 60% of the total number of training rounds, a processing method can be used to process the sample watermark audio, and the enhanced sample watermark audio obtained by the processing can be used to train the generator to be trained, so that the generator to be trained gradually adapts and learns how to maintain the stability and recognizability of the sample watermark information under a single disturbance. When the number of first rounds is greater than or equal to 60% of the total number of training rounds and less than or equal to 100% of the total number of training rounds, at least one processing method can be used to process the enhanced sample watermark audio, and the new enhanced sample watermark audio obtained by the processing can be used to train the generator to be trained, so as to enhance the stability of the sample watermark information under complex disturbances.

[0276] In summary, if the sample watermark audio is perturbed in a variety of ways at the beginning of training to expand the amount of training data, it may cause the generator to be trained to fail to fully learn the basic sample watermark features and converge early. Using a progressive enhancement strategy to gradually increase the diversity of perturbations is beneficial to improving the stability and generalization of the generator to be trained.

[0277] Thus, it can be known that, since the electronic device can also process the sample watermark audio using at least one processing method according to the first round number of training the generator to be trained during the training process, so as to introduce at least one noise into the sample watermark audio and obtain enhanced sample watermark audio, after the generator to be trained is trained using the enhanced sample watermark audio, the trained generator can accurately parse out the watermark information when the watermark audio includes at least one noise, thereby improving the accuracy of parsing the watermark information. Moreover, since at least one processing method is positively correlated with the first round number of training the generator to be trained, the stability and generalization of the generator to be trained can be improved.

[0278] The audio processing method provided in the embodiment of the present application can be executed by an audio processing device. In the embodiment of the present application, an audio processing device executing the audio processing method is taken as an example to illustrate the audio processing device provided in the embodiment of the present application.

[0279] Fig.16 FIG. 1 is a schematic diagram showing the structure of an audio processing device provided in an embodiment of the present application. Fig.16As shown, the audio processing device 60 provided in the embodiment of the present application may include: an acquisition module 61 and a processing module 62.

[0280] The acquisition module 61 is used to acquire N first spectrum features and N watermark features that correspond to each other. The first spectrum features are spectrum features of the audio segment in the audio, and the watermark features are features of the watermark information. N is a positive integer. The processing module 62 is used to input the N first spectrum features and N watermark features acquired by the acquisition module 61 into the watermark processing model, and embed each watermark feature into the corresponding first spectrum feature through the generator of the watermark processing model to obtain N second spectrum features; and generate watermark audio based on the N second spectrum features and the audio.

[0281] The embodiment of the present application provides an audio processing device. Since the audio processing device can obtain N first spectral features of N audio segments in the audio and N watermark features of N watermark information, and embed each watermark feature into the corresponding first spectral feature through a generator of a trained watermark processing model to obtain N second spectral features, instead of directly adding N watermark information to N audio segments, on the one hand, because the first spectral features and watermark features are not easily interfered or destroyed by signal processing, even if the obtained watermark audio is subjected to signal processing, the watermark features will not be interfered or destroyed. On the other hand, because the watermark processing model is a trained model, after the N first spectral features and N watermark features are input into the watermark processing model, the generator of the watermark processing model can accurately embed each watermark feature into the corresponding first spectral feature, thereby improving the reliability of the watermark audio.

[0282] In a possible implementation, the processing module 62 is specifically configured to use each second spectral feature to replace the corresponding first spectral feature in the spectral feature of the audio, respectively, to obtain the replaced spectral feature of the audio; and generate the watermarked audio based on the replaced spectral feature.

[0283] In a possible implementation, the acquisition module 61 is further used to acquire first tag information and watermark content information before acquiring the one-to-one corresponding N first spectrum features and N watermark features, and the first tag information is used to locate the watermark information. The processing module 62 is further used to generate watermark information according to the first tag information and watermark content information acquired by the acquisition module 61.

[0284] In a possible implementation, the processing module 62 is further configured to determine an audio segment from the audio at every preset time interval before the acquisition module 61 acquires the one-to-one corresponding N first spectrum features and N watermark features, and the duration corresponding to an audio segment matches the preset duration.

[0285] In a possible implementation, the generator includes M reversible blocks connected in sequence, where M is a positive integer. The processing module 62 is specifically used to input the N first spectrum features and the N watermark features into the first reversible block among the M reversible blocks, and embed at least part of the features of each watermark feature into the corresponding first spectrum feature through the first reversible block; and input the N first spectrum features and the N watermark features output by the first reversible block into the i-th reversible block among the M reversible blocks, and embed at least part of the features of each watermark feature into the corresponding first spectrum feature through the i-th reversible block, where i is a positive integer greater than 1.

[0286] In a possible implementation, the acquisition module 61 is further used to acquire T first sample spectrum features and T sample watermark features in one-to-one correspondence before acquiring N first spectrum features and N watermark features in one-to-one correspondence, wherein the first sample spectrum features are spectrum features of the sample audio segment in the sample audio, and the sample watermark features are features of the sample watermark information, and T is a positive integer. The processing module 62 is further used to input the T first sample spectrum features and T sample watermark features acquired by the acquisition module 61 into the watermark processing model, and embed each first sample watermark feature into the corresponding first sample spectrum feature through the generator to be trained of the watermark processing model to obtain N second sample spectrum features; and generate sample watermark audio based on the N second sample spectrum features and the sample audio; and determine a first loss function based on the sample watermark audio and the sample audio, and use the first loss function to train the generator to be trained to obtain a generator, and the first loss function is used to characterize the audio type difference between the sample watermark audio and the sample audio.

[0287] In a possible implementation, the processing module 62 is further used to determine a second loss function based on the T first sample spectrum features and the T sample watermark features before using the first loss function to train the generator to be trained, and the second loss function is used to characterize the difference between the T first sample spectrum features and the T sample watermark features. The processing module 62 is specifically used to train the generator to be trained using the first loss function and the second loss function.

[0288] In a possible implementation, the processing module 62 is specifically used to input N second sample spectral features into the discriminator of the watermark processing model, determine the audio type of the sample watermark audio through the discriminator; and determine the first loss function according to the audio type of the sample watermark audio and the audio type of the sample audio.

[0289] In a possible implementation, the processing module 62 is further used to input the N second sample spectral features into the discriminator of the watermark processing model, and before determining the audio type of the sample watermark audio and the audio type of the sample audio through the discriminator, input the N second sample spectral features and the spectral features of the sample watermark audio into the discriminator to be trained of the watermark processing model, and determine the training audio type of the sample watermark audio and the training audio type of the sample audio through the discriminator to be trained; and determine a third loss function according to the training audio type of the sample watermark audio, and determine a fourth loss function according to the training audio type of the sample audio, the third loss function is used to characterize the difference between the training audio type of the sample watermark audio and the audio type of the sample watermark audio, and the fourth loss function is used to characterize the difference between the training audio type of the sample audio and the audio type of the sample audio; and use the third loss function and the fourth loss function to train the discriminator to be trained to obtain the discriminator.

[0290] In a possible implementation, the acquisition module 61 is further used to acquire L third spectrum features after the processing module 62 generates the watermark audio based on the N second spectrum features and the audio, where the third spectrum features are spectrum features in the watermark audio, and L is a positive integer. The processing module 62 is further used to input the L third spectrum features acquired by the acquisition module 61 into the watermark processing model, determine the watermark features in each third spectrum feature through the generator of the watermark processing model, and determine the watermark information based on the L watermark features.

[0291] In a possible implementation, the generator includes M reversible blocks connected in sequence, where M is a positive integer. The processing module 62 is specifically used to input the one-to-one corresponding L third spectrum features and L noise features into the Mth reversible block among the M reversible blocks, and embed at least part of the watermark features in each third spectrum feature into the corresponding noise feature through the Mth reversible block, where the noise feature is a feature of random noise; and input the L third spectrum features and L noise features output by the Mth reversible block into the Mjth reversible block among the M reversible blocks, and embed at least part of the watermark features in each third spectrum feature into the corresponding noise feature through the Mjth reversible block, where j is a positive integer less than M.

[0292] The audio processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile internet device (mobile internet device, MID), an augmented reality (augmentedreality, AR) / virtual reality (virtualreality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (network attached storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0293] The audio processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0294] The audio processing device provided in the embodiment of the present application can achieve Figures 1 to 15 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0295] In some embodiments of the present application, Fig.17 As shown, an embodiment of the present application further provides an electronic device 80, including a processor 81 and a memory 82, wherein the memory 82 stores programs or instructions that can be executed on the processor 81, and when the program or instructions are executed by the processor 81, the various process steps of the above-mentioned audio processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0296] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0297] Fig.18 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0298] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0299] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.18 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0300] Among them, the processor 110 is used to obtain one-to-one corresponding N first spectral features and N watermark features, where the first spectral feature is the spectral feature of the audio segment in the audio, and the watermark feature is the feature of the watermark information, and N is a positive integer; and input the N first spectral features and the N watermark features into the watermark processing model, and embed each watermark feature into the corresponding first spectral feature through the generator of the watermark processing model to obtain N second spectral features; and generate watermark audio based on the N second spectral features and the audio.

[0301] The embodiment of the present application provides an electronic device. Since the electronic device can obtain N first spectral features of N audio segments in the audio and N watermark features of N watermark information, and embed each watermark feature into the corresponding first spectral feature through a generator of a trained watermark processing model to obtain N second spectral features, instead of directly adding N watermark information to N audio segments, on the one hand, because the first spectral features and watermark features are not easily interfered or destroyed by signal processing, even if the obtained watermark audio is subjected to signal processing, the watermark features will not be interfered or destroyed. On the other hand, because the watermark processing model is a trained model, after the N first spectral features and N watermark features are input into the watermark processing model, the generator of the watermark processing model can accurately embed each watermark feature into the corresponding first spectral feature, thereby improving the reliability of the watermark audio.

[0302] In some embodiments of the present application, the processor 110 is specifically configured to use each second spectral feature to replace the corresponding first spectral feature in the spectral feature of the audio, respectively, to obtain the replaced spectral feature of the audio; and generate watermarked audio based on the replaced spectral feature.

[0303] In some embodiments of the present application, the processor 110 is further used to obtain first tag information and watermark content information before obtaining the one-to-one corresponding N first spectral features and N watermark features, where the first tag information is used to locate the watermark information; and generate watermark information based on the first tag information and the watermark content information.

[0304] In some embodiments of the present application, the processor 110 is further used to determine an audio segment from the audio at every preset time interval before obtaining the one-to-one corresponding N first spectral features and N watermark features, and the duration corresponding to an audio segment matches the preset duration.

[0305] In some embodiments of the present application, the generator includes M reversible blocks connected in sequence, where M is a positive integer.

[0306] The processor 110 is specifically used to input the N first spectrum features and the N watermark features into a first reversible block among the M reversible blocks, and embed at least part of the features of each watermark feature into the corresponding first spectrum feature through the first reversible block; and input the N first spectrum features and the N watermark features output by the first reversible block into an i-th reversible block among the M reversible blocks, and embed at least part of the features of each watermark feature into the corresponding first spectrum feature through the i-th reversible block, where i is a positive integer greater than 1.

[0307] In some embodiments of the present application, the processor 110 is further used to obtain one-to-one corresponding T first sample spectral features and T sample watermark features before obtaining one-to-one corresponding N first spectral features and N watermark features, wherein the first sample spectral features are spectral features of a sample audio segment in the sample audio, and the sample watermark features are features of the sample watermark information, and T is a positive integer; and the T first sample spectral features and T sample watermark features are input into a watermark processing model, and each sample watermark feature is embedded into the corresponding first sample spectral feature through a generator to be trained of the watermark processing model to obtain N second sample spectral features; and based on the N second sample spectral features and the sample audio, generate a sample watermark audio; and, determine a first loss function according to the sample watermark audio and the sample audio, and use the first loss function to train the generator to be trained to obtain a generator, and the first loss function is used to characterize the audio type difference between the sample watermark audio and the sample audio.

[0308] In some embodiments of the present application, the processor 110 is further used to determine a second loss function based on the T first sample spectral features and the T sample watermark features before using the first loss function to train the generator to be trained, and the second loss function is used to characterize the difference between the T first sample spectral features and the T sample watermark features.

[0309] The processor 110 is specifically configured to train the generator to be trained using the first loss function and the second loss function.

[0310] In some embodiments of the present application, the processor 110 is specifically used to input N second sample spectral features into the discriminator of the watermark processing model, determine the audio type of the sample watermark audio through the discriminator; and determine the first loss function according to the audio type of the sample watermark audio and the audio type of the sample audio.

[0311] In some embodiments of the present application, the processor 110 is further used to input the N second sample spectral features into the discriminator of the watermark processing model, and before determining the audio type of the sample watermark audio and the audio type of the sample audio through the discriminator, input the N second sample spectral features and the spectral features of the sample watermark audio into the discriminator to be trained of the watermark processing model, and determine the training audio type of the sample watermark audio and the training audio type of the sample audio through the discriminator to be trained; and determine a third loss function according to the training audio type of the sample watermark audio, and determine a fourth loss function according to the training audio type of the sample audio, the third loss function is used to characterize the difference between the training audio type of the sample watermark audio and the audio type of the sample watermark audio, and the fourth loss function is used to characterize the difference between the training audio type of the sample audio and the audio type of the sample audio; and use the third loss function and the fourth loss function to train the discriminator to be trained to obtain the discriminator.

[0312] In some embodiments of the present application, the processor 110 is further used to obtain L third spectral features after generating watermark audio based on N second spectral features and audio, where the third spectral features are spectral features in the watermark audio, and L is a positive integer; and input the L third spectral features into a watermark processing model, determine the watermark features in each third spectral feature through a generator of the watermark processing model; and determine the watermark information based on the L watermark features.

[0313] In some embodiments of the present application, the generator includes M reversible blocks connected in sequence, where M is a positive integer.

[0314] The processor 110 is specifically used to input L third spectrum features and L noise features corresponding to each other into the Mth reversible block among M reversible blocks, and embed at least part of the watermark feature in each third spectrum feature into the corresponding noise feature through the Mth reversible block, where the noise feature is a feature of random noise; and input the L third spectrum features and L noise features output by the Mth reversible block into the Mjth reversible block among M reversible blocks, and embed at least part of the watermark feature in each third spectrum feature into the corresponding noise feature through the Mjth reversible block, where j is a positive integer less than M.

[0315] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0316] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memories.

[0317] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0318] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned audio processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0319] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0320] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned audio processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0321] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0322] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned audio processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0323] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0324] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0325] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An audio processing method, characterized in that: include: Obtaining N first spectrum features and N watermark features that correspond to each other one by one, wherein the first spectrum features are spectrum features of an audio segment in the audio, the watermark features are features of watermark information, and N is a positive integer; Inputting the N first spectral features and the N watermark features into a watermark processing model, and embedding each watermark feature into the corresponding first spectral feature through a generator of the watermark processing model to obtain N second spectral features; A watermark audio is generated based on the N second spectral features and the audio.

2. The method according to claim 1, characterized in that The step of generating a watermarked audio based on the N second frequency spectrum features and the audio comprises: Using each of the second frequency spectrum features to replace the corresponding first frequency spectrum features in the frequency spectrum features of the audio, respectively, to obtain replaced frequency spectrum features of the audio; The watermarked audio is generated based on the replaced spectral features.

3. The method according to claim 1, characterized in that Before obtaining the one-to-one corresponding N first spectrum features and N watermark features, the method further includes: From the audio, one audio segment is determined at every preset time interval, and the duration corresponding to one audio segment matches the preset duration.

4. The method according to claim 1, characterized in that The generator includes M reversible blocks connected in sequence, where M is a positive integer; The step of inputting the N first frequency spectrum features and the N watermark features into a watermark processing model, and embedding each watermark feature into a corresponding first frequency spectrum feature through a generator of the watermark processing model, comprises: Inputting the N first frequency spectrum features and the N watermark features into a first reversible block among the M reversible blocks, and embedding at least a part of each watermark feature into the corresponding first frequency spectrum feature through the first reversible block; The N first spectrum features and the N watermark features output by the first reversible block are input into the i-th reversible block among the M reversible blocks, and at least part of the features of each watermark feature is embedded into the corresponding first spectrum feature through the i-th reversible block, where i is a positive integer greater than 1.

5. The method according to any one of claims 1 to 4, characterized in that Before obtaining the one-to-one corresponding N first spectrum features and N watermark features, the method further includes: Obtaining T first sample spectrum features and T sample watermark features corresponding to each other, wherein the first sample spectrum features are spectrum features of a sample audio segment in the sample audio, the sample watermark features are features of sample watermark information, and T is a positive integer; Inputting T first sample spectrum features and T sample watermark features into the watermark processing model, and embedding each first sample watermark feature into the corresponding sample spectrum feature through the to-be-trained generator of the watermark processing model to obtain N second sample spectrum features; Generate a sample watermark audio based on the N second sample frequency spectrum features and the sample audio; A first loss function is determined according to the sample watermark audio and the sample audio, and the generator to be trained is trained using the first loss function to obtain the generator, wherein the first loss function is used to characterize the audio type difference between the sample watermark audio and the sample audio.

6. The method according to claim 5, characterized in that Before using the first loss function to train the generator to be trained, the method further includes: Determine a second loss function according to the T first sample spectrum features and the T sample watermark features, where the second loss function is used to characterize the difference between the T first sample spectrum features and the T sample watermark features; The using the first loss function to train the generator to be trained includes: The generator to be trained is trained using the first loss function and the second loss function.

7. The method according to claim 5, characterized in that The determining a first loss function according to the sample watermark audio comprises: Inputting the N second sample frequency spectrum features into the discriminator of the watermark processing model, and determining the audio type of the sample watermark audio through the discriminator; The first loss function is determined according to the audio type of the sample watermark audio and the audio type of the sample audio.

8. The method according to claim 7, characterized in that Before inputting the N second sample frequency spectrum features into the discriminator of the watermark processing model and determining the audio type of the sample watermark audio and the audio type of the sample audio by the discriminator, the method further comprises: Inputting the N second sample spectral features and the spectral features of the sample watermark audio into the to-be-trained discriminator of the watermark processing model, and determining the training audio type of the sample watermark audio and the training audio type of the sample audio through the to-be-trained discriminator; Determine a third loss function according to the training audio type of the sample watermark audio, and determine a fourth loss function according to the training audio type of the sample audio, wherein the third loss function is used to characterize the difference between the training audio type of the sample watermark audio and the audio type of the sample watermark audio, and the fourth loss function is used to characterize the difference between the training audio type of the sample audio and the audio type of the sample audio; The discriminator to be trained is trained using the third loss function and the fourth loss function to obtain the discriminator.

9. The method according to claim 1, characterized in that: After generating the watermark audio based on the N second frequency spectrum features and the audio, the method further includes: Obtain L third spectral features, where the third spectral features are spectral features in the watermarked audio, and L is a positive integer; Inputting L third spectrum features into a watermark processing model, and determining the watermark feature in each of the third spectrum features through a generator of the watermark processing model; The watermark information is determined based on the L watermark features.

10. The method according to claim 9, characterized in that The generator includes M reversible blocks connected in sequence, where M is a positive integer; The step of inputting the L third spectrum features into a watermark processing model and determining the watermark feature in each of the third spectrum features by a generator of the watermark processing model comprises: Inputting the L third spectrum features and the L noise features corresponding to each other into the Mth reversible block among the M reversible blocks, and embedding at least part of the watermark feature in each third spectrum feature into the corresponding noise feature through the Mth reversible block, wherein the noise feature is a feature of random noise; The L third spectral features and L noise features output by the Mth reversible block are input into the Mjth reversible block among the M reversible blocks, and at least part of the watermark feature in each of the third spectral features is embedded into the corresponding noise feature through the Mjth reversible block, where j is a positive integer less than M.

11. An audio processing device, characterized in that: The audio processing device comprises: an acquisition module and a processing module; The acquisition module is used to acquire N first spectrum features and N watermark features corresponding to each other, wherein the first spectrum features are spectrum features of the audio segment in the audio, and the watermark features are features of the watermark information, and N is a positive integer; The processing module is used to input the N first spectral features and N watermark features obtained by the acquisition module into the watermark processing model, embed each watermark feature into the corresponding first spectral feature through the generator of the watermark processing model, and obtain N second spectral features; and generate watermark audio based on the N second spectral features and the audio.

12. The device according to claim 11, characterized in that The processing module is specifically used to use each of the second spectrum features to replace the corresponding first spectrum features in the spectrum features of the audio, so as to obtain the replaced spectrum features of the audio; and generate the watermarked audio based on the replaced spectrum features.

13. The device according to claim 11, characterized in that The processing module is further configured to determine, from the audio, an audio segment at every preset time interval before the acquisition module acquires the one-to-one corresponding N first spectrum features and N watermark features, and the duration corresponding to one audio segment matches the preset duration.

14. The device according to claim 11, characterized in that The generator includes M reversible blocks connected in sequence, where M is a positive integer; The processing module is specifically configured to input the N first frequency spectrum features and the N watermark features into a first reversible block among the M reversible blocks, and embed at least part of each watermark feature into the corresponding first frequency spectrum feature through the first reversible block; The N first spectrum features and the N watermark features output by the first reversible block are input into the i-th reversible block among the M reversible blocks, and at least part of the features of each watermark feature is embedded into the corresponding first spectrum feature through the i-th reversible block, where i is a positive integer greater than 1.

15. The device according to any one of claims 11 to 14, characterized in that The acquisition module is further used to acquire one-to-one corresponding T first sample spectrum features and T sample watermark features before acquiring the one-to-one corresponding N first spectrum features and N watermark features, wherein the first sample spectrum features are spectrum features of sample audio segments in the sample audio, and the sample watermark features are features of sample watermark information, and T is a positive integer; The processing module is further used to input the T first sample spectral features and T sample watermark features acquired by the acquisition module into the watermark processing model, and embed each of the first sample watermark features into the corresponding sample spectral features through the generator to be trained of the watermark processing model to obtain N second sample spectral features; and generate sample watermark audio based on the N second sample spectral features and the sample audio; and determine a first loss function based on the sample watermark audio and the sample audio, and use the first loss function to train the generator to be trained to obtain the generator, wherein the first loss function is used to characterize the audio type difference between the sample watermark audio and the sample audio.

16. The device according to claim 15, characterized in that The processing module is further used to determine a second loss function according to the T first sample spectrum features and the T sample watermark features before using the first loss function to train the generator to be trained, wherein the second loss function is used to characterize the difference between the T first sample spectrum features and the T sample watermark features; The processing module is specifically used to train the generator to be trained using the first loss function and the second loss function.

17. The device according to claim 15, characterized in that The processing module is specifically used to input N second sample spectral features into the discriminator of the watermark processing model, determine the audio type of the sample watermark audio through the discriminator; and determine the first loss function according to the audio type of the sample watermark audio and the audio type of the sample audio.

18. The device according to claim 17, characterized in that The processing module is further configured to input the N second sample spectral features into the discriminator of the watermark processing model, and before determining the audio type of the sample watermark audio and the audio type of the sample audio through the discriminator, input the N second sample spectral features and the spectral features of the sample watermark audio into the discriminator to be trained of the watermark processing model, and determine the training audio type of the sample watermark audio and the training audio type of the sample audio through the discriminator to be trained; and determining a third loss function according to the training audio type of the sample watermark audio, and determining a fourth loss function according to the training audio type of the sample audio, wherein the third loss function is used to characterize the difference between the training audio type of the sample watermark audio and the audio type of the sample watermark audio, and the fourth loss function is used to characterize the difference between the training audio type of the sample audio and the audio type of the sample audio; And the discriminator to be trained is trained using the third loss function and the fourth loss function to obtain the discriminator.

19. The device according to claim 11, characterized in that The acquisition module is further configured to acquire L third spectrum features after the processing module generates the watermark audio based on the N second spectrum features and the audio, wherein the third spectrum features are spectrum features in the watermark audio, and L is a positive integer; The processing module is used to input the L third spectrum features acquired by the acquisition module into the watermark processing model, determine the watermark feature in each of the third spectrum features through the generator of the watermark processing model; and determine the watermark information based on the L watermark features.

20. The device according to claim 19, characterized in that The generator includes M reversible blocks connected in sequence, where M is a positive integer; The processing module is specifically used to input the L third spectrum features and L noise features corresponding to each other into the Mth reversible block among the M reversible blocks, and embed at least part of the watermark feature in each third spectrum feature into the corresponding noise feature through the Mth reversible block, wherein the noise feature is a feature of random noise; And the L third spectral features and L noise features output by the Mth reversible block are input into the Mjth reversible block among the M reversible blocks, and at least part of the watermark feature in each third spectral feature is embedded into the corresponding noise feature through the Mjth reversible block, where j is a positive integer less than M.

21. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the audio processing method according to any one of claims 1 to 10 are implemented.

22. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by a processor, the steps of the audio processing method according to any one of claims 1 to 10 are implemented.

Images