Audio summary model training method, system, electronic device and storage medium

Abstract

This invention discloses an audio summarization model training method, system, electronic device, and storage medium. One audio summarization model training method includes: simultaneously training a discriminator and an audio summarization model; wherein the discriminator is used to distinguish between the audio summarization output by the audio summarization model and the labeled text to obtain a condition c, wherein the condition c is used to characterize the similarity between the audio summarization and the labeled text; and performing adversarial training on the discriminator and the audio summarization model to enable the audio summarization model to output an audio summarization based on given conditions. The method of this application simultaneously trains a discriminator for the style of the output text and an audio summarization model. Through adversarial learning, the audio summarization model can be forced to output a summary based on given conditions. Subsequently, by collecting more stylistically diverse labeled text, more controllable, accurate, and diverse audio summarization generation can be achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of audio summary model training, and in particular relates to an audio summary model training method, a system, an electronic device and a storage medium. Background Art

[0002] Automatic audio summarization is a challenging task that requires identifying and understanding audio content and then summarizing it in natural language. The summary may include acoustic scenes, sound events, sound attributes, and even high-level abstract semantic information. Compared with structured label outputs, audio summarization is more relevant to the way humans process audio and is suitable for automatic content description or intelligent human-computer interaction applications.

[0003] Audio summarization has attracted a lot of attention in recent years. Researchers aim to improve the accuracy of descriptions by combining methods such as pre-training and additional keyword input. However, like most natural language generation tasks, audio summarization suffers from a lack of diversity. Systems trained through maximum likelihood estimation (MLE) tend to generate generic outputs, usually the most common patterns in the training corpus. In contrast, manual annotations describe the same audio clip in different styles, including sentence structure and word choice.

[0004] While some previous work has addressed the diversity issue, most of these works suffer from a decrease in accuracy while showing higher diversity. Some work focuses on generating more descriptive, contextually relevant outputs. For example, encouraging the system to put “a knife” instead of “a metal object” in an audio clip of sharpening a knife. This improves the overall diversity of the audio summaries because the descriptions are more detailed when the same number of summaries are generated for an audio clip. Other work strives to increase diversity by generating more outputs for a single input. Multiple outputs given the same input result in a higher diversity of input instances compared to a single-output system. The decrease in summary accuracy brought about by the improvement in diversity is particularly significant for work that promotes set diversity, such as GAN-based methods.

[0005] In audio summarization, Ikawa et al. proposed a system to control output specificity, that is, how specific the content of the generated summary is. The sum of inverse word frequencies is used as an indicator of sentence specificity. This work can generate multiple summaries with different given specificities. Although this statistical condition is simple and clear, it only captures word frequency features without considering patterns such as phrases and sentence structure. It may be the case that the model is encouraged to generate expressions that are not the most common, but are still common (i.e., "sub-common") in the training corpus. Summary of the invention

[0006] The embodiments of the present invention provide an audio summary model training method, system and device, which are used to solve at least one of the above technical problems.

[0007] In a first aspect, an embodiment of the present invention provides an audio summary model training method, comprising: simultaneously training a discriminator and an audio summary model, wherein the discriminator is used to discriminate an audio summary and annotated text output by the audio summary model to obtain a condition c, wherein the condition c is used to characterize the similarity between the audio summary and the annotated text; and adversarially training the discriminator and the audio summary model so that the audio summary model can output an audio summary of a given condition.

[0008] In a second aspect, an embodiment of the present invention provides an audio summary system, comprising: an audio encoder according to any embodiment of the present application; a text decoder according to any embodiment of the present application; and a discriminator according to any embodiment of the present application.

[0009] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the audio summary model training method of any embodiment of the present invention.

[0010] In a fourth aspect, an embodiment of the present invention further provides a computer program product, comprising a computer program stored on a non-volatile computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes the steps of the audio summary model training method of any embodiment of the present invention.

[0011] The method of the embodiment of the present application adopts a closed-loop learning approach, and simultaneously trains a discriminator for the style of the output text and an audio summary model. Through adversarial learning, the audio summary model can be forced to output a summary of given conditions. After that, by collecting annotated texts with more diverse styles, more controllable, accurate and diverse audio summary generation can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1A flowchart of an audio summary model training method provided by one embodiment of the present invention;

[0014] Figure 2 A neural condition summary framework proposed in an embodiment of the present application is provided for an embodiment of the present invention;

[0015] Figure 3 This is an accuracy-diversity curve on Clotho provided by an embodiment of the present invention.

[0016] Figure 4 An accuracy-diversity curve on Audiocaps provided by an embodiment of the present invention;

[0017] Figure 5 Detailed accuracy and diversity results of the proposed method using different inputs c on Clotho provided by an embodiment of the present invention;

[0018] Figure 6 An example of generating summaries with different inputs c by a neural conditional summary model provided by an embodiment of the present invention;

[0019] Figure 7 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Please refer to Figure 1 , which shows a flowchart of an embodiment of an audio summary model training method and system of the present application.

[0022] like Figure 1 As shown, in step 101, a discriminator and an audio summary model are trained simultaneously, wherein the discriminator is used to discriminate the audio summary and the annotated text output by the audio summary model to obtain a condition c, wherein the condition c is used to characterize the similarity between the audio summary and the annotated text;

[0023] In step 102, adversarial training is performed on the discriminator and the audio summary model so that the audio summary model can output an audio summary under given conditions.

[0024] The method of the embodiment of the present application adopts a closed-loop learning approach, and simultaneously trains a discriminator for the style of the output text and an audio summary model. Through adversarial learning, the audio summary model can be forced to output a summary of given conditions. After that, by collecting annotated texts with more diverse styles, more controllable, accurate and diverse audio summary generation can be achieved.

[0025] In some optional embodiments, the adversarial training of the discriminator and the audio summary model includes: performing adversarial training on the audio summary model and the neural network discriminator using the cross entropy loss between the audio summary output by the audio summary model and the annotated text and the conditional loss between the condition c and the true condition c. Thus, by adding an additional conditional loss, the summary model can be encouraged to generate audio summaries with corresponding specificity.

[0026] In some optional embodiments, the audio summary model includes an audio encoder and a text decoder, wherein the audio encoder is used to encode the input audio into an audio embedding sequence, and the text decoder generates word probabilities according to the audio embedding sequence and the condition c of the discriminator. Then, the final audio summary can be obtained according to the word probabilities.

[0027] In some optional embodiments, the adversarial training of the discriminator and the audio summary model includes: alternating between a first stage and a second stage during the adversarial training, wherein the first stage includes: updating only the parameters of the audio summary model, fixing the parameters of the discriminator, the audio summary model estimates word probabilities using audio input and the condition c, and is trained using the cross entropy loss and the conditional loss; the second stage includes: fixing the parameters of the audio summary model, updating only the parameters of the discriminator, and the discriminator is trained on the annotated text and the output of the audio summary model.

[0028] In some optional embodiments, the audio encoder includes a fixed 14-layer convolutional neural network feature extractor and a trainable three-layer bidirectional gated recurrent unit encoder. The text decoder includes a unidirectional single-layer gated recurrent unit. The text decoder uses an attention mechanism to aggregate the audio embedding sequence, and the discriminator includes encoding the input audio summary using a double-layer bidirectional long short-term memory. Thus, the audio encoder can learn the temporal dependencies between audio clips, the text decoder can better aggregate and decode the text, and the discriminator can better estimate the specificity of the input audio summary.

[0029] In some optional embodiments, the above method further includes the following inference stage: obtaining a condition c given by a user, inputting the audio embedding sequence and the condition c given by the user into the trained text decoder, and outputting an audio summary that meets the condition c given by the user. Thus, an audio summary that meets the condition c given by the user can be output.

[0030] The embodiment of the present application also provides an audio summary system, including: an audio encoder according to any one of the embodiments of the present application; a text decoder according to any one of the embodiments of the present application; and a discriminator according to any one of the embodiments of the present application.

[0031] It should be noted that the above method steps are not used to limit the execution order of each step. In fact, some steps may be executed simultaneously or in the opposite order to the specified step order, and this application has no limitation on this.

[0032] The following describes some problems encountered by the inventor in the process of implementing the present invention and a specific embodiment of the solution finally determined, so that those skilled in the art can better understand the solution of the present application.

[0033] In the process of implementing the present application, the inventors found that the prior art has the following defects: the audio summary model cannot control the characteristics of the output summary description. The conditions of the controllable audio summary model based on statistical conditions are fixed and do not consider the characteristics of the overall structure of the sentence.

[0034] When faced with the above defects, those skilled in the art may use more complex statistical information, such as the frequency of multi-word groups or document frequency information. Because the use of statistical information makes the model simpler, while the use of neural network output conditions requires an additional neural network, the model is more complex, the training is difficult, and the additional neural network has no training labels.

[0035] The method of the embodiment of the present application outputs the conditions of a sentence through a neural network, comprehensively considers the words, grammar, sentence structure and other information of the sentence, uses the assumption that "the output style of the model is relatively simple while manual annotation is relatively diverse", and uses automatic labels for training.

[0036] Please refer to Figure 2 , which shows the neural condition summary framework proposed in the embodiment of the present application.

[0037] like Figure 2As shown, during training, a neural network discriminator outputs the condition c of the annotated text. After the input audio is encoded by the audio encoder, the encoded result and the condition c are input into the text decoder together to obtain the output word probability. During training, the cross entropy loss between the word probability and the annotated text and the conditional loss between the decoding result and the true condition c are used for training;

[0038] During inference, the user gives a condition c, and the text decoder uses the encoded audio and condition c as input and outputs a summary text that meets the given condition.

[0039] In the process of implementing the embodiments of the present application, the inventors have also adopted the following alternative scheme: still using statistical conditions as input, but different from the original statistical conditions, the original statistical conditions are the sum of the inverse frequencies of each word in the sentence, and the alternative scheme is the average of the inverse frequencies of each word in the sentence. This scheme can avoid the model from overfitting to the output sentence length that meets the given conditions, but it is still a statistical condition and does not consider the influence of the sentence pattern.

[0040] In the process of implementing the embodiments of the present application, the inventors also used the following beta version: define a style number n, there are n trainable style vectors, a text encoder encodes the annotated text into a vector, calculates the similarity between this vector and the n style vectors, takes their weighted sum with the similarity as the weight, and uses this weighted sum as the style vector of the sentence annotation, which is input into the text decoder for decoding together with the encoded audio. The advantage of this beta version is that it can use a pre-trained text encoder, but the annotation style in the data set may not be obvious enough, so the disadvantage is that it may be difficult to automatically learn a better style vector.

[0041] This solution adopts a closed-loop learning approach, training a discriminator for the style of the output text and an audio summary model at the same time. Through adversarial learning, it can force the audio summary model to output a summary of given conditions. It is expected that in future research and products, more style-diverse annotated texts can be collected to achieve more controllable, accurate and diverse audio summary generation.

[0042] The following specific experiments and experimental data verify the beneficial effects of the embodiments of the present application compared with the prior art.

[0043] In recent years, with the introduction of techniques such as pre-training, transfer learning, and keyword input, audio summarization has made significant progress in generating accurate descriptions of audio events. However, current audio summarization tends to generate repetitive and generic sentences, which often contain the most common phrase collocations and sentence patterns in the training data. Some works in natural language generation improve diversity by focusing on specific content or increasing the number of generated summaries. However, these methods usually enhance diversity at the expense of description accuracy. In this work, we propose a new neural conditional summary model to balance diversity and accuracy. Compared with statistical conditioning, neural conditioning is the posterior given by a neural discriminator. Given the annotated condition, the summary model is trained to generate summaries with similar posterior. The summary model and the discriminator are trained in an adversarial manner. We evaluate the proposed method on Clotho and Audiocaps. The results show that our method can improve the diversity of the output at the cost of minimal accuracy drop compared to the baseline.

[0044] 1. Introduction

[0045] To circumvent the problem of statistical conditioning, we propose a new conditional audio summarization system where the conditioning is provided by a neural network that uses a discriminator to tell whether the summary is generated by a human or a model. The discriminator and the summary model are trained in an adversarial manner to penalize the generation of sub-generic patterns. Compared to GAN-based methods, we incorporate MLE when training the summary model to ensure summary accuracy. By varying the conditioning, we control the specificity of the descriptions and thus the global diversity of the output set. Experiments are conducted on the benchmark datasets Clotho and Audiocaps. Our system achieves the best diversity-accuracy trade-off compared to model-independent methods and statistical conditioning methods. The global diversity of the output set can be controlled by the conditioning, while our method has minimal impact on the summary accuracy.

[0046] 2. Neural Conditioning Audio Summary

[0047] In this section, we first briefly outline our proposed neural conditional audio summarization system. Then we describe each part of the system, including the audio encoder, neural discriminator, and text decoder. Finally, we introduce the adversarial training strategy.

[0048] 2.1. System Overview

[0049] Figure 2 The neural conditional summary framework proposed in the embodiment of the present application is shown. During training, the text decoder takes the annotated conditions and the encoded audio embedding as input to give word probabilities. During reasoning, we manually input the conditions into the text decoder to control the diversity of the output set.

[0050] like Figure 2 As shown in Figure 2, our proposed system generates descriptions using two inputs: an input audio sequence and a conditional embedding. The audio encoder converts the input features into an embedding sequence (T is the embedding sequence length).

[0051] The text decoder generates word probabilities based on the audio embedding sequence and the conditional embedding. The conditional embedding is a weighted sum of two trainable embeddings, where the weight is the conditional value c∈[0,1]. c=0 indicates a generic output style, while c=1 encourages the system to generate content-specific descriptions. During training, the reference caption is fed into the neural discriminator to obtain c. During inference, c is manually fed in to generate descriptions with the corresponding style (generic or specific).

[0052] 2.2. Audio Codec

[0053] Since the features extracted by pre-trained deep neural networks show better performance and effectiveness in training, we adopt PANNs as feature extractors in this work. Specifically, we use a pre-trained 14-layer convolutional neural network (CNN14). The feature map before the global pooling layer is taken as input. Since mean max pooling is performed between convolutional blocks in CNN14, each frame of the feature sequence represents a small segment of the original audio clip. We use a three-layer bidirectional gated recurrent unit (GRU) as an audio encoder to learn the temporal dependencies between these segments. Taking the original audio signal as input, the audio encoder consists of a fixed CNN14 feature extractor and a trainable GRU encoder.

[0054] 2.3. Neural Discriminator

[0055] The neural discriminator takes the input audio summary and estimates its specificity c. The summary can be either human-generated or machine-generated. We encode the input caption using two layers of bidirectional long short-term memory (LSTM). The last time step of the hidden is represented as a sequence and converted to a single value through a linear layer. Finally, c is obtained after sigmoid activation.

[0056] 2.4. Text Decoder

[0057] After obtaining c from reference (training) or manual assignment (inference), the conditional vector c is generated by combining the two style vectors:

[0058] c=c·e specific +(1-c)·e generic

[0059] We use a unidirectional single-layer GRU as a text decoder to estimate the given audio embedding sequence The word probabilities of and c are calculated by using the attention mechanism

[16] . Aggregation is performed. At each time step n, the hidden state is based on the input word w n , aggregate audio embedding and c to update:

[0060]

[0061]

[0062] o n =Linear(h n )

[0063] The word embedding layer WE transforms w n Converted to a continuous vector. The output of the last linear layer is The probability vector of the current time step on , where V is the vocabulary.

[0064] 2.5. Adversarial Training

[0065] Since we use a neural network to provide the conditional c, labels are needed to train the conditional network. Based on the observation that the system output is usually a generic description while the human summary is usually more diverse, we use such intuitive labels: the human summary is a positive sample, while the system output is a negative sample. Therefore, c of the input title is the posterior given by the discriminator. It can be regarded as a "human-like" score that measures how similar the title is to the human summary.

[0066] The encoder-decoder summary network and the conditional network are trained in an adversarial manner. During training, the following two phases are alternated. In the first phase, only the summary model parameters are updated, while the discriminator is fixed. The summary model estimates the word probability at each time step using the audio input A and the reference specificity c. Then the standard cross entropy (CE) loss is calculated as follows:

[0067]

[0068]

[0069]

[0070] Where Enc, Dec, and Dis represent encoder, decoder, and discriminator, respectively. In addition to the CE loss, an additional conditional loss L is added condition , to encourage the summary model to generate descriptions with corresponding specificity. It is the decoded summary specificity The binary cross entropy loss between and c. The total summary loss is L CE and L condition The weighted sum of:

[0071]

[0072]

[0073]

[0074] in is the title generated by the model, and λ is a hyperparameter. We use the reparameterization trick during greedy decoding to ensure that the network can be trained by back-propagation.

[0075] In the second stage, the summary network is fixed, while the discriminator is trained on human summaries and summary network outputs. For an input sentence s, the conditional network training loss is calculated as follows:

[0076] c=Dis(s)

[0077]

[0078] If s is manually labeled, the label y=1, otherwise y=0.

[0079] 3. Experimental Setup

[0080] 3.1. Dataset

[0081] The experiments are conducted on the benchmark audio summarization datasets Clotho and Audiocaps. We use the latest version of Clotho 2.1, which contains about 6k audio clips. Audiocaps contains about 50k audio clips. We use the official training, validation, and test splits for both datasets.

[0082] 3.2. System Configuration

[0083] We follow the same configuration of CNN14 in PANNs

[15] to extract audio features. During the training of the summary model, special tokens <bos>and <eos>is added to the beginning and end of each summary. The baseline sequence-to-sequence summarization model without conditioning input was trained for 25 epochs with a maximum learning rate of 5×10-4. The discriminator was also pre-trained on a mixture of baseline model outputs and human annotations. The neural conditioning model was then initialized with the baseline model parameters and trained for 20 epochs with a maximum learning rate of 2×10-4. In the first 15 epochs, only CE loss (λ=0) was used and the discriminator was fixed. The entire model was then trained with λ set to 5×10-5 for the remaining five epochs. During the training of the baseline and neural conditioning models, the learning rate was increased linearly to a maximum value over the total iterations and then exponentially decayed to 5×10-7. We use a random weight average so that the model parameters over the past five epochs are averaged for evaluation.

[0084] Evaluation Metrics

[0085] The evaluation is performed from two aspects: accuracy and diversity. Standard metrics including BLEU1-4, ROUGEL, METEOR, CIDEr, and SPICE are used to evaluate summary accuracy. SPIDEr is the average of CIDEr and SPICE and is regarded as an overall accuracy metric because it relies on n-grams and semantic similarity. Following previous work, we evaluate diversity through distinct-1, distinct-2, and Self-BLEU. The vocabulary size of the output summary is also calculated to measure diversity. For these diversity metrics, higher values ​​indicate better diversity except self-BLEU. Among them, we choose Self-BLEU as a representative because it measures the {1-4}-gram overlap between generated summaries, while distinct-1 and distinct-2 only consider unigrams and bigrams.

[0086] 4. Results and Analysis

[0087] 4.1. Comparison with Baselines

[0088] We first compare our proposed neural conditioning method with several baselines: a) model-agnostic methods, including beam search and diversified beam search; and b) statistical conditioning. Each compared method is independently evaluated by diversity and accuracy metrics. For fair comparison, we use beam search with a beam size of 5 in all methods. The results of one beam form one output set. In different beam searches, the 1-best results of each set form one output set. For both statistical and proposed neural conditioning, each output set contains the results produced by one input conditioning.

[0089] Figure 3 Accuracy-diversity curves on Clotho are shown.

[0090] Figure 4 Shows the accuracy-diversity curve on Audiocaps

[0091] The results are as follows Figure 3 and Figure 4 As shown. We use (1-SelfBLEU) as the diversity metric in the curve, so higher values ​​indicate better diversity. The summaries generated from different beams in beam search have little change in diversity, while the accuracy decreases slightly from the first beam to the last beam. The output titles from diversity beam search are more diverse than other methods. However, this method achieves better diversity by sacrificing accuracy. When generating a title for the current group, it penalizes words that appear in previous groups, which tend to be more accurate. This shows that previous work on improving the diversity of output instances fails to generate accurate summaries as the diversity of the output set increases.

[0092] The summaries generated by the statistical conditional model perform the worst in terms of accuracy and diversity. We find that summaries generated under high statistics conditions mostly contain repeated nonsense words (e.g., "and", "is", "a"), resulting in low accuracy and diversity. In contrast, our proposed neural conditional summarization model is able to control output diversity without sacrificing accuracy. On Clotho, the diversity variance of the neural conditional output is similar to that of diversity beam search, while the summary accuracy remains almost unchanged. On Audiocaps, the summary accuracy decreases slightly when c=1.0, but the degradation is still minimal while achieving the same diversity score. Clotho contains five reference summaries (possibly with different styles) for each training audio clip, while Audiocaps contains one. This may make it more difficult for the model to learn the mapping from c to the output style of Audiocaps.

[0093] Figure 5 A screenshot of Table 1 of the present application embodiment is shown, which characterizes the detailed accuracy and diversity results of the proposed method on Clotho using different inputs c. "#Vocabulary" represents the output vocabulary. For all indicators except Self-BLEU, higher is better.

[0094] 4.2. Accuracy-diversity trade-off

[0095] To analyze the impact of different input c on the generated summaries, we list the detailed accuracy and diversity results of Clotho with different input c in Table 1. All diversity metrics improve with higher c. Although the differences in the accuracy metrics are small, we can find a trend that higher c leads to lower BLEU, ROUGEL, and METEOR but higher SPICE. This indicates that the summaries generated by higher c have less overlap with the reference n-grams, but better semantic-level accuracy (objects, attributes, and relations). With high input c, the model is encouraged to generate n-grams that are different from those in the reference while keeping the semantic content the same. c = 0.4 achieves the highest CIDEr, indicating that a moderate c leads to the best TF-IDF-based similarity. The results on Audiocaps show a similar trend and are therefore not listed here.

[0096] Figure 6 shows a screenshot of Table 2, which features examples of summaries generated by the neural conditional summarization model with different inputs c.

[0097] To give an intuition of the results, we show examples of summaries generated with different input c on the same audio in the Clotho evaluation split in Table 2. High c leads to more detailed descriptions: “wooden door” instead of “hard surface” and the additional object “windshield”. Thus, the output set diversity improves. In contrast, low c leads to general descriptions that are likely to be correct but contain less information.

[0098] 5. Conclusion

[0099] This paper aims to control the diversity of the set of audio summarization outputs. A neural conditional summarization model is proposed to generate generic or specific outputs with different conditioning signals c. During training, the annotated conditions are provided by a neural discriminator, which is trained in an adversarial manner using the summarization model. Experiments on Clotho and Audiocaps show that the proposed method can generate summaries with different set diversity. Compared with beam search, diversity beam search, and statistical conditioning, the neural conditioning method has the least impact on the accuracy of generating outputs with the same diversity. Detailed accuracy metrics also show that summaries with higher input c have less overlap with the reference n-grams but contain the correct audio content.

[0100] In some other embodiments, the embodiments of the present invention further provide a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the audio summary model training method and system in any of the above method embodiments;

[0101] As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows:

[0102] Training a discriminator and an audio summary model simultaneously, wherein the discriminator is used to discriminate the audio summary and the annotated text output by the audio summary model to obtain a condition c, wherein the condition c is used to characterize the similarity between the audio summary and the annotated text;

[0103] The discriminator and the audio summary model are subjected to adversarial training so that the audio summary model can output an audio summary under given conditions.

[0104] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the audio summary model training method and system, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the audio summary model training method and system via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0105] An embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any one of the above-mentioned audio summary model training methods and systems.

[0106] Figure 7 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, the device includes: one or more processors 710 and a memory 720, Figure 7 A processor 710 is used as an example. The device of the audio summary model training method and system may also include: an input device 730 and an output device 740. The processor 710, the memory 720, the input device 730 and the output device 740 may be connected via a bus or other means. Figure 7 The example of connecting via a bus is taken. The memory 720 is the above-mentioned non-volatile computer-readable storage medium. The processor 710 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 720, that is, the audio summary model training method and system of the above-mentioned method embodiment are realized. The input device 730 can receive input digital or character information, and generate key signal input related to user settings and function control of the communication compensation device. The output device 740 may include display devices such as display screens.

[0107] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0108] As an implementation mode, the electronic device is applied to an audio summary model training method and system, and is used for a client, and includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can:

[0109] Inputting the acquired speech signal into a preprocessing module, wherein the preprocessing module includes framing and signal transformation of the speech signal;

[0110] Inputting the output of the preprocessing module into an acoustic feature prediction network to obtain predicted acoustic features of clean speech;

[0111] The acoustic features of the clean speech are input into a neural network vocoder model to obtain a waveform of the clean speech.

[0112] The electronic device of the embodiment of the present application exists in various forms, including but not limited to:

[0113] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0114] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0115] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0116] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0117] (5) Other electronic devices with data interaction functions.

[0118] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0119] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.< / eos> < / bos>

Claims

1. A method for training an audio summary model, comprising: Training a discriminator and an audio summary model simultaneously, wherein the discriminator is used to discriminate the audio summary and the annotated text output by the audio summary model to obtain a condition, wherein the condition is used to characterize the similarity between the audio summary and the annotated text; The discriminator and the audio summary model are subjected to adversarial training so that the audio summary model can output an audio summary of a given condition, wherein the adversarial training of the discriminator and the audio summary model includes: using the cross entropy loss between the audio summary output by the audio summary model and the annotated text and the conditional loss between the condition and the true condition to perform adversarial training on the audio summary model and the neural network discriminator, the audio summary model includes an audio encoder and a text decoder, the audio encoder is used to encode the input audio into an audio embedding sequence, and the text decoder generates word probabilities according to the audio embedding sequence and the condition of the discriminator; the adversarial training of the discriminator and the audio summary model also includes: using the first stage and the second stage alternately in the adversarial training process, wherein the first stage includes: only updating the parameters of the audio summary model, fixing the parameters of the discriminator, the audio summary model uses the audio input and the condition to estimate the word probability, and uses the cross entropy loss and the conditional loss for training; the second stage includes: fixing the parameters of the audio summary model, only updating the parameters of the discriminator, and the discriminator is trained on the annotated text and the output of the audio summary model.

2. The method according to claim 1, wherein: The audio encoder includes a fixed 14-layer convolutional neural network feature extractor and a trainable three-layer bidirectional gated recurrent unit encoder, the text decoder includes a unidirectional single-layer gated recurrent unit, and the text decoder uses an attention mechanism to aggregate audio embedding sequences.

3. The method according to claim 1, wherein: The discriminator includes encoding the input audio summary using a two-layer bidirectional long short-term memory.

4. The method according to claim 1, further comprising: Acquire the conditions given by the user, input the audio embedding sequence and the conditions given by the user into the trained text decoder, and output the audio summary that meets the conditions given by the user.

5. An audio summarization system, executing the audio summarization model training method according to any one of claims 1 to 4.

6. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 4.

7. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images