SOUND DESCRIPTION GENERATION METHOD, SOUND DESCRIPTION GENERATION DEVICE, AND PROGRAM
A cascaded system with guidance caption retrieval and caption generation modules using pre-trained language models addresses the challenge of limited data in sound description generation, ensuring high-accuracy results.
Patent Information
- Application Number
- JP2022563312
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-11-18
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-11-18
AI Technical Summary
Conventional encoder-decoder frameworks for sound description generation require large amounts of training data, which are often not available, leading to inaccurate results due to insufficient data, as exemplified by the AudioCaps dataset with only 49,838 training descriptions compared to 36 million in WMT2014.
A cascaded system comprising a guidance caption retrieval module and a caption generation module, utilizing pre-trained language models like GPT, where the guidance caption retrieval module identifies similar sounds from training data and provides guidance for the caption generation module to produce accurate descriptions without directly inputting acoustic features into the language model.
Generates accurate sound descriptions even with limited training data by leveraging pre-trained language models effectively, achieving comparable results to conventional methods with carefully designed neural networks.
Smart Images

Figure 0007593413000011 
Figure 0007593413000012 
Figure 0007593413000013
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for generating natural sentences that explain acoustic signals. [Background technology]
[0002] Audio captioning is a process of generating a description of an audio signal using a sequence of observed acoustic features Φ=(φ1, …, φ T ) into a sequence of (sub)word tokens (w1, …, w N ), where φ t ∈R Da (t=1, …, T) is the acoustic feature vector for time index t, T is the number of time frames in the observed acoustic feature sequence, and D a represents the number of dimensions of the acoustic features. n ∈N (n=1, …, N) is the index of the nth token, and N is the length of the (sub)word token sequence.
[0003] In addition, the symbol " → " and the like should be written directly above the character immediately preceding it, but due to limitations in text notation, they are written immediately after the character in question. In mathematical formulas, these symbols are written in their proper position, that is, directly above the character.
[0004] In many previous studies, the above conversion problem has been solved by an encoder-decoder framework using a deep neural network (DNN) (see Non-Patent Documents 1, 2). First, the encoder converts the acoustic feature sequence Φ into a vector ν in another feature space, and the decoder combines the vector ν and the (n-1)th output words w1, …, w n-1 With reference to the nth output word w n Estimate.
[0005]
number
[0006] Here, w → n-1 =(w1, …, w n-1 ), and w n is the posterior probability p(w n |Φ, w → n-1 ) using beam search or the like (see Non-Patent Document 2). [Prior art documents] [Non-patent literature]
[0007] [Non-Patent Document 1] Y. Koizumi, R. Masumura, K. Nishida, M. Yasuda, and S. Saito, “A Transformer-based Audio Captioning Model with Keyword Estimation,” in Proc. Interspeech, 2020. [Non-Patent Document 2] D. Takeuchi, Y. Koizumi, Y. Ohishi, N. Harada, and K. Kashino, “Effects of Wordfrequency based Pre- and Post- Processings,” in Proc. Detect. Classif. Acoust. Scenes Events (DCASE) Workshop, 2020. [Non-Patent Document 3] CD Kim, B. Kim, H. Lee, and G. Kim, “AudioCaps: Generating Captions for Audiosin The Wild,” in Proc. N. Am. Chapter Assoc. Comput. Linguist.: Hum. Lang. Tech. (NAACL-HLT), 2019. Summary of the Invention [Problem to be solved by the invention]
[0008] In order to build a conventional encoder-decoder framework using a large-scale deep neural network, a large amount of training data is required. However, in audio description generation, the available training data is often smaller than in other tasks. In fact, AudioCaps (see Non-Patent Document 3), a representative dataset for audio description generation, contains only 49,838 training descriptions. This is only about 1 / 1000 of the amount of data compared to the approximately 36 million training sentence pairs contained in WMT2014, a dataset for English-French machine translation.
[0009] In view of the above technical problems, an object of the present invention is to generate an explanation of an audio signal with high accuracy even when there is a small amount of training data. [Means for solving the problem]
[0010] In order to solve the above problems, one embodiment of a sound description generation method of the present invention is a sound description generation method for generating a description of a target sound, in which a guidance description search unit acquires multiple descriptions corresponding to an acoustic signal similar to the target sound, and a description generation unit generates a description of the target sound by determining words in order from the beginning based on the acquired descriptions. Effect of the Invention
[0011] According to the present invention, even when there is a small amount of training data, it is possible to generate an explanation for an audio signal with high accuracy. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is a conceptual diagram for explaining an overview of the present invention. [Diagram 2] 2A is a diagram illustrating a procedure for labeling training data as similar / dissimilar, and FIG 2B is a diagram illustrating a procedure for training a model for searching guidance explanatory sentences. [Diagram 3]FIG. 3 is a diagram illustrating a procedure for generating a description of an input audio signal. [Figure 4] FIG. 4 is a diagram illustrating an example of the functional configuration of the sound description generation device. [Diagram 5] FIG. 5 is a diagram illustrating a processing procedure of the sound description generation method. [Figure 6] FIG. 6 is a diagram illustrating an example of the functional configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] [Summary of the invention] One promising strategy when the amount of training data is insufficient is the use of pre-trained models. In tasks such as acoustic event detection and scene classification, pre-trained models such as VGGish (see Reference 1) have been announced to obtain better results with less training data. Similarly, in natural language processing, large-scale pre-trained language models such as BERT (Bidirectional Encoder Representations from Transformers) (see Reference 2) and GPT (Generative Pre-trained Transformer) (see Reference 3) have improved the performance of various tasks. In particular, an autoregressive pre-trained language model such as GPT is closely related to the decoder of equation (2), so that its use is expected to improve the accuracy of sound description generation. Hereinafter, an autoregressive pre-trained language model such as GPT is simply referred to as a "pre-trained language model." The point of the present invention is to generate sound description sentences using such a pre-trained language model.
[0014] [Reference 1] S. Hershey, S. Chaudhuri, DPW Ellis, JF Gemmeke, A. Jansen, RC Moore, M. Plakal, DvPlatt, RA Saurous, B. Seybold, M. Slaney, R. Weiss, and K. Wilson, “CNN Architectures for LargeScale Audio Classification,” in Proc. Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2017. [Reference 2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proc. N. Am. Chapter Assoc. Comput. Linguist.: Hum. Lang. Tech. (NAACL-HLT), 2019. [Reference 3] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. “Language models are unsupervised multitask learners,” Tech. rep., OpenAI, 2019.
[0015] However, such a pre-trained language model cannot be directly applied to phonetic description generation because the pre-trained language model is p(w n | w → n-1), and the feature Φ extracted from the acoustic signal cannot be directly input. Therefore, the specific problem to be solved by this invention is "How can we input the original modal information into the pre-trained language model in order to benefit from the pre-trained language model in cross-modal translation tasks such as sound description generation?"
[0016] The present invention solves the above problems as follows. The present invention is a cascade type system composed of two modules, as shown in FIG. 1. The first module, "Guidance caption retrieval," operates like an encoder in the conventional method. This module evaluates the similarity between the input sound and the sounds in the learning data, and obtains and outputs multiple captions that are assigned to "similar sounds" in the learning data. Hereinafter, this caption is referred to as a "guidance caption." The second module, "Caption generation," operates like a decoder in the conventional method. This module generates a caption for the input sound using a pre-trained language model while referring to the guidance caption. This configuration eliminates the need to directly input voice into the pre-trained language model, and makes it possible to use it for generating sound captions.
[0017] <Search for guidance explanations> The purpose of the first module, "guidance description search," is to obtain guidance descriptions from training data based on sound similarity (hereinafter referred to as "sound similarity"). Here, sound similarity does not simply mean calculating the similarity between sound features. For example, the sound features of a police car siren and an ambulance siren are similar, but the words used in the descriptions are "police car" and "ambulance," respectively, which are completely different. That is, the sound similarity used in the present invention must take a high value when the descriptions of two sounds are similar, even if the corresponding sounds are not similar. To achieve this requirement, the training of this module consists of (a) calculating the sentence similarity between the descriptions included in the training data set (see FIG. 2A), and (b) training a deep neural network that predicts the similarity from two sounds (see FIG. 2B).
[0018] First, we will explain the labeling in (a). Various methods have been proposed to measure the similarity of explanatory sentences, such as BLEU, CIDEr, and SPICE. In the present invention, any of these may be used, but in this example, we will use BERTScore (see Reference 4) for the explanation.
[0019] [Reference 4] T. Zhang, V. Kishore, F. Wu, KQ Weinberger, and Y. Artzi, “BERTScore: Evaluating Text Generation with BERT,” in Proc. of Int. Conf. Learn. Representations (ICLR), 2020.
[0020] The detailed procedure for calculating the similarity between explanatory sentences is as follows. 1. Calculate the BERTScore between all the sentences in the training dataset. 2. After obtaining all the BERTScores in the training dataset, normalize them so that their maximum and minimum values are 1 and 0, respectively. 3. Descriptions that exceed the threshold are labeled as “Similar,” and other descriptions are labeled as “Not similar.” The threshold can be set to, for example, 0.7.
[0021] Next, we will explain the model learning in (b). First, we use some kind of deep neural network to convert the time domain observation signal x into a vector with fixed dimensions. Similarly, all the sound data included in the training data set are also converted into vectors with fixed dimensions. Then, we calculate the distance between these vectors using some kind of function, and the explanations given to the top K training data with the smallest distance are called w →ref The distance can be anything, but for example, the L2 distance D(a, b) = ||ab||2 2 etc. can be used.
[0022] This deep neural network can be implemented in any way. As an example, we will explain how to combine pre-trained VGGish with a trainable embedding network.
[0023] First, the time-domain observed signal x is converted into an acoustic feature sequence Φ using pre-trained VGGish.
[0024]
number
[0025] where Φ∈R Da ×T, and for VGGish it is D. a = 128. VGGish does not require parameter updates during training.
[0026] Next, we use an embedding network to convert the acoustic feature sequence Φ into e.
[0027]
number
[0028] Here, Embed(·) is the embedding network, which can be, for example, a Transformer-encoder layer (see Reference 5).
[0029] [Reference 5] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, AN Gomez, L. Kaiser, and I. Polosukhin, “Attention Is All You Need,” in Proc. Adv. Neural Inf. Process. Syst. (NIPS), 2017.
[0030] Finally, let e be the dimension of R Da×T and normalize e to be |e|=1.
[0031] To train the embedded network, a method such as triplet loss training (see Reference 6) can be used.
[0032] [Reference 6] J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and Y. Wu. “Learning Fine-grained Image Similarity with Deep Ranking,” in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit. (CVPR), 2014.
[0033] In this training, a deep neural network is trained to minimize the following cost function:
[0034]
number
[0035] Here, α is a margin coefficient, which can be set to 0.3 or the like. Also, e a , e p , e n are calculated from sounds called anchor, positive, and negative, respectively. Here, anchor is the input sound (which can also be considered a search key for searching for sounds similar to this sound), positive is a sound corresponding to a description selected at random from among descriptions labeled "Similar" to the anchor, and negative is a sound selected at random from among descriptions labeled "Not similar" to the anchor that meets the following criteria:
[0036]
number
[0037] <Description Generation> Figure 3 shows the processing procedure of the second module, "explanation generation." First, →ref (vector consisting of guidance text) and w → n-1 (a vector consisting of the (n-1)th output words) is input independently to a pre-trained language model. In the present invention, any pre-trained language model may be used, but here, the description will be given assuming that GPT-2 is used.
[0038]
number
[0039] Here, Ψ refs ∈R Dl×M , Ψ hyps n-1 ∈R Dl×(n-1) In the case of the general GPT-2 called “117M”, D l =768.
[0040] Next, to integrate these two pieces of information, we use the multi-head attention (MHA) layer disclosed in Reference 5 above.
[0041]
number
[0042] Here, MultiHeadAttention(a, b) is a multi-head attention mechanism layer, where a is used as the query and b is used as the key and value. n-1 ∈R Dl×(n-1) It is.
[0043] Furthermore, we use a multi-headed attention layer to integrate the features extracted from the observed signal. Here, we reduce the number of parameters by denoting Φ and Ψ n-1 The dimension of the reduced Ψ' n-1 and Φ' are input to the multi-headed attention layer. Dimensionality reduction is performed using a fully connected layer, for example, D r Reduce the dimension to =60.
[0044]
number
[0045] Here, Linear(·) is the number of dimensions of the output of the multi-head attention layer, R Dl This is a fully connected layer to increase the
[0046] Finally, Ψ n-1 and Y n-1 Using the features obtained by adding n |Φ, w → n-1 ) to predict.
[0047]
number
[0048] where LMHead(·) is a pre-trained token prediction layer that is trained to adapt the decoder to the statistics of words available in the training data descriptions.
[0049] <Learning Procedure> The learning procedure of each module used in the sound description generation of the present invention will be described.
[0050] Step 1: Prepare paired data of time-domain acoustic signals and corresponding explanatory text. Hereafter, this will be called the "training dataset."
[0051] Step-2: Calculate the similarity such as BERTScore for all combinations of the descriptions included in the training dataset. If the number of paired data included in the training dataset is P, then this combination will be P × (P-1). After that, using the method described in <Guidance Description Search>, label the description of the pth paired data in the training dataset and the descriptions of the other P-1 paired data as "Similar" or "Not similar."
[0052] Step 3: Train the deep neural network for guidance explanation search using the method described in <Guidance explanation search>. The batch size should be around 128.
[0053] Step-4: Learn the explanatory sentence generation module. During learning, →ref For the , we can use K descriptions randomly selected from descriptions labeled as “Similar”, or we can use K descriptions retrieved by searching for guidance descriptions. Here, we set K to about 5. In addition, any cost function can be used when learning the whole, for example, p(w n |Φ, w → n-1 ) and w nThe cross entropy between layers can be used. The batch size can be set to about 512. The pre-trained language model can be updated simultaneously with other layers, or it can be fixed.
[0054] [Embodiment] Hereinafter, an embodiment of the present invention will be described in detail. In the drawings, components having the same functions are designated by the same reference numerals, and duplicated explanations will be omitted.
[0055] An embodiment of the present invention is a sound description generation device and method that generates a description that explains an input sound signal from the input sound signal using a learning data set consisting of a pair of a previously prepared sound signal and a corresponding description. As shown in Fig. 4, the sound description generation device 1 of the embodiment includes, for example, a learning data storage unit 10, a sound similarity calculation unit 11, a guidance description search unit 12, and a description generation unit 13. The sound description generation device 1 executes each step shown in Fig. 5 to realize the sound description generation method of the embodiment.
[0056] The sound description generating device 1 is a special device configured by loading a special program into a publicly known or dedicated computer having, for example, a central processing unit (CPU), a main storage device (RAM), etc. The sound description generating device 1 executes each process under the control of, for example, the central processing unit. Data input to the sound description generating device 1 and data obtained in each process are stored in, for example, the main storage device, and the data stored in the main storage device is read out to the central processing unit as necessary and used for other processes. At least a part of each processing unit of the sound description generating device 1 may be configured by hardware such as an integrated circuit. Each storage unit provided in the sound description generating device 1 can be configured by, for example, a main storage device such as a random access memory (RAM), an auxiliary storage device configured by a semiconductor memory element such as a hard disk, an optical disk, or a flash memory, or middleware such as a relational database or a key-value store.
[0057] Hereinafter, with reference to FIG. 5, a sound description generation method executed by the sound description generation device 1 of the embodiment will be described.
[0058] A learning data set consisting of a plurality of pieces of learning data is stored in the learning data storage unit 10. Each piece of learning data consists of a pre-collected acoustic signal and a manual explanation of the correct answer for each acoustic signal.
[0059] In step S10, an audio signal for which an explanation is to be generated is input to the sound explanation generation device 1. Hereinafter, this audio signal is referred to as a “target sound.” The input target sound is input to the sound similarity calculation unit 11.
[0060] In step S11, the sound similarity calculation unit 11 reads out the learning data set stored in the learning data storage unit 10, and calculates the sound similarity between the input target sound and each audio signal included in the learning data set. The calculated sound similarity is as described in the above <Guidance explanation search>, and is configured such that the more similar the explanation given to the first audio signal is to the explanation given to the second audio signal, the more likely it is that the first audio signal and the second audio signal are determined to be similar. The similarity between the explanations can be, for example, BERTScore. The sound similarity calculation unit 11 inputs the calculated sound similarity to the guidance explanation search unit 12.
[0061] In step S12, the guidance description search unit 12 acquires a plurality of pieces of learning data similar to the input target sound based on the sound similarity calculated by the sound similarity calculation unit 11. The guidance description search unit 12 inputs the description included in the acquired learning data to the description generation unit 13 as the guidance description.
[0062] In step S13, the description generation unit 13 generates a description of the target sound by determining words in order from the beginning based on the guidance description acquired by the guidance description search unit 12. The description generation method is as described above in <Description Generation>, and the guidance description, the generated word string from the beginning to the immediately preceding word, and the acoustic features of the target sound are integrated by a pre-trained language model and a multi-head attention mechanism to repeatedly determine the current word. The description generation unit 13 outputs the generated description of the target sound to the sound description generation device 1.
[0063] [Experimental Results] In order to confirm the effects of the invention, an experiment was conducted using the AudioCaps dataset (Non-Patent Document 3). The experimental results are shown in Table 1. In the "Method" column, "Conventional" is the evaluation result of the conventional technology described in Non-Patent Document 3, and "Ours" is the evaluation result of the present invention. The evaluation indices used in Non-Patent Document 3 were BLEU1, BLEU2, BLEU3, BLEU4 (denoted as B-1, B-2, B-3, B-4 in the table), METEOR, CIDEr, ROUGE, and BLEU3. L (shown as ROUGE-L in the table), SPICE was used. The comparison method was TopDown-AlignedAtt (1NN), which has the highest accuracy among the methods used in Non-Patent Document 3.
[0064] [Table 1]
[0065] The present invention only adds a few trainable layers after the pre-trained language model. Nevertheless, as shown in Table 1, the evaluation results were comparable to those of the conventional technology using a carefully designed deep neural network architecture in Non-Patent Document 3. This demonstrates that the present invention can generate explanatory text from audio signals with high accuracy even when there is a small amount of training data.
[0066] Although the embodiments of the present invention have been described above, the specific configuration is not limited to these embodiments, and it goes without saying that the present invention includes appropriate design changes and the like within the scope of the gist of the present invention. The various processes described in the embodiments are not only executed in chronological order according to the order described, but may also be executed in parallel or individually depending on the processing capacity of the device executing the processes or as necessary.
[0067] [Programs, recording media] When the various processing functions of each device described in the above embodiment are realized by a computer, the processing contents of the functions that each device should have are described by a program. Then, the various processing functions of each device are realized on the computer by loading this program into the storage unit 1020 of the computer shown in Fig. 6 and operating the arithmetic processing unit 1010, the input unit 1030, the output unit 1040, etc.
[0068] The program describing the processing contents can be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a non-transitory recording medium, such as a magnetic recording device or an optical disk.
[0069] The program may be distributed, for example, by selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of a server computer and transferring the program from the server computer to other computers via a network.
[0070] A computer that executes such a program, for example, first stores a program recorded on a portable recording medium or a program transferred from a server computer in the auxiliary recording unit 1050, which is a non-transient storage device of the computer. Then, when executing a process, the computer reads the program stored in the auxiliary recording unit 1050, which is a non-transient storage device, into the storage unit 1020, which is a temporary storage device, and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from the portable recording medium and execute a process according to the program, or may execute a process according to the received program each time a program is transferred from the server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may execute the above-mentioned process by a so-called ASP (Application Service Provider) type service that realizes a processing function only by issuing an execution instruction and obtaining a result. Note that the program in this embodiment includes information used for processing by an electronic computer and equivalent to a program (data that is not a direct instruction to a computer but has a nature that specifies the processing of a computer, etc.).
[0071] In addition, in this embodiment, the present device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.
Claims
1. A sound description generation method for generating a description of a target sound, comprising: A guidance explanation search unit acquires a plurality of explanations corresponding to an audio signal similar to the target sound, The description generating unit determines a current word of the description of the target sound by using a feature obtained by integrating the acquired description, a word string from the beginning of the description of the target sound to the immediately preceding word, and an acoustic feature of the target sound. A method for generating sound descriptions.
2. 2. The method of claim 1, further comprising: the guidance explanation search unit is configured to make it easier for the first acoustic signal and the second acoustic signal to be determined to be similar as the explanation describing the first acoustic signal and the explanation describing the second acoustic signal become more similar. A method for generating sound descriptions.
3. A sound description generation device that generates a description of a target sound, a guidance explanation search unit that acquires a plurality of explanations corresponding to a sound signal similar to the target sound; a description generation unit that determines a current word of the description of the target sound by using a feature that integrates the acquired description, a word string from the beginning of the description of the target sound to the immediately preceding word, and an acoustic feature of the target sound; A sound description generating device comprising:
4. 3. A program for causing a computer to execute each step of the sound description generation method according to claim 1 or 2.
