Learning device, learning method, and program

Abstract

Provided is, inter alia, a learning device that utilizes text data of original text before summarization without converting the text data into speech through speech synthesis. The learning device comprises: a speech encoder unit that uses a speech encoding model to acquire a first latent expression from a speech feature amount string corresponding to a training speech signal that corresponds to the original text before summarization; a text encoder unit that uses a text encoding model to acquire a second latent expression from training text data corresponding to the original text before summarization; a shared encoder unit that uses a shared encoding model to acquire a third latent expression from the first latent expression and acquire a third latent expression from the second latent expression; a summary decoder unit that uses a summarized text decoding model to approximate summarized text from the third latent expressions; and a model updating unit that updates parameters of a model on the basis of the estimated summarized text and correct answer data corresponding to the summarized text.

Description

Technical Field

[0001] The present invention relates to a technique for generating an abstract from speech.

Background Art

[0002] The speech automatic summarization technique for generating an abstract from speech has the usefulness that an output with higher readability is made than a transcription faithful to the speech, and has been studied for a long time. Basically, it is composed of two models: a speech recognition model that automatically transcribes speech and a text summarization model that generates an abstract from the transcription. This is generally called a "cascaded model".

[0003] On the other hand, an End-to-End speech summarization model that directly generates an abstract without going through transcription from speech is known (see Non-Patent Document 1). This method has become possible due to the recent increase in computer resources and is expected to be promising because of the simplicity of its structure and robustness against speech recognition errors.

[0004] For the learning of the End-to-End speech summarization model in Non-Patent Document 1, only paired data of speech corresponding to the original text before summarization and the abstract can be given. It is difficult to collect a large amount of such paired data, and compared with the cascaded model that can utilize a summarization dataset consisting only of text (text data of the original text before summarization and text data of the abstract) for learning, the learning of the End-to-End speech summarization model is disadvantageous in terms of the ease of obtaining learning data.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] As a solution to the above problems, among the summary datasets consisting only of text (text data of the original text before summarization and text data of the summary text of the original text), the text data of the original text before summarization is automatically converted into speech by speech synthesis, and pseudo-paired data of the speech (synthesized speech) corresponding to the original text before summarization and the summary text is generated as learning data. This method has been confirmed to have a certain improvement in accuracy in the field of speech recognition (see Reference 1).

[0007] (Reference 1) Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, Andrew Y. Ng, "Deep Speech: Scaling up end-to-end speech recognition", 2014. However, the method of giving synthesized speech to an End-to-End speech summarization model has the problem of high cost in terms of the time required for speech synthesis and the storage occupancy for storing the synthesized speech.

[0008] The present invention aims to provide a learning device that utilizes the original text data before summarization without converting it into speech by speech synthesis when training an end-to-end speech summarization model, a creation device that uses the end-to-end speech summarization model trained using the learning device, a learning method, and a program. [Means for solving the problem]

[0009] To solve the above problems, according to one aspect of the present invention, the learning device includes: an audio encoder unit that takes audio features as input and uses an audio encoding model that outputs latent representations corresponding to the audio features to acquire a first latent representation from a sequence of audio features corresponding to a training audio signal corresponding to the original text before summarization; a text encoder unit that takes text data as input and uses a text encoding model that outputs latent representations corresponding to the text data to acquire a second latent representation from training text data corresponding to the original text before summarization; a shared encoder unit that takes latent representations corresponding to text data and latent representations corresponding to audio features as input and uses a shared encoding model that outputs latent representations in a single latent representation space to acquire a third latent representation from the first latent representation and acquire a third latent representation from the second latent representation; a summarization decoder unit that takes a latent representation in which a latent representation corresponding to text data or a latent representation corresponding to audio features is abstractly embedded as input and uses a summarization decoding model that estimates a summary sentence to estimate a summary sentence from a third latent representation; and a model update unit that updates the parameters of the audio encoding model, text encoding model, shared encoding model, and summary decoding model based on the estimated summary sentence and the ground truth data corresponding to the summary sentence. To solve the above problems, according to another aspect of the present invention, the creation device includes: an audio encoder unit that takes audio features as input and uses an audio encoding model that outputs latent representations corresponding to the audio features to be summarized to acquire a first latent representation from a sequence of audio features corresponding to the audio signal to be summarized; a shared encoder unit that takes latent representations corresponding to text data and latent representations corresponding to audio features as input and uses a shared encoding model that outputs latent representations in a single latent representation space to acquire a third latent representation from the first latent representation; and a summary decoder unit that takes latent representations in which latent representations corresponding to audio features are abstractly embedded as input and uses a summary text decoding model that estimates a summary text to estimate a summary text from the third latent representation. The audio encoding model, the shared encoding model, and the summary text decoding model are learned based on the summary text output by the summary text decoding model and the ground truth data corresponding to the summary text. During learning, the shared encoder unit uses the shared encoding model to acquire a third latent representation from the first latent representation corresponding to the audio features and a third latent representation from the second latent representation corresponding to the text data. [Effects of the Invention]

[0010] According to the present invention, the cost increase that occurs when generating synthesized speech data by speech synthesis is significantly mitigated. Furthermore, since the text encoder is only required during training and not during the creation of the summary, the amount of memory consumed and processing time required during creation do not increase. [Brief explanation of the drawing]

[0011] [Figure 1] A diagram showing an example configuration of the summary generation system according to the first embodiment. [Figure 2] Functional block diagram of the learning device according to the first embodiment. [Figure 3] A diagram showing the processing flow of the learning device according to the first embodiment. [Figure 4] Functional block diagram of a learning device that uses synthesized speech. [Figure 5] Functional block diagram of the fabrication apparatus according to the first embodiment. [Figure 6] A diagram showing the processing flow of the manufacturing apparatus according to the first embodiment. [Figure 7] Functional block diagram of a learning device according to a modified example of the first embodiment. [Figure 8] Functional block diagram of a fabrication apparatus according to a modified example of the first embodiment. [Figure 9] This figure shows the accuracy (BERTscore) of the summary text produced by the creation apparatus according to the first embodiment. [Figure 10] A figure showing the accuracy (Rouge-L score) of the summary text produced by the creation apparatus according to the first embodiment. [Figure 11] A diagram showing an example of a computer configuration to which this method is applied. [Modes for carrying out the invention]

[0012] Embodiments of the present invention will be described below. In the drawings used in the following description, components with the same function or steps that perform the same processing will be denoted by the same reference numeral, and redundant explanations will be omitted. In the following description, unless otherwise specified, the processing performed on each element of a vector or matrix will be applied to all elements of that vector or matrix.

[0013] <Key points of the first embodiment> Conventional end-to-end speech summarization models only accept sequences of acoustic features as input. By adding a "text encoder," it becomes possible to accept inputs such as sequences of phonemes and sequences of subwords (data consisting of units that divide words into smaller parts, such as strings) obtained from the original text data before summarization. Inputs such as phonemes and subwords can be quickly converted from the original text data before summarization using the techniques shown in references 2 and 3, respectively, and since sequences of phonemes and subwords are also text data, they do not require much storage space for storage.

[0014] (Reference 2) Kyubyong, "g2p", [online], [Retrieved January 12, 2023], Internet<URL:https: / / github.com / Kyubyong / g2p / blob / master / g2p_en / g2p.py> (Reference 3) soaxelbrooke, "python-bpe", [online], [Retrieved January 12, 2023], Internet<URL:https: / / github.com / soaxelbrooke / python-bpe> In conventional technology, it is considered necessary to estimate speech signals from phoneme sequences using speech synthesis technology because end-to-end speech summarization models can only accept speech input. However, speech synthesis does not add any important information; essentially, only linguistic information needs to be provided to the model. Therefore, the key idea of ​​this embodiment is that if the model is modified to accept text data as input, it is sufficient without having to perform costly speech synthesis.

[0015] <First Embodiment> Figure 1 shows an example configuration of the summary writing system according to the first embodiment.

[0016] The summary generation system includes a learning device 100 and a generation device 200.

[0017] Figure 2 shows a functional block diagram of the learning device 100 according to the first embodiment, and Figure 3 shows its processing flow. Figure 4 shows a functional block diagram of the learning device 90 that uses synthesized speech.

[0018] The learning device 100 includes a feature conversion unit 110, a text conversion unit 120, a speech encoder unit 130, a text encoder unit 140, a shared encoder unit 150, a summary text decoder unit 160, and a model update unit 170.

[0019] The learning device 100 receives a learning audio signal x corresponding to the original text before summarization. LP pairs of paired data (x L (a, p), s(a, p)) consisting of (a, p) and the text data s(a, p) of the summary (correct data), and the learning text data t corresponding to the original text before summarization L Q pairs of paired data (t L (b, q), s(b, q)) consisting of (b, q) and the text data s(b, q) of the summary (correct data) are used as input, an End-to-End speech summarization model is learned, and the learned End-to-End speech summarization model M or its parameters par are output. Let a be a label indicating information related to the learning speech signal, b be a label indicating information related to the learning text data, p = 1, 2,..., P, and q = 1, 2,..., Q. The original text of the learning speech signal x L (a, p) and the learning text data t L (b, q) are assumed to be different considering the ease of collection, but they do not necessarily have to be different and may be the same original text. As described above, it is difficult to collect a large amount of paired data of the speech signal corresponding to the original text before summarization and the text data of the summary. Therefore, in this embodiment, the learning speech signal x corresponding to the original text before summarization L P pairs of paired data (x L (p), t(p)) consisting of (a, p) and the text data s(a, p) of the summary, and the learning text data t corresponding to the original text before summarization L Q pairs of paired data (t L (b, q), s(b, q)) consisting of (b, q) and the text data s(b, q) of the summary are used as learning data. There is no need to necessarily limit the magnitude relationship between P and Q, but in view of the current situation that it is difficult to collect a large amount of paired data of the speech signal corresponding to the original text before summarization and the text data of the summary, it is desirable to set P < Q. For example, Q can be set to about 8 times P.

[0020] As shown in Figure 4, the learning device 90 using synthesized speech includes a feature conversion unit 110, a text conversion unit 120, a speech synthesis unit 95, a speech encoder unit 130, a shared encoder unit 150, a summary text decoder unit 160, and a model update unit 170. The learning device 90 includes the speech synthesis unit 95 but does not include the text encoder unit 140. The learning device 90 receives a learning speech signal x corresponding to the original text before summarization. L Paired data of set P consisting of (a,p) and summary text data s(a,p) (ground truth data) L (a,p),s(a,p)) and the learning text data t corresponding to the original text before summarization. L Q pairs of data (t) consisting of (b,q) and summary text data s(b,q) (ground truth data) L The learning device 90 takes (b,q) and s(b,q)) as inputs, trains an End-to-End speech summarization model, and outputs the trained End-to-End speech summarization model M or its parameters par. The speech synthesis unit 95 processes the training text data t corresponding to the original text before summarization. L It is necessary to perform speech synthesis on (b,q), and furthermore, it is necessary to store the synthesized speech obtained as a result of speech synthesis in a recording unit not shown in the diagram.

[0021] Figure 5 shows a functional block diagram of the manufacturing apparatus 200 according to the first embodiment, and Figure 6 shows its processing flow.

[0022] The creation device 200 includes a feature conversion unit 210, an audio encoder unit 230, a shared encoder unit 250, and a summary text decoder unit 260.

[0023] Prior to the process of creating a summary, the creation device 200 receives a trained End-to-End speech summarization model M or its parameters par.

[0024] The summarization device 200 takes the audio signal x(t) to be summarized as input, summarizes it using a pre-trained End-to-End speech summarization model, and outputs a summarized text s(t). Note that t is a label indicating that the information is related to the information to be summarized. For example, the audio signal to be summarized may be an audio signal captured by a microphone or an audio signal that was previously captured and recorded on some recording medium.

[0025] The learning device 100 and the creation device 200 are special devices configured by loading a special program into a known or dedicated computer having, for example, a central processing unit (CPU) and main memory (RAM). The learning device 100 and the creation device 200 execute their respective processes under the control of, for example, the central processing unit. Data input to the learning device 100 and the creation device 200 and data obtained in each process are stored, for example, in main memory, and the data stored in main memory is read to the central processing unit as needed and used for other processes. Each processing unit of the learning device 100 and the creation device 200 may be composed of hardware such as integrated circuits, at least in part. Each storage unit of the learning device 100 and the creation device 200 can be composed of, for example, main memory such as RAM (Random Access Memory), or middleware such as a relational database or key-value store. However, each storage unit does not necessarily have to be located inside the learning device 100 and the creation device 200. It may be configured as an auxiliary storage device consisting of a hard disk, optical disc, or semiconductor memory element such as flash memory, and may be located outside the learning device 100 and the creation device 200.

[0026] The process in the summary generation system consists of a training phase for the End-to-End speech summarization model and a summary generation phase. First, let's explain the training phase.

[0027] <Learning Stage> The learning device 100 according to the first embodiment will be described using Figures 2 and 3.

[0028] <Feature conversion unit 110> The feature transformation unit 110 processes P training audio signals x corresponding to the original text before summarization. L (a,p) is the input, and the learning audio signal x L (a,p) is converted into audio features (S110), and a sequence of P audio features X L Outputs (a,p). For example, the learning audio signal x recorded by a microphone, etc. L (a,p) is converted into a feature that is easy to process using the model used in the audio encoder unit 130 described later. This embodiment does not depend on the feature converted here, but for example, a 43-dimensional feature can be considered, which is a 40-dimensional logarithmic Mel filter bank with the pitch and its first and second difference added.

[0029] <Text conversion unit 120> The text conversion unit 120 processes Q training text data t corresponding to the original text before summarization. L (b,q) is taken as input. In this embodiment, the training text data t L (b,q) consists of a sequence of words. The text conversion unit 120 receives Q pieces of training text data t L The sequence of words (b,q) is converted into Q phoneme sequences, strings, or subword sequences corresponding to each word and obtained (S120), and then output.

[0030] The conversion from word sequences to phoneme sequences can be achieved using the technique described in Reference 2. The conversion from word sequences to strings can be achieved by replacing empty strings with spaces, and the conversion to other subwords can be achieved using the technique described in Reference 3.

[0031] <Audio encoder unit 130> The audio encoder unit 130 processes P audio feature sequences X L (a,p) is the input, and a speech encoding model is used to obtain P speech feature sequences X L P latent representations Y from (a,p) L(a,p) is obtained (S130) and output. This latent representation is the training audio signal x L This extracts essential information from (a,p) and is used as the learning audio signal x L This is the feature information that best represents (a,p). The speech encoding model is a model that takes speech features as input and outputs a latent representation corresponding to the speech features.

[0032] The audio encoder unit 130 includes, for example, a neural network as an audio encoding model, and the input layer of the neural network contains an audio feature sequence X L Given (a,p), the latent representation Y is the intermediate feature, which is the output of the intermediate layer of the neural network. L The data is obtained as (a,p). The neural network parameters are automatically optimized (=learned) by backpropagation of the loss calculated by the model update unit 170, described later, so that a better latent representation for outputting the summary is produced. For example, the parameters are updated so that the loss calculated by the model update unit 170 is reduced. For example, the configuration of the speech encoder unit 130, text encoder unit 140, shared encoder unit 150, summary decoder unit 160, and model update unit 170, and the optimization of parameters by backpropagation, can be realized using the technology published in Reference 4. (Reference 4) Fhrozen, "espnet", [online], [Retrieved January 12, 2023], Internet<URL:https: / / https: / / github.com / espnet / espnet> <Text encoder section 140> The text encoder unit 140 takes the Q phoneme or subword sequence, which is the output of the text conversion unit 120, as input and uses a text encoding model to convert the Q phoneme or subword sequence into Q latent representations Y L (b,q) is obtained (S140) and output. This latent representation is based on the training text data t L This extracts essential information from (b,q), and is used as training text data t L This is the feature information that best represents (b,q). Note that the training text data tL The word sequence (b,q), as well as the phoneme sequence or subword sequence, are all text data, and a text encoding model can be described as a model that takes text data as input and outputs a latent representation corresponding to that text data.

[0033] The text encoder unit 140 includes, for example, a neural network as a text encoding model, provides a sequence of phonemes or subwords corresponding to the text data to the input layer of the neural network, and uses the intermediate features, which are the output of the intermediate layer of the neural network, as the latent representation Y. L The values ​​are obtained as (b,q). The neural network parameters are automatically optimized (=learned) by backpropagation of the loss calculated in the model update unit 170 described later, so that a better latent representation for outputting the summary sentence is produced. For example, the parameters are updated so that the loss calculated in the model update unit 170 becomes smaller.

[0034] Note that the learning text data t L Latent representation Y corresponding to (b,q) L The data structure of (b,q) and the training audio signal x L Latent representation Y corresponding to (a,p) L The models of the audio encoder unit 130 and the text encoder unit 140 are configured such that the data structure of (a,p) is the same. For example, the latent representation Y of the audio encoder unit 130 L The intermediate layer outputs (a,p), and the latent representation Y of the text encoder unit 140 outputs data with the same data structure. L A hidden layer is constructed that outputs (b,q).

[0035] <Shared encoder unit 150> The shared encoder unit 150 has P latent representations Y which are the output of the audio encoder unit 130. L (a,p), and Q latent representations Y which are the output of the text encoder unit 140. L Given (b,q) as input, a shared encoding model is used to generate P latent representations Y L P latent representations Y from (a,p) LObtain (c,r) and P latent representations Y L P latent representations Y from (b,q) L (c,r) is obtained, and a total of P+Q latent representations Y L (c,r) is obtained (S150) and output. Here, c is a label indicating that it is a latent representation obtained by the shared encoder unit 150, and r = 1, 2, ..., P+Q. The shared encoding model is a model that takes latent representations corresponding to text data and latent representations corresponding to speech features as input and outputs a latent representation of a single latent representation space. Note that the latent representation Y L (c,r) is the latent representation Y corresponding to the text data. L (b,q) or latent representation Y corresponding to the speech features L (a,p) can be described as a latent expression in which it is abstractly embedded.

[0036] The shared encoder unit 150 includes, for example, a neural network as a shared encoding model, and the input layer of the neural network contains a latent representation Y L (a,p), or latent expression Y L Given (b,q), the latent representation Y is the intermediate feature, which is the output of the intermediate layer of the neural network. L The values ​​are obtained as (c,r). The neural network parameters are automatically optimized (=learned) by backpropagation of the loss calculated in the model update unit 170 described later, so that a better latent representation for outputting the summary sentence is produced. For example, the parameters are updated so that the loss calculated in the model update unit 170 becomes smaller.

[0037] <Summary Decoder Section 160> The summary decoder unit 160 processes the latent representation Y which is the output of the shared encoder unit 150. L (c,r) is the input, and the summary text decoding model is used to generate the latent representation Y L (c,r) is used to estimate the summary (S160), and the estimated summary s L Output (r). For example, a summary is represented as a sequence of tokens. A sequence of tokens is a collection of tokens arranged in order, grouped into units that are easy to process. The summary decoding model uses latent representation Y LThis model takes (c,r) as input and estimates a summary.

[0038] The summary text decoder unit 160 includes, for example, a neural network as a summary text decoding model, and the input layer of the neural network contains a latent representation Y L Given (c,r), the output of the neural network's output layer is obtained as a summary. The neural network's parameters are automatically optimized (=learned) by backpropagation of the loss calculated in the model update unit 170 (described later), enabling it to estimate the summary better. For example, the parameters are updated to reduce the loss calculated in the model update unit 170.

[0039] <Model Update Section 170> The model update unit 170 uses the summary text s estimated by the summary text decoder unit. L (r) and summary text s L The input is the correct answer data (text data s(a,p) or s(b,q)) corresponding to (r), and the summary text s L Based on (r) and the text data s(a,p) or s(b,q) (ground truth data), the parameters of the speech encoding model, text encoding model, shared encoding model, and summary text decoding model are updated (S170). For example, summary text s L The parameters of the speech encoding model, text encoding model, shared encoding model, and summary text decoding model are updated so that (r) matches the text data s(a,p) or s(b,q) (ground truth data). For example, the model update unit 170 uses the loss function to update the parameters of the summary text s L The loss is calculated from (r) and the text data s(a,p) or s(b,q) (ground truth data), and the parameters are updated to minimize the loss.

[0040] The model update unit 170 applies a predetermined amount of training data, for example, P pairs of data (x L (a,p),s(a,p)) and paired data of group Q (t LThe parameter update process is performed for all of (b,q) and s(b,q)), and then it is determined whether the parameter update has converged (S175). If it is determined that it has converged (YES in S175), the model M or parameter par at that point is output as the output of the learning device 100 and the process ends. If it is determined that it has not converged (NO in S175), the model update unit 170 sends control signals to each unit to repeat S105 to S170 and controls the processing of each unit. In order to determine whether it has converged or not, conditions such as whether learning has been repeated a certain number of times (for example, several times), whether the difference in parameters before and after the update is below a predetermined threshold, or whether the loss is below a predetermined threshold can be used.

[0041] Next, I will explain the creation process.

[0042] <Creation Stage> The manufacturing apparatus 200 according to the first embodiment will be described with reference to Figures 5 and 6.

[0043] <Feature conversion unit 210> The feature transformation unit 210 takes the audio signal x(t) to be summarized as input, transforms the audio signal x(t) into audio features (S210), and outputs an audio feature sequence X(t). For example, it transforms the audio signal x(t) recorded by a microphone or the like into features that are easy to process with the model used in the audio encoder unit 230 described later. For example, it transforms it into features similar to those of the feature transformation unit 110.

[0044] <Audio encoder unit 230> The audio encoder unit 230 takes an audio feature sequence X(t) as input and uses a trained audio encoding model to obtain a latent representation Y(t) from the audio feature sequence X(t) (S230), and outputs it. This latent representation extracts the essential information of the audio signal x(t) to be summarized and is the feature information that best represents the audio signal x(t) to be summarized.

[0045] The audio encoder unit 230 has a similar structure to the audio encoder unit 130, and for example, includes a neural network as the audio encoding model. The input layer of the neural network is given an audio feature sequence X(t), and the intermediate features, which are the output of the intermediate layer of the neural network, are obtained as the latent representation Y(t). The parameters of the neural network used in the audio encoder unit 230 are included in the trained End-to-End audio summarization model M or its parameters par.

[0046] <Shared encoder unit 250> The shared encoder unit 250 takes the latent representation Y(t), which is the output of the audio encoder unit 230, as input, and uses a trained shared encoding model to obtain the latent representation Y(c,t) from the latent representation Y(t) (S250), and outputs it. Here, c is a label indicating that it is a latent representation obtained by the shared encoder unit 250.

[0047] The shared encoder unit 250 has a similar structure to the shared encoder unit 150, and for example, includes a neural network as a shared encoding model, provides a latent representation Y(t) to the input layer of the neural network, and obtains intermediate features, which are the outputs of the intermediate layers of the neural network, as latent representation Y(c,t). The parameters of the neural network used in the shared encoder unit 250 are included in the trained End-to-End speech summarization model M or its parameters par.

[0048] <Summary Decoder Section 260> The summary text decoder unit 260 takes the latent representation Y(c,t), which is the output of the shared encoder unit 250, as input, and uses a trained summary text decoding model to estimate the summary text from the latent representation Y(c,t) (S260), and outputs the estimated summary text s(t).

[0049] The summary text decoder unit 260 has a similar structure to the summary text decoder unit 160, for example, it includes a neural network as the summary text decoding model, provides a latent representation Y(c,t) to the input layer of the neural network, and obtains the output of the output layer of the neural network as the summary text. The parameters of the neural network used in the summary text decoder unit 260 are included in the trained End-to-End speech summarization model M or its parameters par.

[0050] <Effects> This configuration allows for the low-cost application of text-only data to the training of an end-to-end speech summarization model. Furthermore, it reduces the time and processing costs associated with generating synthesized speech data through speech synthesis. Additionally, since there is no need to store the synthesized speech, storage space is significantly reduced. This reduction in application costs enables the use of external text on a larger scale, which is expected to result in improved quality of summaries output by the model.

[0051] <Variation> In the first embodiment, the learning device 100 was described as including separate models (speech encoding model, text encoding model, shared encoding model, and summary decoding model) in the speech encoder unit 130, text encoder unit 140, shared encoder unit 150, and summary decoding unit 160. However, the speech encoder unit 130, text encoder unit 140, shared encoder unit 150, and summary decoding unit 160 may be configured to include a single model (e.g., a neural network) (see Figure 7). In this single model, the functions of the speech encoding model, text encoding model, shared encoding model, and summary decoding model are realized. For example, the input layer of the neural network may contain P speech feature sequences X LEither (a,p) or a sequence of Q phonemes or subwords is provided, and the output of the neural network's output layer is obtained as a summary sentence. The neural network's parameters are automatically optimized (=learned) by backpropagation of the loss calculated in the model update unit 170, so that a better latent representation for the summary sentence output is produced. For example, summary sentence s L (r) is updated so that it matches the text data s(a,p) or s(b,q) (ground truth data). For example, the model update unit 170 uses a loss function to update the summary sentence s L The loss is calculated from (r) and text data s(a,p) or s(b,q) (ground truth data), and the parameters are updated to minimize the loss. In this case, the creation device 200 includes one model (e.g., a neural network) in the audio encoder unit 230, the shared encoder unit 250, and the summary text decoder unit 260 (see Figure 8). For example, the audio feature sequence X(t) is given to the input layer of the neural network, and the output of the output layer of the neural network is obtained as the summary text s(t). Note that the audio encoder unit 130, the text encoder unit 140, the shared encoder unit 150, and the summary text decoder unit 160 may be configured to include one model (neural network) in any two or more consecutive configurations. The same applies to the audio encoder unit 230, the shared encoder unit 250, and the summary text decoder unit 260.

[0052] <Experimental Results> Figure 9 shows the accuracy of the summary text produced by the creation device according to the first embodiment, and shows the change in accuracy (BERT score) when the difficulty level of the evaluation set is changed. Figure 10 shows the accuracy of the summary text produced by the creation device according to the first embodiment, and shows the change in accuracy (Rouge-L score) when the difficulty level of the evaluation set is changed. In the figures, "End-to-End type + phoneme input" corresponds to the first embodiment, "End-to-End type" corresponds to Non-Patent Literature 1, "End-to-End type + synthesized speech" corresponds to Reference 1, and "Cascade type" corresponds to the cascade model.

[0053] The data used in the experiment was the How2 dataset, which contains approximately 1800 hours of audio-enhanced videos averaging about 70 seconds in length, along with their summaries (see Reference 5).

[0054] (Reference 5) Ramon Sanabria et al., "How2: A Large-scale Dataset for Multimodal Language Understanding, [online], [Retrieved January 12, 2023], Internet<URL:https: / / arxiv.org / pdf / 1811.00347.pdf> The Gigaword corpus (average input of 31.4 words, average summary of 8 words, totaling 3.8 million pairs) was used to augment the data.

[0055] When using speech synthesis, it takes 9 days to synthesize speech using one GPU (NVIDIA® RTX A6000) and requires 3.5TB (3500GB) of storage capacity. On the other hand, in this embodiment (when using phoneme input), it takes 3.5 hours to convert words to phonemes using one CPU core (AMD® EPYC 7502P) and requires 15GB of storage capacity. Furthermore, Figures 9 and 10 show that the estimation accuracy of the creation device according to this embodiment is higher than that of other estimation accuracies.

[0056] <Other variations> The present invention is not limited to the embodiments and modifications described above. For example, the various processes described above may not only be performed sequentially as described, but may also be performed in parallel or individually as needed, depending on the processing capacity of the device performing the processes. Other modifications can be made as appropriate without departing from the spirit of the present invention.

[0057] <Program and recording medium> The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 12, and then causing the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc. to operate.

[0058] The program describing this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory.

[0059] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0060] A computer executing such a program may, for example, first store the program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. Then, when processing is to be executed, the computer reads the program stored on its own storage medium and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the above processing may be executed by a so-called ASP (Application Service Provider) type service, where the server computer does not transfer programs to this computer, but the processing function is realized only by execution instructions and result acquisition. In this form, the program includes information used for processing by an electronic computer that is equivalent to a program (data that is not a direct instruction to the computer but has the property of defining the processing of the computer, etc.).

[0061] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware.

[0062] The following additional information is disclosed regarding the embodiments described above.

[0063] (Additional note 1) Memory and At least one processor connected to the memory, Includes, The aforementioned processor, Using a speech encoding model that takes speech features as input and outputs a latent representation corresponding to the speech features, the first latent representation is obtained from the sequence of speech features corresponding to the training speech signal corresponding to the original text before summarization. Using a text encoding model that takes text data as input and outputs a latent representation corresponding to the text data, a second latent representation is obtained from training text data corresponding to the original text before summarization. Using a shared encoding model that takes latent representations corresponding to text data and latent representations corresponding to speech features as input and outputs latent representations in a single latent representation space, a third latent representation is obtained from the first latent representation, and a third latent representation is obtained from the second latent representation, Using a summary decoding model that takes a latent representation in which a latent representation corresponding to text data or a latent representation corresponding to audio features is abstractly embedded as input, the summary is estimated from the third latent representation. Based on the estimated summary and the corresponding ground truth data, the parameters of the speech encoding model, the text encoding model, the shared encoding model, and the summary decoding model are updated. Learning device.

[0064] (Additional note 2) Memory and At least one processor connected to the memory, Includes, The aforementioned processor, Using a speech encoding model that takes speech features as input and outputs a latent representation corresponding to the speech features, we obtain the first latent representation from the sequence of speech features corresponding to the speech signal to be summarized. Using a shared encoding model that takes latent representations corresponding to text data and latent representations corresponding to audio features as input and outputs latent representations in a single latent representation space, the third latent representation is obtained from the first latent representation. Using a summary decoding model that takes a latent representation in which latent representations corresponding to speech features are abstractly embedded as input, the summary is estimated from the third latent representation. The aforementioned speech encoding model, shared encoding model, and summary decoding model are trained based on the summary output by the summary decoding model and the ground truth data corresponding to that summary. During training, the shared encoder unit uses the shared encoding model to obtain a third latent representation from the first to the third latent representation corresponding to the speech features, and obtains a third latent representation from the second to the third latent representation corresponding to the text data. Manufacturing device.

[0065] (Additional note 3) A non-temporary storage medium that stores a program executable by a computer to perform a learning process, The aforementioned learning process is, Using a speech encoding model that takes speech features as input and outputs a latent representation corresponding to the speech features, the first latent representation is obtained from the sequence of speech features corresponding to the training speech signal corresponding to the original text before summarization. Using a text encoding model that takes text data as input and outputs a latent representation corresponding to the text data, a second latent representation is obtained from training text data corresponding to the original text before summarization. Using a shared encoding model that takes latent representations corresponding to text data and latent representations corresponding to speech features as input and outputs latent representations in a single latent representation space, a third latent representation is obtained from the first latent representation, and a third latent representation is obtained from the second latent representation, Using a summary decoding model that takes a latent representation in which a latent representation corresponding to text data or a latent representation corresponding to audio features is abstractly embedded as input, the summary is estimated from the third latent representation. Based on the estimated summary and the corresponding ground truth data, the parameters of the speech encoding model, the text encoding model, the shared encoding model, and the summary decoding model are updated. Non-transitory storage medium.

[0066] (Additional note 4) A non-temporary storage medium that stores a program executable by a computer to perform a creation process, The aforementioned creation process is, Using a speech encoding model that takes speech features as input and outputs a latent representation corresponding to the speech features, we obtain the first latent representation from the sequence of speech features corresponding to the speech signal to be summarized. Using a shared encoding model that takes latent representations corresponding to text data and latent representations corresponding to audio features as input and outputs latent representations in a single latent representation space, the third latent representation is obtained from the first latent representation. Using a summary decoding model that takes a latent representation in which latent representations corresponding to speech features are abstractly embedded as input, the summary is estimated from the third latent representation. The aforementioned speech encoding model, shared encoding model, and summary decoding model are trained based on the summary output by the summary decoding model and the ground truth data corresponding to that summary. During training, the shared encoder unit uses the shared encoding model to obtain a third latent representation from the first to the third latent representation corresponding to the speech features, and obtains a third latent representation from the second to the third latent representation corresponding to the text data. Non-transitory storage medium.

Claims

1. A speech encoder unit that takes speech features as input and outputs a latent representation corresponding to the speech features, and uses a speech encoding model to obtain a first latent representation from a sequence of speech features corresponding to a training speech signal corresponding to the original text before summarization, A text encoder unit that takes text data as input and uses a text encoding model that outputs a latent representation corresponding to the text data to obtain a second latent representation from training text data corresponding to the original text before summarization, A shared encoder unit that takes latent representations corresponding to text data and latent representations corresponding to audio features as input and outputs latent representations in a single latent representation space, and uses this shared encoding model to obtain a third latent representation from the first latent representation and obtain a third latent representation from the second latent representation, A summary decoder unit that estimates a summary sentence from the third latent expression using a summary sentence decoding model that takes a latent expression in which a latent expression corresponding to text data or a latent expression corresponding to audio features is embedded in a single latent expression space as input, and estimates a summary sentence, The system includes a model update unit that updates the parameters of the speech encoding model, the text encoding model, the shared encoding model, and the summary decoding model based on the estimated summary and the corresponding ground truth data. Learning device.

2. A speech encoder step that uses a speech encoding model that takes speech features as input and outputs a latent representation corresponding to the speech features to obtain a first latent representation from a sequence of speech features corresponding to a training speech signal corresponding to the original text before summarization, A text encoder step involves using a text encoding model that takes text data as input and outputs a latent representation corresponding to the text data to obtain a second latent representation from training text data corresponding to the original text before summarization, and A shared encoder step involves taking latent representations corresponding to text data and latent representations corresponding to speech features as inputs and outputting latent representations in a single latent representation space, to obtain a third latent representation from the first latent representation and to obtain a third latent representation from the second latent representation, A summary decoder step in which a summary is estimated from the third latent expression using a summary decoding model that takes a latent expression in which a latent expression corresponding to text data or a latent expression corresponding to audio features is abstractly embedded as input, and estimates a summary. The model update step includes updating the parameters of the speech encoding model, the text encoding model, the shared encoding model, and the summary decoding model based on the estimated summary and the ground truth data corresponding to the summary, Learning methods.

3. A program for causing a computer to function as a learning device according to claim 1.

Images