Synthetic training data generation device and synthetic training data generation method

Abstract

To generate a variety of learning data as learning data for an ASR model.SOLUTION: A synthetic learning data generator 1 for generating synthetic learning data for learning an ASR model used in automatic speech recognition, includes: a general-purpose text database 11 that stores various text data groups that have been collected in advance; a target domain text database 12 that stores text data groups belonging to a target domain of the automatic speech recognition; a text data selection section 20 that compares the general-purpose text database 11 and the target domain text database 12, and selects learning text data from the general-purpose text database 11 based on the comparison results; and a synthetic learning data generation section 30 that generates synthetic learning data based on the learning text data selected by the text data selection section 20.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an apparatus and method for generating synthetic training data for use in machine learning. [Background technology]

[0002] Automatic speech recognition (ASR) using machine learning, such as deep learning, has been widely known. To effectively train the model used in this automatic speech recognition (ASR model), it is necessary to prepare a large amount of speech data as training data.

[0003] However, preparing a large amount of speech data suitable for machine learning from speech data actually spoken by humans is time-consuming and not easy. Therefore, it has been proposed to use speech data generated using a machine learning model for speech synthesis called a TTS (Text To Speech) model as training data for an ASR model instead of speech data actually spoken by humans (for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Fazel, A., Yang, W., Liu, Y., Barra-Chicote, R., Meng, Y., Maas, R., Droppo, J. (2021) SynthASR: Unlocking Synthetic Data for Speech Recognition. Proc. Interspeech 2021, 896-900, doi: 10.21437 / Interspeech.2021-1882 Summary of the Invention [Problem to be solved by the invention]

[0005] In conventional technologies such as Non-Patent Document 1, the speech data generated by a TTS model depends on the text content input to the TTS model. Therefore, in order to generate speech data suitable for training data for an ASR model, it is necessary to set a large amount of diverse and appropriate text content as input to the TTS model. If this setting is insufficient, the number and diversity of vocabulary in the training data generated using the TTS model will be limited, preventing sufficient training of the ASR model. Thus, there is still room for improvement in conventional training data generation methods.

[0006] The present invention has been made in light of the above background, and aims to provide a synthetic training data generation device that is capable of generating diverse training data as training data for an ASR model. [Means for solving the problem]

[0007] The synthetic training data generation device according to the present invention is a device for generating synthetic training data for training a model used in automatic speech recognition, and includes: a general-purpose text database storing various text data groups collected in advance; and a target domain text database storing text data groups belonging to a target domain of the automatic speech recognition; before a text data selection unit that selects learning text data from the general-purpose text database; and The speech data synthesized from the sentence represented by The synthetic training data as a synthetic training data generation unit that generates The text data selection unit has a candidate text data extraction unit that extracts candidate text data that are candidates for the training text data from the general-purpose text database on a sentence-by-sentence basis, and a vocabulary size maximization unit that calculates a frequency of a word that has not yet appeared in the target domain text database for each sentence represented by the candidate text data extracted by the candidate text data extraction unit, and selects the training text data from the candidate text data based on the calculated frequency of the word that has not yet appeared. The candidate text data extraction unit calculates a similarity between each sentence represented by text data included in the general-purpose text database and each sentence represented by text data included in the target domain text database, and extracts text data corresponding to sentences whose similarity is equal to or greater than a predetermined value from the general-purpose text database as the candidate text data. . The synthetic training data generation method according to the present invention is a method for generating synthetic training data for training a model used in automatic speech recognition, which is a method for generating synthetic training data by using a computer to generate synthetic training data from a general-purpose text database in which various text data sets collected in advance are stored. Each sentence represented by the text data contained in and a target domain text database storing a group of text data belonging to a target domain of the automatic speech recognition. and each sentence represented by text data included in the target domain text database, and extracts text data corresponding to sentences whose similarity is equal to or greater than a predetermined value from the general-purpose text database as candidate text data that are candidates for learning text data; and for each sentence represented by the candidate text data, calculates the frequency of words that have not yet appeared in the target domain text database; andBased on the candidate text data from The aforementioned Selecting learning text data, The speech data synthesized from the sentence represented by The synthetic training data as Generate. [Effects of the Invention]

[0008] According to the present invention, a variety of training data can be generated as training data for an ASR model. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a hardware configuration of a synthetic training data generation device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing the functional configuration of a synthetic training data generation device. [Figure 3] 10 is a flowchart showing a process flow of the synthetic training data generation device for generating synthetic training data. [Figure 4] 10 is a flowchart showing the flow of a candidate text data extraction process. [Figure 5] 10 is a flowchart showing the flow of a vocabulary size maximization process. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, identical or similar components will be designated by the same reference numerals, and duplicate descriptions will be omitted. In the following description, when it is necessary to distinguish between components of the same type, an identifier (numbers, letters, etc.) will be written in parentheses after the reference numeral that collectively refers to the components.

[0011] Fig. 1 is a diagram showing the hardware configuration of a synthetic training data generation device according to one embodiment of the present invention. The synthetic training data generation device 1 shown in Fig. 1 is an information processing device (computer) that generates synthetic training data used in machine learning of an ASR model for automatic speech recognition, and includes a processor 2, a main memory device 3, an auxiliary memory device 4, an input device 5, an output device 6, and a communication device 7.

[0012] The processor 2 is a device that performs arithmetic processing and is configured using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc. The main memory device 3 is a device that stores various programs and data and is configured using, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The auxiliary memory device 4 is a device that stores programs and data and is configured using, for example, an HDD (Hard Disk Drive), a flash memory, an SSD (Solid State Drive), etc. The programs and data stored in the auxiliary memory device 4 are loaded into the main memory device 3 by the processor 2 as needed and are used for processing and control performed by the processor 2.

[0013] The input device 5 is a user interface that receives information from a user, and is, for example, a keyboard, a mouse, a card reader, a touch panel, etc. The output device 6 is a user interface that outputs various types of information (display output, audio output, print output, etc.), and is, for example, a display device (LCD (Liquid Crystal Display), a graphics card, etc.) that visualizes various types of information, an audio output device (speaker), a printer, etc.

[0014] The communication device 7 is a communication interface that communicates with other devices via communication means. The configuration of the communication means is not necessarily limited, and examples thereof include communication means conforming to various communication standards such as USB (Universal Serial Bus) and RS-232C, LAN (Local Area Network), WAN (Wide Area Network), the Internet, and dedicated lines. The communication device 7 is, for example, a NIC (Network Interface Card), a wireless communication module, a USB module, and a serial communication module. The communication device 7 can also function as an input device that receives information from other devices that are communicatively connected. The communication device 7 can also function as an output device that transmits information to other devices that are communicatively connected.

[0015] 2 is a block diagram showing the functional configuration of synthetic training data generation device 1. As shown in the figure, synthetic training data generation device 1 includes the functions of a memory unit 10, a text data selection unit 20, and a synthetic training data generation unit 30. Memory unit 10 is realized, for example, by a main memory unit 3 or an auxiliary memory unit 4. Furthermore, text data selection unit 20 and synthetic training data generation unit 30 are realized, for example, by processor 2 reading a program stored in main memory unit 3 and executing processing in accordance with this program.

[0016] In addition, in the synthetic training data generation device 1, in addition to what is shown in Figure 1, further hardware such as an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or an AI (Artificial Intelligence) chip may be installed, and some or all of the functions shown in Figure 2 may be realized by using this hardware alone or in cooperation with the processor 2.

[0017] The storage unit 10 stores a general-purpose text database 11, a target domain text database 12, a candidate text database 13, a training text database 14, and a synthetic training database 15. The storage unit 10 can store these databases, for example, as tables provided by a DBMS (Database Management System) or files provided by a file system.

[0018] Various types of text data that have been collected in advance are stored in the general-purpose text database 11. For example, data such as texts transcribed from news, social media, language corpora, books, various videos and audio, etc. are stored in the general-purpose text database 11.

[0019] A group of text data belonging to the target domain of the ASR model is stored in the target domain text database 12. For example, the target domain text database 12 stores various text data corresponding to sentence expressions and phrases frequently used in fields where automatic speech recognition using the ASR model is expected to be used.

[0020] The text data selection unit 20 compares the general-purpose text database 11 with the target domain text database 12, and based on the comparison result, selects training text data from the general-purpose text database 11 for generating synthetic training data for the ASR model. The text data selection unit 20 includes a candidate text data extraction unit 21 and a vocabulary size maximization unit 22.

[0021] The candidate text data extraction unit 21 extracts candidate text data, which are candidates for learning text data, from the general-purpose text database 11 on a sentence-by-sentence basis based on the comparison result between the general-purpose text database 11 and the target domain text database 12. The extracted candidate text data is then stored in the candidate text database 13. Details of the method for extracting candidate text data by the candidate text data extraction unit 21 will be described later.

[0022] The vocabulary size maximization unit 22 calculates the frequency of occurrence in the target domain text database 12 for each word in a sentence represented by the candidate text data extracted by the candidate text data extraction unit 21 and stored in the candidate text database 13. Then, based on the calculated frequency of occurrence of each word in the sentence, it selects training text data from the candidate text data stored in the candidate text database 13 and stores the selected training text data in the training text database 14. The method of selecting training text data by the vocabulary size maximization unit 22 will be described in detail later.

[0023] The synthetic training data generation unit 30 generates synthetic training data to be used for training the ASR model based on the training text data selected by the text data selection unit 20 and stored in the training text database 14. The generated synthetic training data is then stored in the synthetic training database 15.

[0024] FIG. 3 is a flowchart showing the flow of processing performed by the synthetic training data generation device 1 to generate synthetic training data.

[0025] In step S10, various text data such as those described above is acquired as general-purpose text data and stored in the general-purpose text database 11. Here, the general-purpose text data can be acquired from various publicly available information sources.

[0026] In step S20, the candidate text data extraction unit 21 of the text data selection unit 20 performs a candidate text data extraction process for extracting candidate text data from the general-purpose text database 11. As a result of this candidate text data extraction process being executed in the candidate text data extraction unit 21, candidate text data that are candidates for learning text data are extracted from the general-purpose text database 11 and stored in the candidate text database 13. Details of the candidate text data extraction process in step S20 will be described later with reference to the flowchart in FIG.

[0027] In step S30, vocabulary size maximization unit 22 of text data selection unit 20 performs a vocabulary size maximization process to select training text data with maximized vocabulary size from the candidate text data. By executing this vocabulary size maximization process in vocabulary size maximization unit 22, training text data is selected from candidate text database 13 and stored in training text database 14. Details of the vocabulary size maximization process in step S30 will be described later with reference to the flowchart in FIG.

[0028] In step S40, synthetic training data is generated from the training text data by synthetic training data generation unit 30. Here, for example, by inputting the training text data stored in training text database 14 through the vocabulary size maximization process in step S30 into a trained TTS model, speech data obtained by speech synthesis of sentences represented by the training text data is generated as synthetic training data to be used for training the ASR model.

[0029] After the process of step S40 is completed, the generated synthetic training data is stored in the synthetic training database 15, and the process shown in the flowchart of FIG. 3 ends.

[0030] FIG. 4 is a flowchart showing the flow of the candidate text data extraction process executed by the candidate text data extraction unit 21 in step S20 of FIG.

[0031] In step S21, each sentence in the general-purpose text database 11 is vectorized. Here, for each sentence represented by the text data included in the general-purpose text database 11, a vector representing the features according to the context of each sentence is calculated using a well-known method such as BERT (Bidirectional Encoder Representations from Transformers).

[0032] In step S22, each sentence in the target domain text database 12 is vectorized. Here, for each sentence represented by the text data included in the target domain text database 12, a vector representing the features according to the context of each sentence is calculated using the same method as in step S21.

[0033] In step S23, the similarity between each sentence in the general-purpose text database 11 and each sentence in the target domain text database 12 is calculated. Here, the cosine similarity between the vector of each sentence in the general-purpose text database 11 calculated in step S21 and the vector of each sentence in the target domain text database 12 calculated in step S22 is calculated, and the calculated cosine similarity values ​​are summed up for each sentence in the general-purpose text database 11. This makes it possible to calculate the similarity between the text data stored in the general-purpose text database 11 and the target domain text database 12 on a sentence-by-sentence basis.

[0034] In the above, an example has been described in which the similarity between each sentence in the general-purpose text database 11 and each sentence in the target domain text database 12 is calculated by vectorizing each sentence in the general-purpose text database 11 and each sentence in the target domain text database 12 and calculating the cosine similarity based on these vectors, but the method of calculating the similarity is not limited to this. As long as it is possible to properly calculate the similarity between the text data stored in the general-purpose text database 11 and the target domain text database 12 on a sentence-by-sentence basis, any method can be used to calculate the similarity.

[0035] In step S24, sentences whose similarity calculated in step S23 is equal to or greater than a predetermined value are selected from the text data stored in the general-purpose text database 11.

[0036] In step S25, it is determined whether any of the sentences selected in step S24 overlap, and if there are any overlaps, one of them is deleted. This prevents the same sentence from being extracted as candidate text data, and suppresses an increase in the amount of unnecessary data in candidate text database 13.

[0037] In step S26, the text data in the general-purpose text database 11 corresponding to each of the sentences selected in step S24, excluding those deleted in step S25, is extracted as candidate text data. The extracted candidate text data is then recorded and stored in the candidate text database 13.

[0038] After the process of step S26 is executed, the candidate text data extraction process shown in the flowchart of FIG. 4 ends, and the process proceeds to step S30 of FIG.

[0039] By executing the candidate text data extraction process described above, the candidate text data extraction unit 21 calculates the similarity between each sentence represented by the text data included in the general-purpose text database 11 and each sentence represented by the text data included in the target domain text database 12, and can extract text data corresponding to sentences whose similarity is equal to or greater than a predetermined value from the general-purpose text database 11 as candidate text data.

[0040] FIG. 5 is a flowchart showing the flow of the vocabulary size maximization process executed by vocabulary size maximization unit 22 in step S30 of FIG.

[0041] In step S31, a list of words that appear in the target domain text database 12 is created. Here, the list of words that appear can be created by extracting and listing words that are included in each sentence of the text data stored in the target domain text database 12. At this time, if the same word appears multiple times, it is preferable to list them as one word regardless of the number of times it appears.

[0042] In step S32, for each sentence of the candidate text data stored in the candidate text database 13, the frequency of non-appearing words in the target domain text database 12 is calculated. Here, by referring to the appearance word list created in step S31, words that do not exist in the appearance word list in the candidate text data stored in the candidate text database 13 are identified as non-appearing words, and the number of identified non-appearing words is calculated for each sentence. This makes it possible to calculate the frequency of non-appearing words in the candidate text data for each sentence.

[0043] In step S33, based on the frequency of unappeared words calculated in step S32, a ranking score is calculated for each sentence of the candidate text data stored in the candidate text database 13. Here, the ranking score for each sentence of the candidate text data is calculated so that, for example, the sentence with a higher frequency of unappeared words is assigned a higher score value and ranked higher. This makes it possible to rank the sentences represented by the candidate text data in descending order of the frequency of unappeared words calculated for each word in the sentence.

[0044] In step S34, based on the ranking scores calculated in step S33, M sentences (M is a predetermined number) in descending order of rank are selected from candidate text database 13. This makes it possible to select M sentences with high frequencies of unappearing words from the candidate text data stored in candidate text database 13.

[0045] In step S35, text data corresponding to each sentence selected in step S34 is selected from the candidate text data stored in candidate text database 13 and added to the training text data stored in training text database 14. As a result, the selected text data is recorded in training text database 14 as new training text data, and training text database 14 is updated.

[0046] In step S36, it is determined whether or not N (N is a predetermined number) or more pieces of training text data have been added to the training text database 14. Here, the number of training text data added in step S35 executed so far is totaled, and it is determined whether the total number is N or more. As a result, if the total number is N or more, the vocabulary size maximization process shown in the flowchart of FIG. 5 is terminated, and the process proceeds to step S40 in FIG. 3. On the other hand, if the total number is less than N, the process proceeds to step S37.

[0047] In step S37, the word list created in step S31 is updated. Here, a word list of the training text data is created by performing the same process as in step S31 on the training text data added in step S35. The word list can be updated by adding this word list to the word list created in step S31.

[0048] After executing the process of step S37, the process returns to step S32, and the updated word list is used to recalculate the frequency of words that have not yet appeared in the candidate text data. Based on the recalculated frequency of words that have not yet appeared, the processes of steps S33 to S36 are executed again to add more training text data. In this way, the processes of steps S32 to S37 are repeated until the number of training text data added to the training text database 14 reaches N or more. Note that this repetitive process can be realized, for example, by an algorithm called an iterative greedy search.

[0049] By executing the vocabulary size maximization process described above, the vocabulary size maximization unit 22 ranks each sentence represented by the candidate text data stored in the candidate text database 13 in descending order of the frequency of unappeared words calculated for each word in the sentence, and can select learning text data from the candidate text data based on the ranking results.

[0050] According to the embodiment of the present invention described above, the following advantageous effects can be obtained.

[0051] (1) The synthetic training data generation device 1 is a device that generates synthetic training data for training an ASR model used in automatic speech recognition. The synthetic training data generation device 1 includes a general-purpose text database 11 that stores various text data groups collected in advance, a target domain text database 12 that stores text data groups belonging to a target domain of automatic speech recognition, a text data selection unit 20 that compares the general-purpose text database 11 with the target domain text database 12 and selects training text data from the general-purpose text database 11 based on the comparison result, and a synthetic training data generation unit 30 that generates synthetic training data based on the training text data selected by the text data selection unit 20. This configuration makes it possible to generate a variety of training data as training data for the ASR model.

[0052] (2) The text data selection unit 20 has a candidate text data extraction unit 21 that extracts candidate text data that are candidates for training text data on a sentence-by-sentence basis from the general-purpose text database 11, and a vocabulary size maximization unit 22 that calculates the frequency of words that have not yet appeared in the target domain text database 12 for each sentence represented by the candidate text data extracted by the candidate text data extraction unit 21, and selects training text data from the candidate text data based on the calculated frequency of words that have not yet appeared. As a result, the general-purpose text database 11 and the target domain text database 12 can be compared, and training text data can be appropriately selected from the general-purpose text database 11 based on the comparison result.

[0053] (3) The candidate text data extraction unit 21 calculates the similarity between each sentence represented by the text data included in the general-purpose text database 11 and each sentence represented by the text data included in the target domain text database 12 (steps S21 to S23), and extracts text data corresponding to sentences whose similarity is equal to or greater than a predetermined value from the general-purpose text database 11 as candidate text data (steps S24 to S26). In this way, candidate text data that are candidates for learning text data can be reliably extracted from the general-purpose text database 11 on a sentence-by-sentence basis.

[0054] (4) The vocabulary size maximizing unit 22 ranks each sentence represented by the candidate text data in descending order of the frequency of unappeared words calculated for each word in the sentence (steps S32 to S33), and selects learning text data from the candidate text data based on the ranking results (steps S34 to S35). This makes it possible to reliably select text data of sentences with a high frequency of unappeared words from among the candidate text data as learning text data.

[0055] It goes without saying that the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the spirit of the present invention. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the described configurations. Furthermore, it is possible to add, delete, or replace part of the configuration of the above-described embodiments with other configurations.

[0056] Furthermore, the above-mentioned configurations, functional units, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-mentioned configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a hard disk, a recording device such as an SSD, an IC card, an SD card, a DVD, or other recording media.

[0057] In addition, in the above figures, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily show all the control lines and information lines that are actually implemented. For example, it can be considered that almost all components are actually connected to each other. [Explanation of symbols]

[0058] 1: Synthetic training data generator 2: Processor 3: Main memory 4: Auxiliary storage device 5: Input device 6: Output device 7: Communication equipment 10: Storage part 11: General-purpose text database 12: Target domain text database 13: Candidate text database 14: Learning text database 15: Synthetic learning database 20: Text data selection section 21: Candidate text data extraction unit 22: Vocabulary size maximization unit 30: Synthetic training data generation unit

Claims

1. 1. An apparatus for generating synthetic training data for training a model used in automatic speech recognition, comprising: a general-purpose text database in which various pre-collected text data groups are stored; a target domain text database storing a group of text data belonging to a target domain of the automatic speech recognition; a text data selection unit that selects learning text data from the general-purpose text database; a synthetic training data generation unit that generates, as the synthetic training data, speech data obtained by speech synthesis of a sentence represented by the training text data selected by the text data selection unit, The text data selection unit a candidate text data extraction unit that extracts candidate text data that are candidates for the learning text data from the general-purpose text database on a sentence-by-sentence basis; a vocabulary size maximization unit that calculates the frequency of a word that has not yet appeared in the target domain text database for each sentence represented by the candidate text data extracted by the candidate text data extraction unit, and selects the training text data from the candidate text data based on the calculated frequency of the word that has not yet appeared, The candidate text data extraction unit calculates the similarity between each sentence represented by text data included in the general-purpose text database and each sentence represented by text data included in the target domain text database, and extracts text data corresponding to sentences whose similarity is equal to or greater than a predetermined value from the general-purpose text database as the candidate text data.

2. 2. The synthetic training data generation device according to claim 1, The vocabulary size maximization unit ranks each sentence represented by the candidate text data in descending order of the frequency of the unappeared words calculated for each word in the sentence, and selects the training text data from the candidate text data based on the ranking result.

3. 1. A method for generating synthetic training data for training a model for use in automatic speech recognition, comprising: By computer, Calculating the similarity between each sentence represented by text data contained in a general-purpose text database, which stores a group of various text data collected in advance, and each sentence represented by text data contained in a target domain text database, which stores a group of text data belonging to a target domain of the automatic speech recognition; extracting text data corresponding to sentences having a similarity equal to or greater than a predetermined value from the general-purpose text database as candidate text data that is a candidate for learning text data; For each sentence represented by the candidate text data, calculate the frequency of unoccurring words in the target domain text database; selecting the training text data from the candidate text data based on the frequency of the unappeared words; A synthetic training data generation method for generating speech data by speech synthesis of sentences represented by the training text data, as the synthetic training data.

Images