Method, device and storage medium for audio corpus expansion

Abstract

The present application discloses a method, device and storage medium for audio corpus expansion. The method includes: obtaining a first audio corpus, determining a first audio feature corresponding to the first audio corpus; calculating the first audio feature using a pre-trained audio generation model to generate a second audio feature, wherein the audio generation model is obtained based on generative adversarial network training; and determining a second audio corpus corresponding to the second audio feature.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a method, device and storage medium for audio corpus expansion. Background Art

[0002] The current speech synthesis technologies include parameter synthesis, concatenation synthesis, and end-to-end speech generation. Currently, concatenation synthesis and end-to-end speech synthesis are more commonly used in the industry. Concatenation synthesis mainly involves the recording personnel recording according to the text to generate a large recording library. When there is a demand for speech synthesis, the corresponding recording is found in the recording library according to the text for recording splicing for user use. End-to-end speech synthesis mainly uses deep learning technology to introduce an attention mechanism. Through a large number of recordings by the recording personnel, text annotation is performed to train a network model with an encoder-decoder structure, and then the text is passed in to perform speech synthesis through the trained model for user use. End-to-end synthesis has certain requirements for the amount of corpus, generally requiring up to 10 hours of recording. At the same time, the corpus needs to cover all common words to minimize the error rate of the speech synthesis system. At this time, if it is recorded manually, it usually takes 3 months to record, and the later corpus repair may require the recording personnel to record. Therefore, the process of recording and maintaining corpus using end-to-end speech synthesis is too dependent on manpower and cumbersome, which consumes a lot of time and manpower costs, affecting economic applicability.

[0003] With respect to the technical problem in the prior art that the recording and maintenance of corpus in the speech synthesis process is too dependent on manpower, thus consuming a lot of time and manpower costs, no effective solution has been proposed so far. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, device and storage medium for expanding audio corpus, so as to at least solve the technical problem in the prior art that the recording and maintenance of corpus in the speech synthesis process is too dependent on manpower, thus consuming a lot of time and manpower costs.

[0005] According to one aspect of an embodiment of the present disclosure, a method for expanding an audio corpus is provided, including: acquiring a first audio corpus, determining a first audio feature corresponding to the first audio corpus; calculating the first audio feature using a pre-trained audio generation model to generate a second audio feature, wherein the audio generation model is trained based on a generative adversarial network; and determining a second audio corpus corresponding to the second audio feature.

[0006] According to another aspect of an embodiment of the present disclosure, a storage medium is further provided, the storage medium including a stored program, wherein when the program is running, a processor executes any one of the methods described above.

[0007] According to another aspect of an embodiment of the present disclosure, a device for expanding an audio corpus is also provided, including: a data acquisition module, used to acquire a first audio corpus and determine a first audio feature corresponding to the first audio corpus; a feature generation module, used to calculate the first audio feature using a pre-trained audio generation model to generate a second audio feature, wherein the audio generation model is obtained based on generative adversarial network training; and a corpus generation module, used to determine a second audio corpus corresponding to the second audio feature.

[0008] According to another aspect of an embodiment of the present disclosure, a device for expanding an audio corpus is also provided, including: a processor; and a memory connected to the processor, for providing the processor with instructions for processing the following processing steps: obtaining a first audio corpus, and determining a first audio feature corresponding to the first audio corpus; calculating the first audio feature using a pre-trained audio generation model to generate a second audio feature, wherein the audio generation model is obtained based on generative adversarial network training; and determining a second audio corpus corresponding to the second audio feature.

[0009] In the disclosed embodiment, a large amount of corpus can be generated by using a speech generation model obtained by training a generative adversarial network, which can meet the training requirements of an end-to-end speech synthesis system. Compared with the prior art, this solution does not require a large amount of time and labor costs to record corpus, so it can quickly solve the problem of insufficient corpus in speech synthesis, reduce the time for the recorder to record corpus, and avoid the drawback of over-reliance on manpower in recording corpus. In addition, it can also ensure that high-quality speech corpus is obtained at the same time. Therefore, the technical effect of obtaining a large amount of corpus efficiently and with high quality is achieved. This solves the technical problem that the recording and maintenance of corpus in the speech synthesis process in the prior art is too dependent on manpower, thus consuming a lot of time and labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0011] Figure 1 is a hardware structure block diagram of a computing device for implementing the method according to Embodiment 1 of the present disclosure;

[0012] Figure 2 is a flowchart of the method for expanding audio corpus according to the first aspect of Embodiment 1 of the present disclosure;

[0013] Figure 3 is a schematic diagram of the audio generation model training and use process according to Embodiment 1 of the present disclosure;

[0014] Figure 4 is a schematic diagram of the model structure of the audio generation model according to Embodiment 1 of the present disclosure;

[0015] Figure 5 is a schematic diagram of an apparatus for expanding audio corpus according to Embodiment 2 of the present disclosure; and

[0016] Figure 6 It is a schematic diagram of the device for expanding the audio corpus according to Embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0018] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0019] Example 1

[0020] According to this embodiment, a method embodiment of audio corpus expansion is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0021] The method embodiment provided in this embodiment can be executed in a server or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computing device for implementing a method for audio corpus expansion. Figure 1As shown, the computing device may include one or more processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory for storing data, and a transmission device for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0022] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computing device. As involved in the embodiments of the present disclosure, the data processing circuitry acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0023] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method of audio corpus expansion in the embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the method of audio corpus expansion of the above-mentioned application is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computing device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0024] The transmission device is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computing device. In one example, the transmission device includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0025] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of the computing device.

[0026] It should be noted that, in some optional embodiments, the above Figure 1 The computing device shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computing devices described above.

[0027] In the above operating environment, according to the first aspect of this embodiment, a method for expanding an audio corpus is provided. The method can be applied to a server of a speech synthesis system to expand the corpus. Figure 2 A schematic diagram showing the process of the method is shown in FIG. Figure 2 As shown, the method includes:

[0028] S202: Acquire a first audio corpus and determine a first audio feature corresponding to the first audio corpus;

[0029] S204: Calculate the first audio feature using a pre-trained audio generation model to generate a second audio feature, wherein the audio generation model is trained based on a generative adversarial network; and

[0030] S206: Determine a second audio corpus corresponding to the second audio feature.

[0031] As described in the background technology, the current speech synthesis technology includes parameter synthesis, splicing synthesis, and end-to-end speech generation. At present, the industry uses splicing synthesis and end-to-end speech synthesis more. Splicing synthesis mainly involves the recording personnel recording according to the text to generate a large recording library. When there is a demand for speech synthesis, the corresponding recording is found in the recording library according to the text for recording splicing for user use. End-to-end speech synthesis mainly uses deep learning technology to introduce an attention mechanism, and through a large number of recordings of the recording personnel, the text is annotated to train a network model of an Encoder-Decoder structure, and then the text is passed in to perform speech synthesis through the trained model for user use. End-to-end synthesis has a certain amount of requirements for corpus, generally requiring up to 10 hours of recording. At the same time, the corpus needs to cover all common words to minimize the error rate of the speech synthesis system. At this time, if it is generally recorded manually, it takes 3 months to record, and the later corpus repair may require the recording personnel to record. Therefore, the process of recording and maintaining corpus using end-to-end speech synthesis is too dependent on manpower and cumbersome, and it takes a lot of time and manpower costs, affecting economic applicability.

[0032] In view of the technical problems existing in the background technology, in the technical solution of this embodiment, in step S202, the system server first obtains the first audio corpus and determines the first audio feature corresponding to the first audio corpus. For example: the system can filter out the first audio corpus from the corpus previously recorded by the recording personnel. If it has not been recorded before, the first audio corpus can be re-recorded. In addition, audio can be obtained from the Internet as the first audio corpus. Among them, the number of first audio corpora can be selected according to the actual application scenario. The number of first audio corpora in this embodiment is preferably 40,000. After obtaining the first audio corpus, the server also needs to determine the first audio feature corresponding to the first audio corpus. In a preferred example, the Mel spectrum feature corresponding to the first audio corpus is determined.

[0033] Further, in step S204, the server calculates the first audio feature using a pre-trained audio generation model to generate a second audio feature. The audio generation model is obtained based on adversarial network training. In a preferred embodiment, the speech generation model is obtained based on bidirectional generative adversarial network (BiGan) training. For example, refer to Figure 3As shown, 5000 speech corpora are used as training samples to train the generative adversarial network to obtain the speech generation model. After the model training is completed, the server can input the first audio feature into the audio generation model, and then the model can generate the second audio feature based on the first audio feature. For example, the first audio feature of the above 40,000 corpora is input into the speech generation model based on the generative adversarial network, and 40,000 second audio features can be generated accordingly.

[0034] Finally, in step S206, the server determines the second audio corpus corresponding to the second audio feature. For example, if the second audio feature is a Mel-spectrogram feature, the server may convert the Mel-spectrogram feature into the corresponding second audio corpus.

[0035] In addition, after obtaining the generated second audio corpus, the corpus can also be processed for speech synthesis. For example: the server converts the generated 40,000 second audio corpus into text through ASR, and then asks the annotation personnel to check the text and the generated recordings. After the inspection is completed, the common phrase coverage statistics of the 40,000 recordings are performed. There are about 150,000 common words. If full coverage can be achieved, the corpus requirements are met. The 40,000 generated corpus basically meets the 150,000 common words. The corpus that meets the requirements is used to train the end-to-end speech synthesis system.

[0036] In this way, a large amount of corpus can be generated using the speech generation model obtained by training based on the generative adversarial network, which can meet the training of the end-to-end speech synthesis system. Compared with the prior art, this solution does not need to spend a lot of time and manpower costs to record the corpus, so it can quickly solve the problem of insufficient corpus in speech synthesis, reduce the time for the recorder to record the corpus, and avoid the drawback of over-reliance on manpower in recording the corpus. In addition, it can also ensure that high-quality speech corpus is obtained at the same time. Therefore, the technical effect of obtaining a large amount of corpus efficiently and with high quality is achieved. In addition, the technical problem that the recording and maintenance of the corpus in the speech synthesis process in the prior art is too dependent on manpower, thus consuming a lot of time and manpower costs is solved.

[0037] Optionally, the first audio feature is calculated using a pre-trained audio generation model to generate a second audio feature, including: using an encoder of the audio generation model to calculate the first audio feature to generate an audio vector corresponding to the first audio feature; and using a decoder of the audio generation model to calculate the audio vector to generate a second audio feature.

[0038] Specifically, the speech generation model based on the bidirectional generative adversarial network (BiGan) can generally include an encoder, a decoder (also referred to as a generator) and a discriminator. In the operation of calculating the first audio feature and generating the second audio feature using the pre-trained audio generation model, the first audio feature is first calculated using the encoder of the audio generation model to generate an audio vector corresponding to the first audio feature. Further, the audio vector is calculated using the decoder of the audio generation model to generate the second audio feature.

[0039] Optionally, refer to Figure 3 and Figure 4 As shown, this embodiment trains the audio generation model through the following steps:

[0040] Step 1: Obtain a predetermined amount of recording sample data from the recording corpus, for example, extract 5,000 voice corpora. Then generate real sample audio features (for example, Mel spectrum features) corresponding to the recording sample data, and the sample audio features are represented by x, for example;

[0041] Step 2: sampling from a certain distribution (this distribution can be arbitrary, here we take Gaussian distribution) to obtain a first eigenvector, for example: obtaining a predetermined number of first eigenvectors from samples that obey a normal distribution, the first eigenvector is represented by z', for example;

[0042] Step 3: Calculate the sample audio feature using the encoder of the bidirectional generative adversarial network (BiGan) to determine the second feature vector corresponding to the sample audio feature, that is, the input sample audio feature is passed through the encoder to obtain the second feature vector, and the second feature vector is represented by z, for example;

[0043] Step 4: Calculate the first feature vector using a decoder of a bidirectional generative adversarial network (BiGan) to determine a generated audio feature corresponding to the first feature vector, where the generated audio feature is represented by x', for example;

[0044] Step 5: Pair the sample audio feature with the corresponding second feature vector, and pair the first feature vector with the corresponding generated audio feature to obtain multiple (x, z) and (x', z'), and then input (x, z) and (x', z') into the discriminator of the bidirectional generative adversarial network (BiGan); and

[0045] Step 6: Since some (x, z) and (x', z') are generated by the encoder and some by the decoder, input these (x, z) and (x', z') into the discriminator to let it determine whether they are generated by the encoder or the decoder. Continuously adjust the parameters of the bidirectional generative adversarial network according to the recognition results of the discriminator until the discriminator cannot make an accurate judgment, and the model training is successful.

[0046] In a specific example, the training process is as follows:

[0047] Initialize encoder, decoder, and discriminator (three independent neural networks)

[0048] Iterate sequentially:

[0049] Sample x1, x2, xM from 5000 recordings

[0050] Input x1, x2, xM into the encoder to generate output z = En(x)

[0051] Sample z'1,z'2,z'M from a distribution

[0052] Input z'1, z'2, z'M into the decoder to generate output x'=De(z)

[0053] Input (x,z) into the discriminator and update the discriminator (mainly by increasing the score of Dis(x,z) and decreasing the score of Dis(x',z'))

[0054] Update the encoder and decoder. Note: This is the opposite of updating the discriminator above. Because the encoder and decoder are trying to confuse the discriminator, so the previous step is to reduce Dis(x',z'), here we also need to reduce Dis(x,z) to achieve a balance;

[0055] After the encoder, decoder, and discriminator network training is completed, they can be used for audio generation.

[0056] Optionally, obtaining a predetermined amount of recording sample data from a recording corpus includes: performing similarity matching on the recording data in the recording corpus; and selecting a predetermined amount of data from the recording data having a similarity less than a predetermined threshold as the recording sample data.

[0057] Specifically, in the operation of obtaining a predetermined number of recording sample data from the recording corpus, the server needs to perform similarity matching on the recording data in the recording corpus, for example, calculating the similarity between the two corpora of "Hello, this is the house sales department" and "Hello, do you have a house to sell?". Further, the similarity value is compared with a preset threshold, and a predetermined number of recording data with a similarity less than the preset threshold is selected as recording sample data, for example, a predetermined number of recording data with a similarity less than 30% is selected as recording sample data. Thus, the model training can be made more accurate.

[0058] Optionally, performing similarity matching on the recorded data in the recorded data corpus includes: determining the number of identical characters contained in the recorded data in the recorded data corpus; and performing similarity matching on the recorded data in the recorded data corpus based on the number of identical characters.

[0059] Specifically, in the operation of similarity matching of the recorded data in the recorded corpus, the server first determines the number of identical characters contained in the recorded data in the recorded corpus, for example, the two corpora of 'Hello, this is the house sales department' and 'Hello, do you have a house to sell' contain 4 identical characters. Further, based on the number of identical characters, the recorded data in the recorded corpus are similarity matched, for example: the length of the above text is 11, the number of identical characters is 4, then the similarity of the above two corpora is 4 / 11. Thus, the similarity between the texts can be quickly determined by a simple algorithm.

[0060] In addition, reference Figure 1 As shown, according to the second aspect of this embodiment, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is run, a processor executes any one of the above methods.

[0061] Therefore, according to this embodiment, a large amount of corpus can be generated by using the speech generation model obtained by training based on the generative adversarial network, so as to meet the training of the end-to-end speech synthesis system. Compared with the prior art, this solution does not need to spend a lot of time and manpower costs to record the corpus, so it can quickly solve the problem of insufficient corpus in speech synthesis, reduce the time for the recorder to record the corpus, and avoid the drawback of over-reliance on manpower in recording the corpus. In addition, it can also ensure that high-quality speech corpus is obtained at the same time. Therefore, the technical effect of obtaining a large amount of corpus efficiently and with high quality is achieved. In addition, the technical problem that the recording and maintenance of the corpus in the speech synthesis process in the prior art is too dependent on manpower, thus consuming a lot of time and manpower costs is solved.

[0062] In addition, this solution has a good control over the cost of recording corpus, reducing the company's over-reliance on recording personnel and reducing recording costs, while ensuring high-quality speech corpus for the speech synthesis system. It quickly solves the problem of insufficient corpus in speech synthesis and reduces the time it takes for recorders to record corpus. Recorders can be replaced by annotators. Generally, the threshold and cost of annotators are lower than those of recorders. Multiple annotators can be used to generate recording annotations to replace recording personnel, reducing the company's costs and risks for recording personnel.

[0063] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0064] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0065] Example 2

[0066] Figure 5 The apparatus 500 for expanding the audio corpus according to this embodiment is shown, and the apparatus 500 corresponds to the method according to the first aspect of embodiment 1. Figure 5 As shown, the device 500 includes: a data acquisition module 510, used to acquire a first audio corpus and determine a first audio feature corresponding to the first audio corpus; a feature generation module 520, used to calculate the first audio feature using a pre-trained audio generation model to generate a second audio feature, wherein the audio generation model is obtained based on generative adversarial network training; and a corpus generation module 530, used to determine a second audio corpus corresponding to the second audio feature.

[0067] Optionally, the audio generation model is trained based on a bidirectional generative adversarial network (BiGan).

[0068] Optionally, the feature generation module 520 includes: a vector generation submodule, used to use the encoder of the audio generation model to calculate the first audio feature and generate an audio vector corresponding to the first audio feature; and a feature generation submodule, used to use the decoder of the audio generation model to calculate the audio vector and generate a second audio feature.

[0069] Optionally, the data acquisition module 510 includes: a feature determination submodule, configured to determine a Mel-spectrogram feature corresponding to the first audio corpus.

[0070] Optionally, the device 500 also includes: a training module, used to train the audio generation model according to the following steps: obtaining a predetermined number of recording sample data from a recording corpus, and generating sample audio features corresponding to the recording sample data; obtaining a predetermined number of first feature vectors from samples that obey a normal distribution; using an encoder of a bidirectional generative adversarial network (BiGan) to calculate the sample audio features, and determine a second feature vector corresponding to the sample audio features; using a decoder of a bidirectional generative adversarial network (BiGan) to calculate the first feature vector, and determine the generated audio features corresponding to the first feature vector; pairing the sample audio features with the corresponding second feature vectors, pairing the first feature vectors with the corresponding generated audio features, and inputting them into the discriminator of the bidirectional generative adversarial network (BiGan); and adjusting the bidirectional generative adversarial network according to the recognition result of the discriminator.

[0071] Optionally, obtaining a predetermined amount of recording sample data from a recording corpus includes: performing similarity matching on the recording data in the recording corpus; and selecting a predetermined amount of data from the recording data having a similarity less than a predetermined threshold as the recording sample data.

[0072] Optionally, performing similarity matching on the recorded data in the recorded data corpus includes: determining the number of identical characters contained in the recorded data in the recorded data corpus; and performing similarity matching on the recorded data in the recorded data corpus based on the number of identical characters.

[0073] Therefore, according to this embodiment, a large amount of corpus can be generated by using the speech generation model obtained by training based on the generative adversarial network, so as to meet the training of the end-to-end speech synthesis system. Compared with the prior art, this solution does not need to spend a lot of time and manpower costs to record the corpus, so it can quickly solve the problem of insufficient corpus in speech synthesis, reduce the time for the recorder to record the corpus, and avoid the drawback of over-reliance on manpower in recording the corpus. In addition, it can also ensure that high-quality speech corpus is obtained at the same time. Therefore, the technical effect of obtaining a large amount of corpus efficiently and with high quality is achieved. In addition, the technical problem that the recording and maintenance of the corpus in the speech synthesis process in the prior art is too dependent on manpower, thus consuming a lot of time and manpower costs is solved.

[0074] Example 3

[0075] Figure 6 The apparatus 600 for expanding the audio corpus according to this embodiment is shown, and the apparatus 600 corresponds to the method according to the first aspect of embodiment 1. Figure 6 As shown, the device 600 includes: a processor 610; and a memory 620, connected to the processor 610, for providing instructions for the processor 610 to process the following processing steps: obtaining a first audio corpus, determining a first audio feature corresponding to the first audio corpus; calculating the first audio feature using a pre-trained audio generation model to generate a second audio feature, wherein the audio generation model is obtained based on generative adversarial network training; and determining a second audio corpus corresponding to the second audio feature.

[0076] Optionally, the audio generation model is trained based on a bidirectional generative adversarial network (BiGan).

[0077] Optionally, the first audio feature is calculated using a pre-trained audio generation model to generate a second audio feature, including: using an encoder of the audio generation model to calculate the first audio feature to generate an audio vector corresponding to the first audio feature; and using a decoder of the audio generation model to calculate the audio vector to generate a second audio feature.

[0078] Optionally, determining a first audio feature corresponding to the first audio corpus includes: determining a Mel-spectrogram feature corresponding to the first audio corpus.

[0079] Optionally, the memory 620 is also used to provide the processor 610 with instructions for training the audio generation model according to the following steps: obtaining a predetermined number of recording sample data from a recording corpus, and generating sample audio features corresponding to the recording sample data; obtaining a predetermined number of first feature vectors from samples that obey a normal distribution; using an encoder of a bidirectional generative adversarial network (BiGan) to calculate the sample audio features, and determine a second feature vector corresponding to the sample audio features; using a decoder of a bidirectional generative adversarial network (BiGan) to calculate the first feature vector, and determine the generated audio features corresponding to the first feature vector; pairing the sample audio features with the corresponding second feature vectors, pairing the first feature vectors with the corresponding generated audio features, and inputting them into the discriminator of the bidirectional generative adversarial network (BiGan); and adjusting the bidirectional generative adversarial network according to the recognition result of the discriminator.

[0080] Optionally, obtaining a predetermined amount of recording sample data from a recording corpus includes: performing similarity matching on the recording data in the recording corpus; and selecting a predetermined amount of data from the recording data having a similarity less than a predetermined threshold as the recording sample data.

[0081] Optionally, performing similarity matching on the recorded data in the recorded data corpus includes: determining the number of identical characters contained in the recorded data in the recorded data corpus; and performing similarity matching on the recorded data in the recorded data corpus based on the number of identical characters.

[0082] Therefore, according to this embodiment, a large amount of corpus can be generated by using the speech generation model obtained by training based on the generative adversarial network, so as to meet the training of the end-to-end speech synthesis system. Compared with the prior art, this solution does not need to spend a lot of time and manpower costs to record the corpus, so it can quickly solve the problem of insufficient corpus in speech synthesis, reduce the time for the recorder to record the corpus, and avoid the drawback of over-reliance on manpower in recording the corpus. In addition, it can also ensure that high-quality speech corpus is obtained at the same time. Therefore, the technical effect of obtaining a large amount of corpus efficiently and with high quality is achieved. In addition, the technical problem that the recording and maintenance of the corpus in the speech synthesis process in the prior art is too dependent on manpower, thus consuming a lot of time and manpower costs is solved.

[0083] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0084] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0086] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0087] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0088] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0089] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for expanding audio corpus, characterized in that: include: Acquire a first audio corpus, and determine a first audio feature corresponding to the first audio corpus; Calculate the first audio feature using a pre-trained audio generation model to generate a second audio feature, wherein the audio generation model is trained based on a generative adversarial network; as well as Determining a second audio corpus corresponding to the second audio feature; The speech generation model of the generative adversarial network includes a generator consisting of an encoder, a decoder, and a discriminator; The audio generation model for training a generative adversarial network includes: Obtain a predetermined amount of recording sample data from the recording corpus, and generate a real sample audio feature x corresponding to the recording sample data; Sample the first eigenvector z' from an arbitrary distribution; Using the encoder of the generative adversarial network to calculate the sample audio feature x, and determine a second feature vector z corresponding to the sample audio feature; Calculate the first feature vector z' using a decoder of the generative adversarial network to determine a generated audio feature x' corresponding to the first feature vector; Pair the sample audio feature x with the corresponding second feature vector z, and pair the first feature vector z' with the corresponding generated audio feature x' to obtain multiple (x, z) and (x', z'), and then input (x, z) and (x', z') into the discriminator of the generative adversarial network; The discriminator determines whether (x, z) and (x', z') are generated by the encoder or decoder. The parameters of the generative adversarial network are adjusted according to the recognition result of the discriminator until the discriminator cannot make a judgment and the model training is completed.

2. The method according to claim 1, characterized in that The audio generation model is trained based on a bidirectional generative adversarial network (BiGan).

3. The method according to claim 2, characterized in that Calculating the first audio feature using a pre-trained audio generation model to generate a second audio feature includes: Calculating the first audio feature using an encoder of the audio generation model to generate an audio vector corresponding to the first audio feature; and The audio vector is calculated using a decoder of the audio generation model to generate the second audio feature.

4. The method according to claim 1, characterized in that: Determining a first audio feature corresponding to the first audio corpus includes: determining a Mel-spectrogram feature corresponding to the first audio corpus.

5. The method according to claim 2, characterized in that: The audio generation model is also trained according to the following steps: Acquire a predetermined amount of recording sample data from a recording corpus, and generate sample audio features corresponding to the recording sample data; Obtain a predetermined number of first eigenvectors from samples that obey a normal distribution; Calculating the sample audio feature using the encoder of the bidirectional generative adversarial network (BiGan) to determine a second feature vector corresponding to the sample audio feature; Calculating the first feature vector using a decoder of the bidirectional generative adversarial network (BiGan) to determine a generated audio feature corresponding to the first feature vector; Pairing the sample audio feature with the corresponding second feature vector, pairing the first feature vector with the corresponding generated audio feature, and inputting them into the discriminator of the bidirectional generative adversarial network (BiGan); as well as The bidirectional generative adversarial network is adjusted according to the recognition result of the discriminator.

6. The method according to claim 5, characterized in that Obtain a predetermined amount of recording sample data from the recording corpus, including: Performing similarity matching on the recording data in the recording corpus; and A predetermined amount of data from the recorded data whose similarity is less than a predetermined threshold is selected as the recorded sample data.

7. The method according to claim 6, characterized in that Performing similarity matching on the recording data in the recording corpus includes: Determining the number of identical characters contained in the recording data in the recording corpus; and Based on the number of identical characters, similarity matching is performed on the recording data in the recording corpus.

8. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the processor executes the method according to any one of claims 1 to 7.

9. A device for expanding audio corpus, characterized in that: include: A data acquisition module, configured to acquire a first audio corpus and determine a first audio feature corresponding to the first audio corpus; A feature generation module, configured to calculate the first audio feature using a pre-trained audio generation model to generate a second audio feature, wherein the audio generation model is trained based on a generative adversarial network; as well as A corpus generation module, used to determine a second audio corpus corresponding to the second audio feature; The speech generation model of the generative adversarial network includes a generator consisting of an encoder, a decoder, and a discriminator; The audio generation model for training a generative adversarial network includes: Obtain a predetermined amount of recording sample data from the recording corpus, and generate a real sample audio feature x corresponding to the recording sample data; A first feature vector z' is sampled from an arbitrary distribution; the encoder of the generative adversarial network is used to calculate the sample audio feature x to determine the second feature vector z corresponding to the sample audio feature; the decoder of the generative adversarial network is used to calculate the first feature vector z' to determine the generated audio feature x' corresponding to the first feature vector; the sample audio feature x is paired with the corresponding second feature vector z, and the first feature vector z' is paired with the corresponding generated audio feature x' to obtain multiple (x, z) and (x', z'), and then (x, z) and (x', z') are input into the discriminator of the generative adversarial network; the discriminator determines whether (x, z) and (x', z') are generated by the encoder or the decoder, and the parameters of the generative adversarial network are adjusted according to the recognition result of the discriminator until the discriminator cannot make a judgment and the model training is completed.

10. A device for expanding audio corpus, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Acquire a first audio corpus, and determine a first audio feature corresponding to the first audio corpus; Calculate the first audio feature using a pre-trained audio generation model to generate a second audio feature, wherein the audio generation model is trained based on a generative adversarial network; as well as Determining a second audio corpus corresponding to the second audio feature; The speech generation model of the generative adversarial network includes a generator consisting of an encoder, a decoder, and a discriminator; The audio generation model for training a generative adversarial network includes: Obtain a predetermined amount of recording sample data from the recording corpus, and generate a real sample audio feature x corresponding to the recording sample data; A first feature vector z' is sampled from an arbitrary distribution; the encoder of the generative adversarial network is used to calculate the sample audio feature x to determine the second feature vector z corresponding to the sample audio feature; the decoder of the generative adversarial network is used to calculate the first feature vector z' to determine the generated audio feature x' corresponding to the first feature vector; the sample audio feature x is paired with the corresponding second feature vector z, and the first feature vector z' is paired with the corresponding generated audio feature x' to obtain multiple (x, z) and (x', z'), and then (x, z) and (x', z') are input into the discriminator of the generative adversarial network; the discriminator determines whether (x, z) and (x', z') are generated by the encoder or the decoder, and the parameters of the generative adversarial network are adjusted according to the recognition result of the discriminator until the discriminator cannot make a judgment and the model training is completed.

Images