Voice processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a voice processing method and device, equipment and a storage medium. A method for voice processing includes: generating a text sample set for a plurality of language categories based on source texts of the plurality of language categories, the sample number distribution of the text sample set on the plurality of language categories satisfying a predetermined condition; based on each text sample in the text sample set, generating a voice sample set for a plurality of language categories by using a voice synthesis model; and generating a training sample set for the voice processing model based on the text sample set and the voice sample set. Therefore, the speech sample set can be synthesized according to the texts of the multiple language categories, so that the performance of the speech processing model for the multiple language categories can be improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for speech processing. Background Art

[0002] With the continuous development of science and technology, speech processing is also taking up an increasingly large proportion in human-computer interaction applications. For example, text can be converted into corresponding speech signals through recognition and understanding. Correspondingly, speech signals can also be converted into corresponding text through recognition and understanding. Speech processing brings many conveniences to users. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for speech processing is provided, comprising: presenting a virtual resource identifier in a content display page of media content; generating text sample sets for multiple language categories based on source texts of multiple language categories, wherein the distribution of the number of samples of the text sample sets in the multiple language categories satisfies a predetermined condition; generating speech sample sets for the multiple language categories using a speech synthesis model based on each text sample in the text sample set; and generating a training sample set for the speech processing model based on the text sample set and the speech sample set.

[0004] In a second aspect of the present disclosure, a device for speech processing is provided, comprising: a text sample set generation module, configured to generate text sample sets for multiple language categories based on source texts of multiple language categories, wherein the distribution of the number of samples of the text sample sets in the multiple language categories meets a predetermined condition; a speech sample set generation module, configured to generate speech sample sets for the multiple language categories using a speech synthesis model based on each text sample in the text sample set; and a training sample set generation module, configured to generate a training sample set for the speech processing model based on the text sample set and the speech sample set.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method of the first aspect is implemented.

[0008] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A flowchart showing a process for speech processing according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram showing an example architecture for speech processing according to some embodiments of the present disclosure;

[0013] Figure 4 A block diagram showing an apparatus for speech processing according to some embodiments of the present disclosure; and

[0014] Figure 5 A block diagram of a device capable of implementing one or more embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0016] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0017] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0018] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0019] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0020] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0021] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0023] As briefly mentioned above, speech signals can be converted into corresponding texts through recognition and understanding. It has a wide range of applications in human-computer interaction, voice assistants, meeting records, and the like. Conventionally, the Automatic Speech Recognition (ASR) model can be trained by self-supervised pre-training and downstream task fine-tuning to convert speech signals into corresponding texts. That is, by performing self-supervised pre-training on a large amount of unlabeled speech data and fine-tuning the labeled data for specific tasks. In this way, although the unlabeled data can be fully utilized, the performance of the downstream task is limited by the scale of the fine-tuning data.

[0024] Furthermore, a speech recognition model is trained by using a large amount of annotated data with noise to convert the speech signal into corresponding text. Then, in this way, a large amount of annotated data needs to be collected. Accordingly, the speech signal can also be converted into corresponding text by including a self-attention mechanism, an encoder and a decoder, and a model that supports multi-task learning. However, training such a model requires a large amount of annotated data and a large number of different types of languages, resulting in high costs. The recognition accuracy for low-resource languages ​​is low, and the uneven distribution of data easily leads to large differences in the performance of the model between different languages.

[0025] In view of this, according to some embodiments of the present disclosure, a speech processing scheme is proposed. According to the scheme of the embodiment of the present disclosure, based on source texts of multiple language categories, text sample sets for multiple language categories are generated, wherein the sample number distribution of the text sample sets in multiple language categories meets a predetermined condition. Accordingly, according to each text sample in the text sample set, a speech sample set for multiple language categories is generated using a speech synthesis model. Then, according to the text sample set and the speech sample set, a training sample set for the speech processing model is generated.

[0026] In this way, by synthesizing speech sample sets based on texts of multiple language categories, it is possible to ensure that the quality of the synthesized speech sample sets meets the training requirements of the speech processing model. Furthermore, based on the synthesized speech sample sets, the quality of texts of at least one language category generated by the speech processing model can be improved. Furthermore, by ensuring that the number of samples between the multiple language categories included in the text sample set meets a predetermined condition, the recognition accuracy of the speech processing model on low-resource languages ​​and the performance on translation tasks can be improved.

[0027] Example embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0028] Example Environment

[0029] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1In the environment 100, it is expected to train and use such a speech processing model 130, which is configured for a variety of application environments. For example, the speech processing model 130 can output text for the sample set 110 of multiple language categories based on the sample set 110 of multiple language categories input by the user, etc. In some embodiments, the multiple language categories can refer to various languages ​​formed according to the phonetics, vocabulary, grammatical rules, etc. of the language. For example, the multiple language categories can include languages ​​of different language families. For another example, the multiple language categories can include different languages, such as English, Chinese, etc. The sample set 110 of multiple language categories may include a text sample set 112 and a speech sample set 113. In some embodiments, the speech sample set 113 may be obtained by utilizing a speech synthesis model 170. For example, a text sample and multiple speaker samples can be input into the speech synthesis model 170 to obtain the speech sample set 113.

[0030] like Figure 1 As shown, environment 100 includes a model training system 150 . Figure 1 The upper part shows the process of the model training phase, and the lower part shows the process of the model application phase. Before training, the parameter values ​​of the speech processing model 130 may have initial values, or may have pre-trained parameter values ​​obtained through a pre-training process. The speech processing model 130 may be trained based on the text sample set 112 and the speech sample set obtained by the speech synthesis model 170. During the training process, the parameter values ​​of the speech processing model 130 may be updated and adjusted. After the speech processing model 130 is trained, the speech processing model 130 may be obtained. ′ At this time, the speech processing model 130 ′ The parameter values ​​of the speech processing model 130 have been updated, and based on the updated parameter values, the speech processing model 130 can be used to implement various types of speech processing tasks in the model application stage. For example, the speech processing model 130 can be used to implement speech recognition tasks, translation tasks, and the like.

[0031] In the model training stage, the speech processing model 130 can be trained based on a sample set 110 of multiple language categories including a text sample set 112 and a speech sample set 113 for training the speech processing model 130, and using a model training system 150. Here, the speech processing model 130 can generate a speech sample set 113 corresponding to the speaker sample set through a speech synthesis model 170 based on the sample set 110 of multiple language categories. Subsequently, the model training system 150 can also train the speech processing model 130 based on the multiple text samples included in the text sample set 112 and the multiple language samples included in the speech sample set 113. Specifically, the training process can be iteratively performed using a large number of training samples. After the training is completed, the speech processing model 130 may include knowledge about the task to be processed. In the model application stage, the speech processing model 130 ′ (The speech processing model 130 at this time ′ The trained parameter values ​​can be used to perform the corresponding task. For example, in a speech recognition task, the speech processing model 130 ′ The target speech 142 can be received and the corresponding text 144 of the target language category can be output. In the speech recognition task, the target speech 142 and the text 144 are of the same language category. For example, in the speech translation task, the speech processing model 130 ′ A target speech 142 may be received, and a corresponding target language-category text 144 may be output. In a speech translation task, the target speech 142 and the text 144 are of different language categories.

[0032] exist Figure 1 In the present invention, the model training system 150 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. The terminal device may involve any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Servers include but are not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0033] It should be understood that Figure 1 The components and arrangements in the environment 100 shown are merely examples, and a computing system suitable for implementing the exemplary implementation described in the present disclosure may include one or more different components, other components, and / or different arrangements. The implementation of the present disclosure is not limited in this respect.

[0034] Some example embodiments of the present disclosure will be described below with reference to the accompanying drawings. In the following, example embodiments will be described mainly with respect to a model training system 150 .

[0035] The following will refer to Figure 2 The present disclosure is described for speech processing. Figure 2 A flow chart of an example process 200 for speech processing according to some embodiments of the present disclosure is shown. In an embodiment of the present disclosure, it is desirable to train such a speech processing model 130, which is configured to be trained by a synthesized speech sample set and then output text. That is, after being trained by the synthesized speech sample, high-quality text is accurately output based on the input speech.

[0036] Therefore, the following will refer to Figure 2 and Figure 3 Description The present disclosure obtains a training sample set for training a speech processing model to accurately output high-quality texts in different language categories. Figure 3 A schematic diagram of an example architecture 300 for speech processing according to some embodiments of the present disclosure is shown.

[0037] Reference Figure 2 In block 210, the model training system 150 generates a text sample set for multiple language categories based on the source texts of the multiple language categories. In some embodiments, the distribution of the number of samples of the text sample set in the multiple language categories satisfies a predetermined condition. In some examples, the model training system 150 can obtain source texts of multiple language categories from various public resources to generate the text sample set 112. For example, the source texts of multiple language categories can be obtained from various types of public resources such as encyclopedias, websites, and e-books.

[0038] The number of each language category in the multiple language categories included in the text sample set 112 meets the preset conditions. That is, the number of each language category in the multiple language categories acquired by the model training system 150 is basically the same. By making the number of each language category in the multiple language categories basically the same, the recognition accuracy of the speech processing model on low-resource languages ​​can be improved. Low-resource languages ​​can indicate languages ​​that lack large-scale text data, annotated data, or computing resource support, for example, languages ​​with a small number of users.

[0039] In some embodiments, model training system 150 may perform a preprocessing operation on a source text of a given language category in a plurality of language categories to generate a processed source text. In some examples, model training system 150 may perform a preprocessing operation on a source text of each language category in a plurality of language categories of source texts collected by it.

[0040] In some embodiments, the model training system 150 can delete texts whose semantic information does not meet the predetermined conditions in a given source text. For source texts of any language category among source texts of multiple language categories, if the model training system 150 determines that there are texts in the source text of a certain language category that do not meet the predetermined conditions, the texts that do not meet the predetermined conditions can be deleted. That is, the model training system 150 can perform cleaning operations on source texts of any language category, for example, deleting information such as links and URLs in the source text.

[0041] In some embodiments, the model training system 150 can perform sentence segmentation on a given source text. For source texts of any language category among source texts of multiple language categories, the model training system 150 splits paragraphs in the source text of a certain language category into sentences.

[0042] In some embodiments, the model training system 150 can replace texts in a given source text that are not in a given language category with texts in a given language category. For source texts in any language category among source texts in multiple language categories, if the model training system 150 determines that there are texts in a source text of a certain language category that are not in the language category, the texts can be replaced with texts that belong to the language category. For example, for texts in which the source text belongs to the Chinese category, non-Chinese texts can be replaced with Chinese texts (for example, Arabic numerals can be converted to uppercase Chinese characters).

[0043] Further, after performing preprocessing operations on source texts of multiple language categories, the model training system 150 can add the processed source texts to the text sample set as text samples corresponding to a given language category. It is understandable that the model training system 150 can classify and store the processed source texts according to the language category.

[0044] In block 220, the model training system 150 generates speech sample sets for multiple language categories using a speech synthesis model based on each text sample in the text sample set. In some embodiments, the model training system 150 may input each text sample in the text sample set into the speech synthesis model 170 to generate speech sample sets for multiple language categories.

[0045] In some embodiments, the model training system 150 may generate speech sample sets for multiple language categories in the following manner. It is understandable that for each text sample in each text sample, multiple speech samples corresponding to the text sample may be generated in the following manner as part of the speech sample set. For ease of understanding, the following description will be made taking the first text sample in each text sample as an example.

[0046] The model training system 150 can obtain multiple speaker samples, each of which has corresponding speech features. It can be understood that the model training system 150 obtains the speech of multiple speakers as multiple speaker samples, and each speaker's speech has corresponding speech features. For example, the language category (for example, Chinese speech, English speech, etc.), style (for example, rigorous style, humorous style, etc.), timbre, etc. corresponding to the speech.

[0047] In some embodiments, the model training system 150 may select the multiple speaker samples from the speaker sample set. Each speaker sample in the speaker sample set corresponds to one of multiple language categories. In some embodiments, the speaker sample set may be implemented as a speaker embedding vector library. The speaker embedding vector may be a vector that maps complex, high-dimensional or sparse data (e.g., speech, etc.) to a low-dimensional, dense vector space through an embedding vector. That is, the speaker embedding vector library is used to store vectors corresponding to the speech of speakers of multiple language categories. In some embodiments, the speaker sample set may be implemented as a speaker voice library. The speaker voice library may be used to store the speech of speakers of multiple language categories. For example, audio of a speaker reading a text in a certain language. In some examples, the speaker sample set may be pre-created and include speaker samples of multiple language categories.

[0048] In some embodiments, the number of speaker samples corresponding to each of the multiple language categories is greater than a predetermined number. In some examples, each of the multiple language categories included in the speaker sample set contains at least a predetermined number of speaker samples. For example, each language contains at least 100 different speaker styles. In the present disclosure, each language can contain any suitable number of different speaker styles, and the above example numbers are merely exemplary, and the present disclosure does not limit this.

[0049] In some embodiments, the model training system 150 may select multiple speaker samples from the speaker sample set in the following manner. The model training system 150 assigns a weight to the speaker sample according to the language category of the first text sample and the language category corresponding to the speaker sample in the speaker sample set. Further, the model training system 150 selects multiple speaker samples according to the weight of each speaker sample in the speaker sample set.

[0050] In some examples, the model training system 150 can determine the weight assigned to the current speaker sample according to the language category of the text sample 314 to be transcribed and the language category corresponding to the speaker sample in the speaker sample set. For example, the weights for different language categories can be predetermined weight values. Then, the model training system 150 selects multiple speaker samples according to the weight of each speaker sample. In other embodiments, the model training system 150 can also randomly select multiple speaker samples from the speaker sample set.

[0051] In some embodiments, the model training system 150 may generate a plurality of speech samples corresponding to the plurality of speaker samples respectively using a speech synthesis model according to the corresponding speech information of the plurality of speaker samples and the first text sample as part of the speech sample set. Figure 3 In the example architecture 300 shown, the model training system 150 can use the speech synthesis model 170 to generate multiple speech samples corresponding to multiple speaker samples respectively according to the corresponding speech information 313 and text samples 314 of the multiple speaker samples as part of the speech sample set 113.

[0052] It is understandable that when the speaker sample set is implemented as a speaker voice library, the voice information 313 may include the voices of multiple speakers. In this embodiment, the speech synthesis model 170 may convert the voices of the speakers in the speaker voice library into speaker embedding vectors for generating a speech sample set. When the speaker sample set is implemented as a speaker embedding vector library, the voice information 313 may include the voice features of multiple speakers (i.e., the vectors corresponding to the speakers' voices). In some examples, the model training system 150 may also use reasoning tools (e.g., Deep Speed-Inference) to assist the speech synthesis model 170 in generating speaker sample sets. Thereby improving the efficiency of the speech synthesis model 170 in generating speaker sample sets.

[0053] Therefore, by synthesizing speech sample sets based on texts in multiple language categories, it is possible to ensure that the quality of the synthesized speech sample sets meets the training requirements of the speech processing model.

[0054] In some embodiments, the model training system 150 can generate speech sample sets for multiple language categories in the following manner. It can be understood that for each text sample in each text sample, the following manner is used to generate multiple speech samples corresponding to the text sample as part of the speech sample set. In particular, in the scenario where the speech processing model is configured to perform a translation task, the accuracy of the translation can be improved at least based on the multiple speech samples. For ease of understanding, the second text sample of the first language category in each text sample will be described as an example.

[0055] The model training system 150 may obtain a third text sample of the second language category corresponding to the second text sample. Further, the model training system 150 may generate at least one speech sample as part of the speech sample set based on one of the second text sample or the third text sample and the speech information of the speaker using a speech synthesis model.

[0056] In some examples, the model training system 150 obtains a B text sample belonging to the B speech category corresponding to the A text sample for the A text sample belonging to the A language category. It can be understood that the B text sample is the translation of the A text sample. Then, the model training system 150 randomly selects a text sample (for example, the B text sample) from the B text sample and the A text sample and inputs it into the speech synthesis model 170 to obtain at least one speech sample.

[0057] Accordingly, in the scenario of training the speech processing model 130, the model training system 150 can determine the training sample as part of the training sample set based on at least one speech sample and another text sample in the second text sample or the third text sample. Continuing with the above example, the model training system 150 determines the training sample as part of the training sample set for training the speech processing model 130 based on the A text sample selected from the B text sample and the A text sample, and at least one speech sample generated by the speech synthesis model 170.

[0058] In summary, synthesizing speech sample sets from texts in multiple language categories can ensure that the quality of the synthesized speech sample sets meets the training requirements of the speech processing model, thereby improving the recognition accuracy of the speech processing model on low-resource languages ​​and the performance on translation tasks.

[0059] Continuing with process 200, at box 230, the model training system 150 generates a training sample set for the speech processing model based on the text sample set and the speech sample set. In some embodiments, the model training system 150 can generate training samples for the speech processing model based on the text sample set 112 and the speech sample set 113 generated via the speech synthesis model 170.

[0060] In some embodiments, the model training system 150 may perform post-processing operations on the speech samples in the speech sample set to obtain a processed speech sample set. In some embodiments, the post-processing operations may indicate simulated environmental factors. In some embodiments, the post-processing operations may include adding environmental noise. Figure 3In the example architecture 300 shown, the model training system 150 can add environmental noise 315 to the speech samples in the speech sample set 113. In some examples, the model training system 150 can use the merging unit 311 to merge the environmental noise 315 with the speech samples in the speech sample set 113 to obtain a processed speech sample as part of the processed speech sample set 312.

[0061] In some embodiments, post-processing operations may include adjusting the speech waveform according to the impulse response. In some embodiments, the impulse response simulates the effect of the speaker's environment on the sound. In some examples, the impulse response may indicate the time domain (time-amplitude) response characteristics of a system under test when an impulse excitation signal is input. For example, the reverberation, echo, etc. effects on the sound in a room environment. Figure 3 In the example architecture 300 shown, the model training system 150 can use the merging unit 311 to adjust the speech waveform of the speech sample in the speech sample set 113 according to the impulse response 316, thereby obtaining the processed speech sample as part of the processed speech sample set 312.

[0062] Then, the model training system 150 determines a training sample set based on the text sample set and the processed speech sample set. In some examples, the model training system 150 determines a training sample set for training the speech processing model 130 based on the text sample set 314 and the processed speech sample set 312.

[0063] In summary, the present disclosure synthesizes speech sample sets based on texts of multiple language categories, which can ensure that the quality of the synthesized speech sample sets meets the training requirements of the speech processing model and reduce the need and cost of manual annotation. Furthermore, based on the synthesized speech sample set, the quality of text of at least one language category generated by the speech processing model can be improved. Furthermore, by ensuring that the number of samples between the multiple language categories included in the text sample set meets a predetermined condition, the recognition accuracy of the speech processing model on low-resource languages ​​and the performance on translation tasks can be improved. This also narrows the performance gap between speech processing models in different languages.

[0064] Example devices and equipment

[0065] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes.

[0066] Figure 4 A schematic structural block diagram of an apparatus 400 for speech processing according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in a model training system 150. Each module / component in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0067] like Figure 4 As shown, the device 400 includes a text sample set generation module 410, which is configured to generate text sample sets for multiple language categories based on source texts of multiple language categories, wherein the number of samples of the text sample set in multiple language categories is distributed to meet a predetermined condition. The device 400 also includes a speech sample set generation module 420, which is configured to generate speech sample sets for multiple language categories using a speech synthesis model based on each text sample in the text sample set. The device 400 also includes a training sample set generation module 430, which is configured to generate a training sample set for a speech processing model based on the text sample set and the speech sample set.

[0068] In some embodiments, the speech sample set generation module 420 is further configured to obtain multiple speaker samples for the first text sample in each text sample, each speaker sample in the multiple speaker samples having corresponding speech characteristics; and based on the corresponding speech information of the multiple speaker samples and the first text sample, generate multiple speech samples corresponding to the multiple speaker samples respectively using a speech synthesis model as part of the speech sample set.

[0069] In some embodiments, the apparatus 400 further includes a speaker sample acquisition module configured to select a plurality of speaker samples from a speaker sample set, wherein each speaker sample in the speaker sample set corresponds to one of a plurality of language categories.

[0070] In some embodiments, the number of speaker samples corresponding to each of the plurality of language categories is greater than a predetermined number.

[0071] In some embodiments, the speaker sample acquisition module is further configured to assign a weight to the speaker sample based on the language category of the first text sample and the language category corresponding to the speaker sample in the speaker sample set; and select multiple speaker samples based on the weights of each speaker sample in the speaker sample set.

[0072] In some embodiments, the speech sample set generation module 420 is further configured to obtain, for each text sample that is a second text sample of the first language category, a third text sample of the second language category corresponding to the second text sample; and based on the second text sample or one of the third text samples and the speaker's speech information, generate at least one speech sample as part of the speech sample set using a speech synthesis model, and wherein generating a training sample set for the speech processing model includes: determining the training sample as part of the training sample set based on at least one speech sample and another text sample of the second text sample or the third text sample.

[0073] In some embodiments, the speech sample set generation module 420 is further configured to perform post-processing operations on the speech samples in the speech sample set to obtain a processed speech sample set, wherein the post-processing operations are used to simulate environmental factors; and determine a training sample set based on the text sample set and the processed speech sample set.

[0074] In some embodiments, the post-processing operation includes adding environmental noise, or adjusting the speech waveform according to an impulse response, which simulates the effect of the speaker's environment on the sound.

[0075] In some embodiments, the text sample set generation module 410 is further configured to perform a preprocessing operation on a given source text of a given language category among multiple language categories to generate a processed source text; and add the processed source text to the text sample set as a text sample corresponding to the given language category.

[0076] In some embodiments, the preprocessing operation includes at least one of the following: deleting words in a given source text whose semantic information does not meet a predetermined condition, segmenting the given source text into sentences, or replacing words in a given source text that are not in a given language category with words in a given language category.

[0077] The units and / or modules included in the device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 400 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0078] It should be understood that one or more steps in the above method can be performed by a suitable electronic device or a combination of electronic devices. Such an electronic device or a combination of electronic devices may include, for example, Figure 1 Model training system 150 in.

[0079] Figure 5 1 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 5 The electronic device 500 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to implement Figure 1Model training system 150.

[0080] like Figure 5 As shown, the electronic device 500 is in the form of a general electronic device. The components of the electronic device 500 may include, but are not limited to, one or more processors or processing units 510, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit 5105 may be an actual or virtual processor and is capable of performing various processes according to the program stored in the memory 520. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0081] The electronic device 500 typically includes a plurality of computer storage media. Such media may be any available media accessible to the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 500.

[0082] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 520 may include a computer program product 525 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0083] The communication unit 540 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0084] The input device 550 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 560 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 500 may also communicate with one or more external devices (not shown) through the communication unit 540 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 500, or communicate with any device that allows the electronic device 500 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0085] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0086] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0087] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0088] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0089] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0090] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for speech processing, Based on source texts of multiple language categories, generating text sample sets for the multiple language categories, wherein the distribution of the number of samples of the text sample sets in the multiple language categories meets a predetermined condition; Based on each text sample in the text sample set, using a speech synthesis model, generating speech sample sets for the multiple language categories; as well as Based on the text sample set and the speech sample set, a training sample set for a speech processing model is generated.

2. The method of claim 1, wherein generating speech sample sets for the plurality of language categories comprises: For the first text sample among the text samples, Acquire a plurality of speaker samples, each of the plurality of speaker samples having corresponding speech characteristics; as well as Based on the corresponding speech information of the multiple speaker samples and the first text sample, the speech synthesis model is used to generate multiple speech samples corresponding to the multiple speaker samples respectively as part of the speech sample set.

3. The method according to claim 2, wherein obtaining the plurality of speaker samples comprises: The plurality of speaker samples are selected from a speaker sample set, each speaker sample in the speaker sample set corresponding to one of the plurality of language categories. 4 . The method according to claim 3 , wherein the number of speaker samples corresponding to each of the plurality of language categories is greater than a predetermined number.

5. The method according to claim 2, wherein selecting the plurality of speaker samples from a speaker sample set comprises: assigning a weight to the speaker sample based on the language category of the first text sample and the language category corresponding to the speaker sample in the speaker sample set; as well as The multiple speaker samples are selected based on the weight of each speaker sample in the speaker sample set.

6. The method of claim 1, wherein generating speech sample sets for the plurality of language categories comprises: For a second text sample of the first language category among the text samples, obtaining a third text sample of the second language category corresponding to the second text sample; as well as Based on one of the second text sample or the third text sample and the speech information of the speaker, at least one speech sample is generated as a part of the speech sample set by using the speech synthesis model, and The training sample set generated for the speech processing model includes: Based on the at least one speech sample and the other text sample of the second text sample or the third text sample, a training sample is determined as a part of the training sample set.

7. The method according to claim 1, wherein generating a training sample set for a speech processing model comprises: Performing a post-processing operation on the speech samples in the speech sample set to obtain a processed speech sample set, wherein the post-processing operation is used to simulate environmental factors; as well as The training sample set is determined based on the text sample set and the processed speech sample set.

8. The method of claim 7, wherein the post-processing operation comprises: Add ambient noise, or The speech waveform is adjusted according to an impulse response that simulates the effect of the speaker's environment on the sound.

9. The method according to claim 1, wherein generating text sample sets for the plurality of language categories: performing a preprocessing operation on a given source text of a given language category among the plurality of language categories to generate a processed source text; and The processed source text is added to the text sample set as a text sample corresponding to the given language category.

10. The method according to claim 9, wherein the pre-processing operation comprises at least one of the following: Deleting the words in the given source text whose semantic information does not meet the predetermined conditions, Sentence the given source text, or The characters in the given source text that are not in the given language category are replaced with characters in the given language category.

11. A device for speech processing, comprising: A text sample set generation module is configured to generate text sample sets for multiple language categories based on source texts of multiple language categories, wherein the distribution of the number of samples of the text sample sets in the multiple language categories meets a predetermined condition; A speech sample set generation module is configured to generate speech sample sets for the multiple language categories based on each text sample in the text sample set and using a speech synthesis model; as well as The training sample set generation module is configured to generate a training sample set for a speech processing model based on the text sample set and the speech sample set.

12. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit.

13. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images