Model training method and device

Abstract

This specification provides a model training method and device, wherein the model training method includes: obtaining phoneme sample data and speech sample data containing at least two speech categories; performing initial training on a generator and a discriminator based on the speech sample data to obtain an intermediate generator and an intermediate discriminator; performing deep training on the intermediate generator, the intermediate discriminator, and an initial phoneme classification prediction network based on the phoneme sample data; and obtaining a phoneme classification prediction network and a target generator that meet the training stop conditions based on the training results.

Description

Technical Field

[0001] This specification relates to the field of machine learning technology, and in particular to model training methods and devices. Background Art

[0002] With the development of internet technology, speech recognition has become a widely used technology in various fields. Many existing oral assessment methods are based on a cascade of acoustic models and regressors. An accurate and robust acoustic model is crucial for effective oral assessment. Children and adults, as distinct speakers, have different pronunciation methods and effects. Given the significant differences in their pronunciation characteristics, accurate oral assessment for these different groups requires significant computing resources. Therefore, an effective solution is urgently needed to address this issue. Summary of the Invention

[0003] In view of this, the embodiments of this specification provide a model training method. This specification also relates to a model training device, a speech processing method, a speech processing device, a computing device, and a computer-readable storage medium to address the technical deficiencies in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a model training method is provided, including:

[0005] Acquire phoneme sample data and speech sample data containing at least two speech categories;

[0006] Performing initial training on the generator and the discriminator according to the speech sample data to obtain an intermediate generator and an intermediate discriminator;

[0007] Performing deep training on the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the phoneme sample data;

[0008] According to the training results, a phoneme classification prediction network and a target generator that meet the training stop conditions are obtained.

[0009] Optionally, the initial training process of the discriminator is as follows:

[0010] Adjusting the parameter state of the generator to a fixed state, and inputting the speech sample data into the generator in the fixed state for processing to obtain a first speech sample feature;

[0011] The discriminator is initially trained based on the first speech sample feature to obtain the intermediate discriminator that meets the initial training stop condition.

[0012] Optionally, the initial training process of the generator is as follows:

[0013] Adjusting the parameter state of the discriminator to a fixed state, and inputting the speech sample data into the generator for processing to obtain a second speech sample feature;

[0014] Inputting the second speech sample feature into a fixed-state discriminator for processing to obtain a domain classification result;

[0015] The generator is parameterized based on the domain classification result and the speech sample data, and the intermediate generator that meets the initial training stop condition is obtained according to the parameter adjustment result.

[0016] Optionally, the performing deep training on the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the phoneme sample data includes:

[0017] Parsing the phoneme sample data to obtain labeled sample data containing phoneme labels and unlabeled sample data not containing phoneme labels;

[0018] Performing deep training on the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the labeled sample data, and

[0019] The intermediate discriminator and the intermediate generator are deeply trained based on the unlabeled sample data.

[0020] Optionally, the performing deep training on the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the labeled sample data includes:

[0021] Adjusting the parameter state of the intermediate generator to a fixed state, and performing a first depth training on the initial phoneme classification prediction network and the intermediate discriminator based on the labeled sample data;

[0022] The parameter state of the intermediate discriminator is adjusted to a fixed state, and the initial phoneme classification prediction network and the intermediate generator are subjected to a second depth training based on the labeled sample data.

[0023] Optionally, the performing deep training on the intermediate discriminator and the intermediate generator based on the unlabeled sample data includes:

[0024] Adjusting the parameter states of the intermediate generator and the initial phoneme classification prediction network to fixed states, and performing a third depth training on the intermediate discriminator based on the unlabeled sample data;

[0025] The parameter states of the intermediate discriminator and the initial phoneme classification prediction network are adjusted to fixed states, and the intermediate generator is subjected to fourth depth training based on the unlabeled sample data.

[0026] Optionally, obtaining a phoneme classification prediction network and a target generator that meet a training stop condition according to the training results includes:

[0027] When it is determined according to the training result that the initial classification prediction network and the intermediate generator meet the preset number of iterations, the phoneme classification prediction network and the target generator are obtained; or

[0028] When it is determined according to the training results that the loss values of the initial classification prediction network and the intermediate generator reach a loss value threshold, the phoneme classification prediction network and the target generator are obtained.

[0029] According to a second aspect of the embodiments of this specification, a model training device is provided, comprising:

[0030] A data acquisition module is configured to acquire phoneme sample data and speech sample data containing at least two speech categories;

[0031] An initial training module is configured to perform initial training on the generator and the discriminator according to the speech sample data to obtain an intermediate generator and an intermediate discriminator;

[0032] A deep training module, configured to perform deep training on the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the phoneme sample data;

[0033] The training stop module is configured to obtain a phoneme classification prediction network and a target generator that meet the training stop conditions according to the training results.

[0034] According to a third aspect of the embodiments of this specification, a speech processing method is provided, including:

[0035] Obtaining a speech to be processed and a reference text corresponding to the speech to be processed;

[0036] Inputting the speech to be processed into the target generator in the above method for processing to obtain speech sample features;

[0037] Inputting the speech sample features into the phoneme classification prediction network in the above method for processing to obtain a phoneme recognition result;

[0038] A pronunciation detection result corresponding to each phoneme in the speech to be processed is determined based on the reference text and the phoneme recognition result.

[0039] Optionally, determining a pronunciation detection result corresponding to each phoneme in the to-be-processed speech based on the reference text and the phoneme recognition result includes:

[0040] Aligning the reference text and the phoneme recognition result to obtain an alignment matrix;

[0041] Creating an output probability matrix corresponding to each phoneme in the speech to be processed according to the alignment matrix;

[0042] The output probability matrix corresponding to each phoneme is input into the multi-layer perceptron network for processing, and the pronunciation detection result corresponding to each phoneme is obtained according to the processing result.

[0043] Optionally, inputting the to-be-processed speech into a target generator for processing to obtain speech sample features includes:

[0044] Inputting the speech to be processed into the target generator, performing dimensionality-raising processing on the speech to be processed through the first fully connected layer in the target generator to obtain initial speech sample features;

[0045] The initial speech sample features are subjected to dimensionality reduction processing by the second fully connected layer in the target generator to obtain the speech sample features and output the target generator.

[0046] According to a fourth aspect of the embodiments of this specification, there is provided a speech processing device, including:

[0047] An acquisition module is configured to acquire a speech to be processed and a reference text corresponding to the speech to be processed;

[0048] A first processing module is configured to input the speech to be processed into the target generator in the above method for processing to obtain speech sample features;

[0049] A second processing module is configured to input the speech sample features into the phoneme classification prediction network in the above method for processing to obtain a phoneme recognition result;

[0050] The determination module is configured to determine a pronunciation detection result corresponding to each phoneme in the speech to be processed based on the reference text and the phoneme recognition result.

[0051] According to a fifth aspect of the embodiments of this specification, there is provided a computing device, including:

[0052] memory and processor;

[0053] The memory is used to store computer-executable instructions, and the processor is used to implement the steps of the model training method or speech processing method when executing the computer-executable instructions.

[0054] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which implement the model training method or speech processing steps when executed by a processor.

[0055] The model training method provided in this specification, after obtaining phoneme sample data and speech sample data containing at least two speech categories, can perform initial training on the generator and discriminator based on the speech sample data to obtain an intermediate generator and an intermediate discriminator in order to achieve accurate phoneme detection for different user groups. This can improve the capabilities of the generator and the discriminator through adversarial training. Then, based on the phoneme sample data, the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network are deeply trained to combine the three to obtain a target generator and a phoneme classification prediction network that can detect phonemes. Combining adversarial training and multi-task learning to train the network can not only improve the prediction ability of the network, but also reduce the impact of the pronunciation characteristics of different user groups, thereby achieving accurate phoneme detection and improving detection accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of a model training method provided in one embodiment of this specification;

[0057] Figure 2 This is a schematic diagram of the structure of a model training device provided in one embodiment of this specification;

[0058] Figure 3 This is a flow chart of a speech processing method provided in one embodiment of this specification;

[0059] Figure 4 is a schematic diagram of a target generator in a speech processing method provided in an embodiment of this specification;

[0060] Figure 5 is a schematic diagram of a phoneme classification prediction network in a speech processing method provided in one embodiment of this specification;

[0061] Figure 6 This is a processing flow chart of a speech processing method applied to a spoken language evaluation scenario provided in an embodiment of this specification;

[0062] Figure 7 This is a structural diagram of a speech processing device provided in one embodiment of this specification;

[0063] Figure 8 This is a structural block diagram of a computing device provided in one embodiment of this specification. DETAILED DESCRIPTION

[0064] In the following description, many specific details are set forth in order to provide a thorough understanding of this specification. However, this specification can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited by the specific implementations disclosed below.

[0065] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a", "the", and "said" used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0066] It should be understood that although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0067] First, the noun terms related to one or more embodiments of this specification are explained.

[0068] Robustness: The ability of a system to survive in abnormal and dangerous situations. It refers to the characteristic of a control system to maintain certain other performances under a certain (structure, size) parameter perturbation. For example, whether computer software can avoid crashing or freezing under input errors, disk failures, network overloads or intentional attacks is the robustness of the software.

[0069] Phoneme: The smallest speech unit divided according to the natural attributes of speech. Analyzed according to the pronunciation actions in a syllable, one action constitutes one phoneme. Phonemes are divided into two major categories: vowels and consonants. For example, the Chinese syllable "ā" has only one phoneme, "ài" has two phonemes, "dài" has three phonemes, etc.

[0070] In this specification, a model training method is provided. This specification also relates to a model training device, a speech processing method, a speech processing device, a computing device, and a computer-readable storage medium, which will be described in detail one by one in the following embodiments.

[0071] In practical applications, speech recognition training for young children is difficult to label and obtain. At the same time, there are significant differences between the pronunciation characteristics of young children and those of adults. When using acoustic models obtained from a large amount of adult speech training for young children's pronunciation evaluation, there will be certain deviations in the effect. Moreover, when the network is relatively complex, fine-tuning on the target domain data is prone to overfitting. Therefore, an effective solution is urgently needed to solve the above problems.

[0072] The model training method provided in this specification, after obtaining phoneme sample data and speech sample data containing at least two speech categories, can perform initial training on the generator and discriminator based on the speech sample data to obtain an intermediate generator and an intermediate discriminator in order to achieve accurate phoneme detection for different user groups. This can improve the capabilities of the generator and the discriminator through adversarial training. Then, based on the phoneme sample data, the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network are deeply trained to combine the three to obtain a target generator and a phoneme classification prediction network that can detect phonemes. Combining adversarial training and multi-task learning to train the network can not only improve the prediction ability of the network, but also reduce the impact of the pronunciation characteristics of different user groups, thereby achieving accurate phoneme detection and improving detection accuracy and efficiency.

[0073] Figure 1 A flow chart of a model training method according to an embodiment of this specification is shown, which specifically includes the following steps:

[0074] Step S102: Acquire phoneme sample data and speech sample data containing at least two speech categories.

[0075] Specifically, phoneme sample data refers to the data used to train the discriminator, generator, and initial phoneme classification prediction network during the deep training phase, and includes both sample data with and without phoneme labels. Accordingly, speech categories refer to categories corresponding to the pronunciation characteristics of different phonetic bodies, such as the pronunciation of young children and adults belonging to two different speech categories. Accordingly, speech sample data refers to sample data consisting of speech corresponding to the two speech categories, used for initial training of the generator and discriminator. It should be noted that initial training and deep training represent two different stages of training.

[0076] In practice, due to a lack of data, most phoneme classification prediction networks are trained using sample data corresponding to adults. However, the network trained with this data has low accuracy in recognizing children's pronunciation and cannot provide reasonable predictions. Therefore, in order to improve the versatility of the phoneme classification prediction network, a discriminator and a generator are combined to train the phoneme classification prediction network. During the training process, the discriminator and the generator are used to improve the prediction ability. In addition, the main function of the generator is to remove information with attribute characteristics from the speech produced by the speaker, so that the processed results will not be affected by the attribute characteristics of the speaker itself. In other words, whether the speech is produced by a child or an adult, after being processed by the generator, it will be mapped to the same distribution space and will not be affected by the different pronunciation methods of children or adults. This allows the subsequent network to ignore the attribute characteristics of the speaker when making predictions.

[0077] The main function of the discriminator is to improve the predictive ability of the generator during the training phase. Specifically, after the generator converts the input speech into speech features, the discriminator will identify the speech features to determine whether the speech features come from a child or an adult. If the discriminator accurately identifies it, it means that the predictive ability of the generator needs to be improved, and the generator can continue to be trained; if the discriminator cannot accurately identify it, it means that the generator can eliminate the pronunciation features of the pronunciation body after processing the speech, and the generator can be stored and reused.

[0078] The main function of the phoneme classification prediction network is to identify the probability map corresponding to the phonemes in the speech, which can be used for subsequent analysis of the pronunciation accuracy of each phoneme.

[0079] Step S104: Initially train the generator and the discriminator according to the speech sample data to obtain an intermediate generator and an intermediate discriminator.

[0080] Specifically, after obtaining the phoneme sample data and speech sample data as mentioned above, further, in order to be able to train a generator with strong conversion ability and a discriminator that can accurately detect the generator, the initial phoneme classification prediction network can be fixed first, and then the discriminator and generator can be initially trained based on speech sample data containing at least two speech categories, that is, the discriminator and generator are trained in one stage, so that the generator can learn the ability to eliminate the pronunciation features of the pronunciation body, and the discriminator can learn the ability to distinguish the speech categories of the pronunciation body, so as to be used for joint training of the initial phoneme classification prediction network in the deep training stage.

[0081] Among them, the generator specifically refers to a feature adaptive network, which is used to eliminate the pronunciation features of the pronunciation body during speech processing; the discriminator specifically refers to a domain classification network, which is used to confront the generator during model training and to improve the generator's prediction ability; correspondingly, the intermediate generator specifically refers to the generator obtained after one stage of training with speech sample data, and the intermediate discriminator specifically refers to the discriminator obtained after one stage of training with speech sample data.

[0082] Furthermore, when training the discriminator, since the input of the discriminator is the output of the generator, in order to improve the domain classification ability of the discriminator, that is, to improve the discriminator's ability to accurately distinguish the speech categories of the speech, the generator can be fixed first, and then the discriminator can be trained. In this embodiment, the specific implementation method is as follows:

[0083] The parameter state of the generator is adjusted to a fixed state, and the speech sample data is input into the generator in the fixed state for processing to obtain a first speech sample feature; the discriminator is initially trained based on the first speech sample feature to obtain the intermediate discriminator that meets the initial training stop condition.

[0084] Specifically, a fixed generator refers to a generator whose adjustable parameters are fixed and only used to convert speech data into the input of the discriminator, so as to assist the discriminator in learning the ability to distinguish the speech categories of the pronunciation body; that is, the parameter state of the generator is adjusted to a fixed state, which is the adjustable parameter of the fixed generator; correspondingly, the first speech sample feature specifically refers to the vector expression after the speech sample data is converted by the generator in a fixed state.

[0085] Based on this, during the initial training process, after obtaining speech sample data containing at least two speech categories, the parameter state of the generator can be adjusted to a fixed state first, and then the speech sample data can be input into the generator with fixed parameters for processing to obtain the first speech sample feature that meets the discriminator input, and finally the discriminator is trained in one stage based on the first speech sample feature to obtain an intermediate discriminator that meets the initial training stop condition.

[0086] In summary, by using a fixed generator to initially train the discriminator, we can avoid the impact of the generator's conversion ability, thereby ensuring that the discriminator can fully learn the ability to distinguish pronunciation features, and subsequently assist in improving the generator's prediction ability.

[0087] Furthermore, when training the generator, since the discriminator can distinguish the speech categories of pronunciation features, the generator can be adjusted according to the recognition results of the discriminator, so that the generator can improve the effect of eliminating pronunciation features. In this embodiment, the specific implementation method is as follows:

[0088] The parameter state of the discriminator is adjusted to a fixed state, and the speech sample data is input into the generator for processing to obtain a second speech sample feature; the second speech sample feature is input into the fixed-state discriminator for processing to obtain a domain classification result; the generator is adjusted based on the domain classification result and the speech sample data, and the intermediate generator that meets the initial training stop condition is obtained according to the parameter adjustment result.

[0089] Specifically, the fixed discriminator specifically refers to fixing the adjustable parameters of the discriminator and only using it to identify the speech category to which the speech sample features belong, and to assist the generator in learning the ability to eliminate the pronunciation features of the pronunciation body; that is, the parameter state of the discriminator is adjusted to a fixed state, which is the adjustable parameter of the fixed discriminator; accordingly, the second speech sample feature specifically refers to the vector expression after the speech sample data is converted by the generator with unfixed parameters; accordingly, the domain classification result specifically refers to the classification result obtained after the fixed-state discriminator processes the second speech sample feature, which is used to characterize the speech category corresponding to the speech sample data.

[0090] Based on this, after obtaining speech sample data containing at least two speech categories, the adjustable parameters of the discriminator can be fixed first, and then the speech sample data can be input into the generator for processing to obtain the converted second speech sample features; the second speech sample features can then be input into the fixed-state discriminator for processing to obtain the domain classification results corresponding to the speech sample data; finally, the generator can be adjusted based on the domain classification results and the speech sample data, so that the generator can learn the pronunciation features of the eliminated pronunciation body to obtain an intermediate generator that meets the initial training stop conditions.

[0091] It should be noted that the initial training stop condition can be the number of iterations, that is, the training is stopped after a set number of iterations to obtain an intermediate discriminator and an intermediate generator.

[0092] In addition, when training the discriminator and generator in the initial training phase, the prediction capabilities of the discriminator and generator can be obtained by monitoring the loss value. The training goal is as follows:

[0093]

[0094] Among them, D corresponds to the discriminator, and the output value represents the probability of the adult voice category; G represents the generator; X a represents the input features corresponding to adults, X c Indicates the input features corresponding to the child; I a Indicates that when the input feature corresponds to an adult, it is 1, otherwise it is 0; I cIndicates that when the input feature corresponds to a child, it is 1, otherwise it is 0; when the generator is fixed, the discriminator is trained so that L s1 The larger the discriminator is, the higher the accuracy of speech classification is. When the discriminator is fixed, the generator is trained so that L s1 The smaller it is, the more accurate the generator is in eliminating pronunciation features.

[0095] That is to say, when training the discriminator in the initial training stage, the generator is fixed, and then the loss value L is improved by continuously iteratively training the discriminator. s1 , so that the intermediate discriminator that meets the initial training stop condition is trained; at the same time, the discriminator is fixed, and then the loss value L is reduced by continuously iteratively training the generator s1 , so that an intermediate generator that meets the initial training stop condition is trained.

[0096] For example, when it is necessary to train a phoneme classification prediction network that can detect phonemes without considering the pronunciation characteristics of the speaker, the initial phoneme classification prediction network can be pre-trained with adult speech transcription data so that the network can learn phoneme detection capabilities. Based on this, in order to enable the trained phoneme classification prediction network to accurately recognize phonemes in children's speech, the generator (feature adaptive network) and discriminator (domain classification network) can be trained through adversarial training; further, first obtain speech sample data corresponding to adults and children, then fix the generator, and input the speech sample data into the fixed generator for processing to obtain the first speech sample feature; then train the discriminator based on the first speech sample feature, and obtain an intermediate discriminator that meets the first-stage training stop condition through the training results.

[0097] Furthermore, the discriminator is fixed, and the speech sample data is input into the generator for processing to obtain the second speech sample feature, and then the second speech sample feature is processed by the fixed-state discriminator to obtain the domain classification result; finally, the generator parameters are adjusted based on the domain classification result and the speech sample feature, and an intermediate generator that meets the first-stage training stop condition can be obtained to realize the subsequent joint training of the phoneme classification prediction network, so that the trained network can accurately recognize and detect the phonemes of adults and children, thereby improving the versatility of the network.

[0098] In summary, by using a fixed discriminator to train the generator, and a fixed generator to train the discriminator for initial training, not only can the discriminator and generator learn speech recognition capabilities, but they can also check and balance each other, thereby improving the recognition accuracy of the two and facilitating the subsequent training of a more versatile phoneme classification prediction network.

[0099] Step S106: performing deep training on the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the phoneme sample data.

[0100] Specifically, after completing the initial training of the discriminator and generator as mentioned above, the intermediate discriminator obtained at this time already has a certain ability to recognize speech categories, and the intermediate generator already has a certain ability to convert speech (the ability to eliminate pronunciation features). In order to be able to train a more universal phoneme classification prediction network, the intermediate generator, the intermediate discriminator and the initial phoneme classification prediction network can be deeply trained based on the phoneme sample data at this time, so as to fine-tune the intermediate generator, the intermediate discriminator and the initial phoneme classification prediction network through a smaller learning rate until the target generator and phoneme classification prediction network that meet the usage requirements are adjusted.

[0101] Among them, phoneme sample data specifically refers to sample data composed of sample data containing phoneme labels and sample data not containing phoneme labels, which is used to train each network in the deep training stage; correspondingly, deep training specifically refers to the stage of secondary training of the intermediate generator, the intermediate discriminator and the initial phoneme classification prediction through phoneme sample data.

[0102] Furthermore, when performing deep training on the intermediate generator, intermediate discriminator, and initial phoneme classification prediction, considering that the phoneme sample data contains unlabeled sample data and labeled sample data, and the phoneme classification prediction network requires supervised training, unlabeled sample data cannot be used for training. Therefore, multi-task learning can be used to achieve joint learning. In this embodiment, the specific implementation method is as follows:

[0103] Step S1062 , parsing the phoneme sample data to obtain labeled sample data containing phoneme labels and unlabeled sample data not containing phoneme labels.

[0104] Specifically, after obtaining the phoneme sample data, in order to enable supervised training of the initial phoneme classification prediction network and unsupervised training of the discriminator and generator, the phoneme sample data can be parsed to obtain labeled sample data containing phoneme labels and unlabeled sample data without phoneme labels. Labeled sample data specifically refers to sample data containing phoneme labels and speech transcription data, and unlabeled sample data specifically refers to sample data without phoneme labels and speech transcription data. The labeled sample data can be used to perform supervised training on the initial phoneme classification prediction network, and the unlabeled sample data can be used to perform unsupervised training on the intermediate discriminator and intermediate generator.

[0105] Step S1064: performing deep training on the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the labeled sample data.

[0106] Specifically, after obtaining the labeled sample data containing phoneme labels as mentioned above, in order to train a phoneme classification prediction network with higher phoneme detection accuracy, the intermediate generator, intermediate discriminator and initial phoneme classification prediction network can be deeply trained using the labeled sample data.

[0107] Based on this, in the process of deep training the intermediate generator, the intermediate discriminator and the initial phoneme classification prediction network, considering that the discriminator is a network used to improve the conversion ability of the generator, it is also necessary to fully train the discriminator to balance the generator, thereby improving the phoneme detection ability of the phoneme classification prediction network. In this embodiment, the specific implementation method is as follows:

[0108] The parameter state of the intermediate generator is adjusted to a fixed state, and the initial phoneme classification prediction network and the intermediate discriminator are subjected to a first depth training based on the labeled sample data; the parameter state of the intermediate discriminator is adjusted to a fixed state, and the initial phoneme classification prediction network and the intermediate generator are subjected to a second depth training based on the labeled sample data.

[0109] Specifically, when it is necessary to use labeled sample data to train the intermediate generator, the intermediate discriminator and the initial phoneme classification prediction network, the parameter state of the intermediate generator can be adjusted to a fixed state first, and then the initial phoneme classification prediction network and the intermediate discriminator can be subjected to a first deep training based on the labeled sample data. During the first deep training process, the initial phoneme classification prediction network is actually fine-tuned based on the labeled sample data and its corresponding phoneme detection results, and the intermediate discriminator is updated at the same time; secondly, the parameter state of the intermediate discriminator is also adjusted to a fixed state, and then the initial phoneme classification prediction network and the intermediate generator are subjected to a second deep training based on the labeled sample data. During the second deep training process, the initial phoneme classification prediction network is actually fine-tuned based on the labeled sample data and its corresponding phoneme detection results, and the intermediate generator is updated at the same time.

[0110] When the fixed intermediate generator performs the first deep training on the initial phoneme classification prediction network and the intermediate discriminator, the labeled sample data is actually input into the fixed intermediate generator for processing to obtain the first labeled sample feature; then the first labeled sample feature is input into the initial phoneme classification prediction network to obtain the labeled phoneme detection result, and is input into the intermediate discriminator for processing to obtain the domain classification result; finally, the initial phoneme classification prediction network is fine-tuned based on the labeled phoneme detection result and the labeled sample data, and the intermediate discriminator is updated based on the domain classification result and the labeled sample data. Through continuous iteration, the first deep training of the two is achieved.

[0111] When the fixed intermediate discriminator performs the second deep training on the initial phoneme classification prediction network and the intermediate generator, the labeled sample data is actually input into the generator for processing to obtain the second labeled sample features; then the second labeled sample features are input into the initial phoneme classification prediction network to obtain the labeled phoneme detection results, and are input into the fixed intermediate discriminator for processing to obtain the domain classification results; at this time, the domain classification results can be used to analyze the ability of the fixed generator to eliminate pronunciation features, and finally the initial phoneme classification prediction network is fine-tuned based on the labeled phoneme detection results and the labeled sample data, and the intermediate generator is updated based on the second labeled sample features and labeled sample data corresponding to the domain classification results, so as to achieve the second deep training of the two.

[0112] In summary, by using labeled sample data for two rounds of deep training, not only can the network's prediction ability be improved, but also the network's versatility can be improved, thereby further ensuring the accuracy of phoneme detection in the speech processing stage.

[0113] Step S1066: Perform deep training on the intermediate discriminator and the intermediate generator based on the unlabeled sample data.

[0114] Specifically, after obtaining the unlabeled sample data that does not contain phoneme labels as mentioned above, in order to be able to train a phoneme classification prediction network with higher phoneme detection accuracy, the intermediate discriminator and the intermediate generator can be deeply trained using the unlabeled sample data. That is to say, the capabilities of the two can be further improved through unsupervised training to assist in improving the phoneme detection capability of the phoneme classification prediction network.

[0115] Based on this, in the process of deep training the intermediate generator and the intermediate discriminator, considering that unlabeled sample data can only be used to train unsupervised networks, the intermediate discriminator and the intermediate generator can be deeply trained based on the unlabeled sample data. In this embodiment, the specific implementation method is as follows:

[0116] The parameter states of the intermediate generator and the initial phoneme classification prediction network are adjusted to fixed states, and the intermediate discriminator is subjected to third depth training based on the unlabeled sample data; the parameter states of the intermediate discriminator and the initial phoneme classification prediction network are adjusted to fixed states, and the intermediate generator is subjected to fourth depth training based on the unlabeled sample data.

[0117] Specifically, when it is necessary to use unlabeled sample data to train the intermediate generator and the intermediate discriminator, the parameter states of the intermediate generator and the initial phoneme classification network can be adjusted to a fixed state first, and then the intermediate discriminator can be subjected to a third depth training based on the unlabeled sample data. During the third depth training process, the intermediate discriminator is actually updated based on the unlabeled sample data; secondly, the parameter states of the intermediate discriminator and the initial phoneme classification prediction network are adjusted to a fixed state, and then the intermediate generator is subjected to a fourth depth training based on the unlabeled sample data. During the fourth depth training process, the intermediate generator is actually updated based on the unlabeled sample data.

[0118] When the intermediate generator and the initial phoneme classification prediction network are fixed to perform the third deep training on the intermediate discriminator, the unlabeled sample data is actually input into the fixed-state generator for processing to obtain the first unlabeled sample features; then the first unlabeled sample features are input into the intermediate discriminator for processing to obtain the domain classification results; finally, the intermediate discriminator is updated based on the domain classification results and the unlabeled sample data, and the third deep training of the intermediate discriminator is achieved through continuous iteration.

[0119] When the intermediate generator is trained for the fourth depth with a fixed intermediate discriminator and an initial phoneme classification prediction network, the unlabeled sample data is actually input into the generator for processing to obtain the second unlabeled sample features; then the second unlabeled sample features are input into the fixed intermediate discriminator for processing to obtain the domain classification results; finally, the intermediate generator is updated based on the second unlabeled sample features and unlabeled sample data corresponding to the domain classification results to achieve the fourth depth training of the intermediate generator.

[0120] Continuing with the previous example, after a phase of training the generator and discriminator, labeled sample data containing phoneme labels and unlabeled sample data without phoneme labels are now available. The intermediate generator can be fixed first, and then the initial phoneme classification prediction network can be fine-tuned based on the labeled sample data, and the intermediate discriminator can be updated. Simultaneously, the intermediate discriminator can be updated again using the unlabeled sample data. The intermediate discriminator can then be fixed again, and the initial phoneme classification prediction network can be fine-tuned based on the labeled sample data, and the intermediate generator can be updated. Simultaneously, the intermediate generator can be updated again using the unlabeled sample data, completing the network training through continuous iteration.

[0121] In summary, by adopting multi-antibody training and multi-task learning methods for network training, not only can the network prediction ability be improved, but also the versatility of the trained network can be improved, so that accurate phoneme detection can be achieved in different speech recognition scenarios.

[0122] Step S108: Obtain a phoneme classification prediction network and a target generator that meet the training stop condition according to the training results.

[0123] Specifically, after deep training of the intermediate generator, intermediate discriminator, and initial phoneme classification prediction network, it is now possible to detect whether the three meet the training stop condition. This allows the phoneme classification prediction network and target generator to achieve these conditions based on the training results. During the speech processing phase, the target generator can convert the input speech into a vector unaffected by pronunciation characteristics, and then the phoneme classification prediction network can perform phoneme detection, effectively ensuring the accuracy of phoneme detection.

[0124] Furthermore, when training the initial phoneme classification prediction network and the intermediate generator, the training stop condition can be the number of iterations or the loss value calculation, that is, when it is determined according to the training results that the initial classification prediction network and the intermediate generator meet the preset number of iterations, the phoneme classification prediction network and the target generator are obtained; or, when it is determined according to the training results that the loss value of the initial classification prediction network and the intermediate generator reaches the loss value threshold, the phoneme classification prediction network and the target generator are obtained.

[0125] In the deep training stage, the capabilities of the initial phoneme classification prediction network and the intermediate generator can be understood through the loss value, so as to determine the phoneme classification prediction network and the target generator that meet the requirements according to the loss value. The training goal is as follows:

[0126]

[0127] Among them, N represents the total number of samples corresponding to adults and children, n represents the total number of samples corresponding to adults, and L ce Represents cross entropy loss, which is used to characterize phoneme classification loss and is used on labeled sample data of adults.

[0128] In practical applications, the training stop condition of deep training can be selected according to actual needs, and this embodiment does not impose any limitation here.

[0129] Continuing with the above example, after continuously training the intermediate generator, intermediate discriminator, and initial phoneme classification prediction network, when the training meets the number of iterations, the phoneme classification prediction network and target generator that meet the usage requirements can be selected. In the speech processing stage, the target generator can be used to convert the input speech into a vector that is not affected by pronunciation features, and then the phoneme classification prediction network can be used to perform phoneme detection, which can effectively ensure the accuracy of phoneme detection.

[0130] The model training method provided in this specification, after obtaining phoneme sample data and speech sample data containing at least two speech categories, can perform initial training on the generator and discriminator based on the speech sample data to obtain an intermediate generator and an intermediate discriminator in order to achieve accurate phoneme detection for different user groups. This can improve the capabilities of the generator and the discriminator through adversarial training. Then, based on the phoneme sample data, the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network are deeply trained to combine the three to obtain a target generator and a phoneme classification prediction network that can detect phonemes. Combining adversarial training and multi-task learning to train the network can not only improve the prediction ability of the network, but also reduce the impact of the pronunciation characteristics of different user groups, thereby achieving accurate phoneme detection and improving detection accuracy and efficiency.

[0131] Corresponding to the above method embodiment, this specification also provides a model training device embodiment, Figure 2 FIG1 shows a schematic diagram of the structure of a model training device provided in an embodiment of this specification. Figure 2 As shown, the device includes:

[0132] A data acquisition module 202 is configured to acquire phoneme sample data and speech sample data containing at least two speech categories;

[0133] An initial training module 204 is configured to perform initial training on the generator and the discriminator based on the speech sample data to obtain an intermediate generator and an intermediate discriminator;

[0134] A deep training module 206 is configured to perform deep training on the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the phoneme sample data;

[0135] The training stop module 208 is configured to obtain a phoneme classification prediction network and a target generator that meet the training stop condition according to the training results.

[0136] In an optional embodiment, the initial training process of the discriminator is as follows:

[0137] The parameter state of the generator is adjusted to a fixed state, and the speech sample data is input into the generator in the fixed state for processing to obtain a first speech sample feature; the discriminator is initially trained based on the first speech sample feature to obtain the intermediate discriminator that meets the initial training stop condition.

[0138] In an optional embodiment, the initial training process of the generator is as follows:

[0139] The parameter state of the discriminator is adjusted to a fixed state, and the speech sample data is input into the generator for processing to obtain a second speech sample feature; the second speech sample feature is input into the fixed-state discriminator for processing to obtain a domain classification result; the generator is adjusted based on the domain classification result and the speech sample data, and the intermediate generator that meets the initial training stop condition is obtained according to the parameter adjustment result.

[0140] In an optional embodiment, the depth training module 206 is further configured to:

[0141] The phoneme sample data is parsed to obtain labeled sample data containing phoneme labels and unlabeled sample data not containing phoneme labels; the intermediate generator, the intermediate discriminator and the initial phoneme classification prediction network are deeply trained based on the labeled sample data, and the intermediate discriminator and the intermediate generator are deeply trained based on the unlabeled sample data.

[0142] In an optional embodiment, the depth training module 206 is further configured to:

[0143] The parameter state of the intermediate generator is adjusted to a fixed state, and the initial phoneme classification prediction network and the intermediate discriminator are subjected to a first depth training based on the labeled sample data; the parameter state of the intermediate discriminator is adjusted to a fixed state, and the initial phoneme classification prediction network and the intermediate generator are subjected to a second depth training based on the labeled sample data.

[0144] In an optional embodiment, the depth training module 206 is further configured to:

[0145] The parameter states of the intermediate generator and the initial phoneme classification prediction network are adjusted to fixed states, and the intermediate discriminator is subjected to third depth training based on the unlabeled sample data; the parameter states of the intermediate discriminator and the initial phoneme classification prediction network are adjusted to fixed states, and the intermediate generator is subjected to fourth depth training based on the unlabeled sample data.

[0146] In an optional embodiment, the training stop module 208 is further configured to:

[0147] When it is determined according to the training results that the initial classification prediction network and the intermediate generator meet the preset number of iterations, the phoneme classification prediction network and the target generator are obtained; or, when it is determined according to the training results that the loss values of the initial classification prediction network and the intermediate generator reach the loss value threshold, the phoneme classification prediction network and the target generator are obtained.

[0148] The model training device provided in this specification, after obtaining phoneme sample data and speech sample data containing at least two speech categories, can perform initial training on the generator and discriminator based on the speech sample data to obtain an intermediate generator and an intermediate discriminator in order to achieve accurate phoneme detection for different user groups. This can improve the capabilities of the generator and the discriminator through adversarial training. This can then be used to perform deep training on the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the phoneme sample data to combine the three to obtain a target generator and a phoneme classification prediction network that can detect phonemes. Combining adversarial training and multi-task learning to train the network can not only improve the prediction ability of the network, but also reduce the impact of the pronunciation characteristics of different user groups, thereby achieving accurate phoneme detection and improving detection accuracy and efficiency.

[0149] The above is a schematic scheme of a model training device of this embodiment. It should be noted that the technical scheme of the model training device and the technical scheme of the above-mentioned model training method are of the same concept. For details not described in detail in the technical scheme of the model training device, please refer to the description of the technical scheme of the above-mentioned model training method.

[0150] This embodiment also provides an embodiment of a speech processing method, which is as follows:

[0151] Figure 3 A flow chart of a speech processing method according to an embodiment of the present invention is shown, which specifically includes the following steps:

[0152] Step S302: Acquire the speech to be processed and the reference text corresponding to the speech to be processed.

[0153] The speech processing method provided in this embodiment uses the phoneme classification prediction network and target generator obtained by training in the above embodiments when realizing the recognition of the processed speech. The training process of the phoneme classification prediction network and the target generator can be found in the corresponding description content in the above embodiments, and this embodiment will not be elaborated here.

[0154] Based on this, the speech to be processed specifically refers to the speech that needs to be phoneme detected, which is used to detect the user's oral ability; correspondingly, the reference text specifically refers to the text that the user refers to when reading when emitting the speech to be processed. That is to say, the user needs to read according to the content of the reference text in order to collect the speech to be processed through the terminal device and realize the evaluation of the user's oral ability.

[0155] Step S304: input the speech to be processed into the target generator in the model training method for processing to obtain speech sample features.

[0156] Specifically, after obtaining the speech to be processed and the reference text as described above, in order to avoid the influence of the pronunciation characteristics of the speaker, the speech to be processed can be input into the target generator for processing to obtain speech sample features characterized only from the audio dimension. In other words, regardless of whether the speaker of the speech to be processed is an adult or a child, the target generator can map the pronunciation characteristics corresponding to the adult or the child to the same distribution space, so that when the subsequent phoneme classification prediction network performs phoneme detection, it is not necessary to consider whether the speaker is an adult or a child, which has higher versatility.

[0157] Furthermore, when the target generator is used to process the speech to be processed, in order to eliminate the pronunciation features, a feature adaptive network including two fully connected layers can be selected as the target generator. In this embodiment, the specific implementation is as follows:

[0158] The speech to be processed is input into the target generator, and the speech to be processed is subjected to dimensionality increase processing through the first fully connected layer in the target generator to obtain initial speech sample features; the initial speech sample features are subjected to dimensionality reduction processing through the second fully connected layer in the target generator to obtain the speech sample features and output the target generator.

[0159] Specifically, after the speech to be processed is input into the target generator, the first fully connected layer in the target generator can be used to increase the dimension of the speech to be processed to obtain the initial speech sample features, and then the second fully connected layer in the target generator can be used to reduce the dimension of the initial speech sample features to obtain the speech sample features and output the target generator for subsequent phoneme detection.

[0160] For example, after obtaining the voice to be processed uploaded by the child user and the reference text "what" corresponding to the voice to be processed, refer to Figure 4As shown, the speech to be processed can be converted into 40-dimensional speech features to be processed, and then input into the target generator. The 40-dimensional speech features to be processed are processed by the first fully connected layer to increase the dimension of the speech to be processed to obtain 100-dimensional initial speech sample features. The 100-dimensional initial speech sample features are then processed by the second fully connected layer to reduce the dimension of the speech to obtain 40-dimensional speech sample features with the same input dimension, which are used for subsequent input into the phoneme classification prediction network to detect the pronunciation of the child user.

[0161] In summary, by using a two-layer fully connected target generator to process the processed speech, the accuracy of eliminating pronunciation features can be effectively improved, the impact of pronunciation features can be reduced, and the subsequent accurate detection of each phoneme in the processed speech can be achieved.

[0162] It should be noted that the dimensions after dimensionality increase and dimensionality reduction are the same to avoid introducing other factors that may affect the recognition accuracy.

[0163] Step S306: Input the speech sample features into the phoneme classification prediction network in the model training method for processing to obtain a phoneme recognition result.

[0164] Specifically, after obtaining the speech sample features through the target generator as mentioned above, the speech sample features can be further input into the phoneme classification prediction network to detect each phoneme in the processed speech through the phoneme classification prediction network to obtain the phoneme detection result; wherein, the phoneme detection result specifically refers to the characteristic expression of the number of pronunciation types corresponding to each phoneme in the processed speech, which is used for subsequent detection of whether the pronunciation of each phoneme is correct.

[0165] In practical applications, the phoneme classification prediction network can be implemented using the residual connection TDNN structure, see Figure 5 As shown in the schematic diagram, TDNN is connected through a residual network to achieve local full connection in each layer that can consider contextual features. At the same time, in order to solve the problem of learning difficulties, jump edge connections are added to the basis of TDNN, so that the learning task is changed to residual learning, which helps the phoneme classification prediction network converge.

[0166] When performing phoneme recognition, the output layer of the phoneme classification prediction network will output a feature expression with a dimension of (T, C), where T represents the number of frames and C represents the number of types after the three-phonemes are clustered. That is to say, one phoneme in the speech to be processed will correspond to multiple three-phonemes, and many three-phonemes have similarities. Therefore, the number of types of three-phonemes corresponding to each phoneme can be determined by clustering, and the phoneme recognition results corresponding to the speech to be processed can be constructed in turn, which is convenient for subsequent detection of whether the pronunciation of each phoneme is accurate.

[0167] Step S308 : determining a pronunciation detection result corresponding to each phoneme in the speech to be processed based on the reference text and the phoneme recognition result.

[0168] Specifically, after obtaining the phoneme recognition results, it indicates that each phoneme in the processed speech has been recognized and the corresponding expression of each phoneme has been determined. In order to detect the pronunciation accuracy of each phoneme, it can be verified based on the reference text to determine the pronunciation detection result corresponding to each phoneme in the processed speech. The pronunciation detection result specifically refers to the result of detecting whether each phoneme is correct.

[0169] Furthermore, when determining the pronunciation detection result corresponding to each phoneme, in order to ensure the detection accuracy, a forced alignment method can be adopted. In this embodiment, the specific implementation method is as follows:

[0170] The reference text and the phoneme recognition result are aligned to obtain an alignment matrix; an output probability matrix corresponding to each phoneme in the speech to be processed is created based on the alignment matrix; the output probability matrix corresponding to each phoneme is input into a multi-layer perceptron network for processing, and a pronunciation detection result corresponding to each phoneme is obtained based on the processing result.

[0171] Specifically, the alignment matrix refers to the matrix representation obtained by forcibly aligning the reference text and the phoneme recognition results; the output probability matrix refers to the forced alignment results corresponding to each phoneme. Based on this, after obtaining the phoneme recognition results, the reference text and the phoneme recognition results can be forcibly aligned. That is, the audio frame for aligning each phoneme is determined by the reference text, and each character unit in the reference text is positioned to the vector representation of each phoneme in the speech to be processed to obtain an alignment matrix. Then, based on the alignment matrix, the output probability matrix corresponding to each phoneme can be created. Finally, this is input into the multi-layer perceptron network to predict the pronunciation detection result of each phoneme, that is, to determine whether the user's pronunciation is correct.

[0172] Continuing with the above example, after obtaining the 40-dimensional speech sample features, they are input into the phoneme classification prediction network. Based on the processing results, a probability map corresponding to the speech to be processed is obtained. The probability map is then forced to align with the reference text "what" to obtain an alignment matrix. At this point, an output probability matrix corresponding to each phoneme can be created based on the alignment matrix, that is, the phoneme "w" corresponds to matrix P1, "h" corresponds to matrix P2, "a" corresponds to matrix P3, and "t" corresponds to matrix P4. Finally, each matrix is combined with the probability map and input into the multi-layer perceptron network for prediction. According to the prediction results, it is determined that the pronunciation of "t" is inaccurate.

[0173] In summary, by using forced alignment to prepare for phoneme pronunciation detection, the detection accuracy can be effectively guaranteed. Then, by combining the pre-trained multi-layer perceptron network for pronunciation detection, the detection efficiency can be effectively improved.

[0174] The speech processing method provided in this embodiment can convert the speech to be processed and its corresponding reference text through a target generator after obtaining the speech to be processed, so as to ensure that the expression of the converted speech sample features will not be affected by the pronunciation features of the pronunciation body. The speech sample features are then input into the phoneme classification prediction network for processing to obtain the phoneme recognition results corresponding to each phoneme; finally, the pronunciation detection of the phonemes is performed by forced alignment to obtain the pronunciation detection results of each phoneme, which effectively ensures the detection accuracy and the versatility of the phoneme classification prediction network.

[0175] The following combined Figure 6 , taking the application of the speech processing method provided in this specification in the oral evaluation scenario as an example, the speech processing method is further explained. Figure 6 The flowchart of a speech processing method for a spoken language evaluation scenario provided in an embodiment of this specification is shown, which specifically includes the following steps:

[0176] Step S602: Acquire the speech to be processed and its corresponding reading text uploaded by the user through the client.

[0177] Step S604: input the speech to be processed into the target generator, perform dimensionality-upgrading processing on the speech to be processed through the first fully connected layer in the target generator, and obtain the initial speech sample features.

[0178] Step S606: Perform dimensionality reduction processing on the initial speech sample features through the second fully connected layer in the target generator to obtain the speech sample features and output the target generator.

[0179] Step S608: Input the speech sample features into the phoneme classification prediction network for processing to obtain phoneme detection results.

[0180] Step S610 , aligning the reading text and the phoneme recognition result to obtain an alignment matrix.

[0181] Step S612: Create an output probability matrix corresponding to each phoneme in the speech to be processed according to the alignment matrix.

[0182] In step S614, the output probability matrix corresponding to each phoneme is input into the multi-layer perceptron network for processing, and the pronunciation detection result corresponding to each phoneme is obtained according to the processing result.

[0183] Step S616: creating reminder information based on the mispronounced phonemes in the pronunciation detection result and sending the reminder information to the client.

[0184] To sum up, after obtaining the speech to be processed and its corresponding reference text, it can be converted through the target generator to ensure that the expression of the converted speech sample features will not be affected by the pronunciation features of the pronunciation body. The speech sample features are then input into the phoneme classification prediction network for processing to obtain the phoneme recognition results corresponding to each phoneme; finally, the pronunciation detection of the phonemes is performed through forced alignment to obtain the pronunciation detection results of each phoneme, which effectively ensures the detection accuracy and the versatility of the phoneme classification prediction network.

[0185] The speech processing method provided in this embodiment can convert the speech to be processed and its corresponding reference text through a target generator after obtaining the speech to be processed, so as to ensure that the expression of the converted speech sample features will not be affected by the pronunciation features of the pronunciation body. The speech sample features are then input into the phoneme classification prediction network for processing to obtain the phoneme recognition results corresponding to each phoneme; finally, the pronunciation detection of the phonemes is performed by forced alignment to obtain the pronunciation detection results of each phoneme, which effectively ensures the detection accuracy and the versatility of the phoneme classification prediction network.

[0186] Corresponding to the above method embodiment, this specification also provides a speech processing device embodiment, Figure 7 FIG. 1 shows a schematic diagram of the structure of a speech processing device provided in an embodiment of this specification. Figure 7 As shown, the device includes:

[0187] An acquisition module 702 is configured to acquire a speech to be processed and a reference text corresponding to the speech to be processed;

[0188] The first processing module 704 is configured to input the speech to be processed into the target generator in the model training method for processing to obtain speech sample features;

[0189] The second processing module 706 is configured to input the speech sample features into the phoneme classification prediction network in the model training method for processing to obtain a phoneme recognition result;

[0190] The determination module 708 is configured to determine a pronunciation detection result corresponding to each phoneme in the speech to be processed based on the reference text and the phoneme recognition result.

[0191] In an optional embodiment, the determining module 708 is further configured to:

[0192] The reference text and the phoneme recognition result are aligned to obtain an alignment matrix; an output probability matrix corresponding to each phoneme in the speech to be processed is created based on the alignment matrix; the output probability matrix corresponding to each phoneme is input into a multi-layer perceptron network for processing, and a pronunciation detection result corresponding to each phoneme is obtained based on the processing result.

[0193] In an optional embodiment, the first processing module 704 is further configured to:

[0194] The speech to be processed is input into the target generator, and the speech to be processed is subjected to dimensionality increase processing through the first fully connected layer in the target generator to obtain initial speech sample features; the initial speech sample features are subjected to dimensionality reduction processing through the second fully connected layer in the target generator to obtain the speech sample features and output the target generator.

[0195] The speech processing device provided in this embodiment can convert the speech to be processed and its corresponding reference text through a target generator after obtaining it, so as to ensure that the expression of the converted speech sample features will not be affected by the pronunciation features of the pronunciation body. The speech sample features are then input into the phoneme classification prediction network for processing to obtain the phoneme recognition results corresponding to each phoneme; finally, the pronunciation detection of the phonemes is performed by forced alignment to obtain the pronunciation detection results of each phoneme, which effectively ensures the detection accuracy and the versatility of the phoneme classification prediction network.

[0196] The above is a schematic diagram of a speech processing device according to this embodiment. It should be noted that the technical solution of the speech processing device and the technical solution of the speech processing method described above are based on the same concept. For details not described in detail in the technical solution of the speech processing device, please refer to the description of the technical solution of the speech processing method described above.

[0197] Figure 8 8 shows a block diagram of a computing device 800 according to an embodiment of the present disclosure. Components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0198] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface card (NIC)), whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0199] In one embodiment of the present specification, the above components of the computing device 800 and Figure 8 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 8 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0200] The computing device 800 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 800 can also be a mobile or stationary server.

[0201] Among them, the processor 820 is used to implement the steps of the model training method or the speech processing method when executing computer-executable instructions.

[0202] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the model training method or speech processing method described above are based on the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the model training method or speech processing method described above.

[0203] An embodiment of the present specification also provides a computer-readable storage medium storing computer instructions, which are used for a model training method or a speech processing method when executed by a processor.

[0204] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium is identical to the technical solution of the model training method or speech processing method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the model training method or speech processing method described above.

[0205] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0206] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0207] It should be noted that for the aforementioned method embodiments, for ease of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that this specification is not limited to the order of the actions described, because according to this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this specification.

[0208] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0209] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of this specification, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A model training method, characterized in that: include: Acquire phoneme sample data and speech sample data containing at least two speech categories, wherein the speech categories specifically refer to categories corresponding to pronunciation features of different pronunciation organs; Performing initial training on the generator and the discriminator according to the speech sample data to obtain an intermediate generator and an intermediate discriminator; The intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network are deeply trained based on the phoneme sample data, wherein the deep training of the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the phoneme sample data includes: parsing the phoneme sample data to obtain labeled sample data containing phoneme labels and unlabeled sample data not containing phoneme labels; deeply training the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the labeled sample data, and deeply training the intermediate discriminator and the intermediate generator based on the unlabeled sample data; According to the training results, a phoneme classification prediction network and a target generator that meet the training stop conditions are obtained.

2. The model training method according to claim 1, characterized in that The initial training process of the discriminator is as follows: Adjusting the parameter state of the generator to a fixed state, and inputting the speech sample data into the generator in the fixed state for processing to obtain a first speech sample feature; The discriminator is initially trained based on the first speech sample feature to obtain the intermediate discriminator that meets the initial training stop condition.

3. The model training method according to claim 1, characterized in that The initial training process of the generator is as follows: Adjusting the parameter state of the discriminator to a fixed state, and inputting the speech sample data into the generator for processing to obtain a second speech sample feature; Inputting the second speech sample feature into a fixed-state discriminator for processing to obtain a domain classification result; The generator is parameterized based on the domain classification result and the speech sample data, and the intermediate generator that meets the initial training stop condition is obtained according to the parameter adjustment result.

4. The model training method according to claim 1, characterized in that The deep training of the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the labeled sample data includes: Adjusting the parameter state of the intermediate generator to a fixed state, and performing a first depth training on the initial phoneme classification prediction network and the intermediate discriminator based on the labeled sample data; The parameter state of the intermediate discriminator is adjusted to a fixed state, and the initial phoneme classification prediction network and the intermediate generator are subjected to a second depth training based on the labeled sample data.

5. The model training method according to claim 1, characterized in that The performing deep training on the intermediate discriminator and the intermediate generator based on the unlabeled sample data includes: Adjusting the parameter states of the intermediate generator and the initial phoneme classification prediction network to fixed states, and performing a third depth training on the intermediate discriminator based on the unlabeled sample data; The parameter states of the intermediate discriminator and the initial phoneme classification prediction network are adjusted to fixed states, and the intermediate generator is subjected to fourth depth training based on the unlabeled sample data.

6. The model training method according to claim 1, characterized in that The method of obtaining a phoneme classification prediction network and a target generator that meet the training stop condition according to the training results includes: When it is determined according to the training result that the initial phoneme classification prediction network and the intermediate generator meet the preset number of iterations, the phoneme classification prediction network and the target generator are obtained; or When it is determined according to the training results that the loss values of the initial phoneme classification prediction network and the intermediate generator reach a loss value threshold, the phoneme classification prediction network and the target generator are obtained.

7. A model training device, characterized in that: include: A data acquisition module is configured to acquire phoneme sample data and speech sample data containing at least two speech categories, wherein the speech categories specifically refer to categories corresponding to pronunciation features of different pronunciation organs; An initial training module is configured to perform initial training on the generator and the discriminator according to the speech sample data to obtain an intermediate generator and an intermediate discriminator; A deep training module is configured to perform deep training on the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the phoneme sample data, wherein the deep training on the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the phoneme sample data comprises: parsing the phoneme sample data to obtain labeled sample data containing phoneme labels and unlabeled sample data not containing phoneme labels; performing deep training on the intermediate generator, the intermediate discriminator, and the initial phoneme classification prediction network based on the labeled sample data, and performing deep training on the intermediate discriminator and the intermediate generator based on the unlabeled sample data; The training stop module is configured to obtain a phoneme classification prediction network and a target generator that meet the training stop conditions according to the training results.

8. A speech processing method, characterized in that: include: Obtaining a speech to be processed and a reference text corresponding to the speech to be processed; Inputting the speech to be processed into the target generator in the method according to any one of claims 1 to 6 for processing to obtain speech sample features; Inputting the speech sample features into the phoneme classification prediction network in the method according to any one of claims 1 to 6 for processing to obtain a phoneme recognition result; A pronunciation detection result corresponding to each phoneme in the speech to be processed is determined based on the reference text and the phoneme recognition result.

9. The speech processing method according to claim 8, characterized in that: The determining, based on the reference text and the phoneme recognition result, a pronunciation detection result corresponding to each phoneme in the speech to be processed, includes: Aligning the reference text and the phoneme recognition result to obtain an alignment matrix; Creating an output probability matrix corresponding to each phoneme in the speech to be processed according to the alignment matrix; The output probability matrix corresponding to each phoneme is input into the multi-layer perceptron network for processing, and the pronunciation detection result corresponding to each phoneme is obtained according to the processing result.

10. The speech processing method according to claim 8, characterized in that: The step of inputting the speech to be processed into a target generator for processing to obtain speech sample features includes: Inputting the speech to be processed into the target generator, performing dimensionality-raising processing on the speech to be processed through the first fully connected layer in the target generator to obtain initial speech sample features; The initial speech sample features are subjected to dimensionality reduction processing by the second fully connected layer in the target generator to obtain the speech sample features and output the target generator.

11. A speech processing device, characterized in that: include: An acquisition module is configured to acquire a speech to be processed and a reference text corresponding to the speech to be processed; A first processing module is configured to input the speech to be processed into the target generator in the method according to any one of claims 1 to 6 for processing to obtain speech sample features; A second processing module is configured to input the speech sample features into the phoneme classification prediction network in the method according to any one of claims 1 to 6 for processing to obtain a phoneme recognition result; The determination module is configured to determine a pronunciation detection result corresponding to each phoneme in the speech to be processed based on the reference text and the phoneme recognition result.

12. A computing device, characterized in that The method comprises a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the method according to any one of claims 1 to 6 or 8 to 10.

13. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 or 8 to 10 are implemented.

Images