Training method, recognition method and device for Hausa language speech model

Abstract

The present invention relates to the field of speech recognition technology, and in particular to a training method, a recognition method and a device for a Hausa speech model, wherein the training method comprises obtaining an English audio sample and a first text sample corresponding thereto, and a Hausa audio sample and a second text sample corresponding thereto; training the Hausa speech model using the English audio sample and the first text sample corresponding thereto, determining the initial parameters of a convolution unit in the Hausa speech model to obtain an initial Hausa speech model, wherein the Hausa speech model comprises a convolution unit and a fully connected layer connected to the convolution unit; training the initial Hausa speech model using the Hausa audio sample and the second text sample corresponding thereto, adjusting the initial parameters and the parameters of the fully connected layer to determine a target Hausa speech model. The transfer learning method can avoid the problem of insufficient Hausa audio samples and ensure the accuracy of the trained Hausa speech model.

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition, and in particular to a training method, a recognition method and a device for a Hausa language speech model. Background Art

[0002] Speech recognition is the process of converting human voice signals into text. It is one of the important technologies in the field of artificial intelligence perception. With the development of deep learning technology, the accuracy and speed of speech recognition have made great progress. Nowadays, speech recognition technology has penetrated into many applications in our daily life, such as smart speakers, shopping guide robots and other products. However, most of the existing speech recognition research focuses only on the most widely used languages, such as English and Chinese, which has resulted in the application of speech recognition being limited to more developed regions and cities.

[0003] There are 6,809 languages ​​in the world, most of which are minority languages ​​with very few speakers. Research on speech recognition for minority languages ​​is a key bridge to bring people of different languages ​​closer together. Among them, Hausa belongs to the Chadic language family of the Afro-Asiatic language family and is one of the three most important languages ​​in Africa. For speech recognition models, training generally requires thousands of hours of audio, and Hausa, as a minority language, has less sample data that can be collected. Due to the lack of sample data, the accuracy of the trained speech recognition model is low, which in turn leads to a low accuracy rate of speech recognition. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide a training method, a recognition method and a device for a Hausa language speech model to solve the problem of low accuracy of speech recognition.

[0005] According to a first aspect, an embodiment of the present invention provides a method for training a Hausa language speech model, comprising:

[0006] Obtaining an English audio sample and a first text sample corresponding thereto, and a Hausa audio sample and a second text sample corresponding thereto;

[0007] Training a Hausa speech model using the English audio sample and its corresponding first text sample, determining initial parameters of a convolution unit in the Hausa speech model to obtain an initial Hausa speech model, wherein the Hausa speech model includes the convolution unit and a fully connected layer connected to the convolution unit;

[0008] The initial Hausa speech model is trained using the Hausa audio sample and its corresponding second text sample, and the initial parameters and the parameters of the fully connected layer are adjusted to determine a target Hausa speech model.

[0009] The training method of the Hausa speech model provided by the embodiment of the present invention, since the characters of Hausa and English differ by only 4 letters, when training the Hausa speech model, the Hausa speech model is first trained with English audio samples to obtain the initial parameters of the convolution unit, and then the initial Hausa speech model can be obtained; the initial Hausa speech model is then trained again with the Hausa audio samples, and the parameters of the convolution unit and the fully connected layer are fine-tuned. That is, the transfer learning method can not only avoid the problem of insufficient Hausa audio samples, but also ensure the accuracy of the trained Hausa speech model.

[0010] In combination with the first aspect, in a first implementation of the first aspect, the using the English audio sample and the corresponding first text sample to train the Hausa speech model to determine the initial parameters of the convolutional unit and the fully connected layer in the Hausa speech model includes:

[0011] Inputting the English audio sample into a Hausa language speech model, and extracting features from the English audio sample using the convolution unit to obtain audio features;

[0012] Using the fully connected layer to classify the audio features, and determine a first predicted text corresponding to the English audio sample;

[0013] Based on the first predicted text and the error of the first text sample, the parameters of the convolution unit are updated to determine the initial parameters of the convolution unit.

[0014] The training method of the Hausa speech model provided by the embodiment of the present invention includes a convolution unit and a fully connected layer connected to the convolution unit, and the obtained Hausa speech model can also be understood as a fully convolutional network. Since the fully convolutional network can simultaneously observe the historical information and future information of the audio sequence, no additional features are required to record, so the parameters of the network model can also be designed to be relatively small, the computing resources required to be consumed can also be greatly reduced, and it can be run on a mobile terminal.

[0015] In combination with the first embodiment of the first aspect, in the second embodiment of the first aspect, the convolution unit includes at least one convolution sub-unit, and each of the convolution sub-units in the convolution unit is connected in a residual manner; wherein the convolution sub-unit includes a connected depthwise separable convolution layer, a normalization layer, and an activation function layer.

[0016] The training method of the Hausa speech model provided by the embodiment of the present invention adopts a depthwise separable convolution layer in each convolution subunit to perform feature extraction, which greatly reduces the amount of calculation and improves the efficiency of training.

[0017] In combination with the first aspect, in a third implementation of the first aspect, the using the Hausa audio sample and its corresponding second text sample to train the initial Hausa speech model, and adjusting the initial parameters and the parameters of the fully connected layer to determine the target Hausa speech model includes:

[0018] Using the initial parameters of the convolution unit and the preset parameters of the fully connected layer, an initial Hausa language speech model is obtained;

[0019] Inputting the Hausa audio sample into the initial Hausa speech model to obtain a second predicted text;

[0020] Based on the second predicted text and the error of the second text sample, the initial parameters and the parameters of the fully connected layer are updated to determine the target Hausa speech model.

[0021] The training method of the Hausa speech model provided by the embodiment of the present invention determines the initial parameters of the convolution unit after training the Hausa speech model with English audio samples; and then trains the initial parameters and the parameters of the language-related fully connected layer in the Hausa speech model with the Hausa audio file, thereby ensuring the accuracy of the Hausa speech model obtained by training.

[0022] In combination with the first aspect, or the first to third embodiments of the first aspect, in a fourth embodiment of the first aspect, the target Hausa language speech model further includes a language model connected to the fully connected layer, and the training method further includes:

[0023] Fixing the adjusted initial parameters and the parameters of the fully connected layer, and initializing the parameters of the language model to obtain a preset Hausa language speech model;

[0024] Inputting the Hausa audio sample into the preset Hausa speech model to obtain a third predicted text;

[0025] Based on the third predicted text and the error of the second text sample, the parameters of the language model are adjusted to determine the target Hausa language speech model.

[0026] The training method of the Hausa speech model provided by the embodiment of the present invention further connects a language model after the fully connected layer to further optimize the predicted text output by the fully connected layer, thereby improving the accuracy of the Hausa speech model.

[0027] In combination with the fourth implementation manner of the first aspect, in the fifth implementation manner of the first aspect, inputting the Hausa language audio sample into the preset Hausa language speech model to obtain the third predicted text includes:

[0028] Using the language units in the language model to perform character probability statistics on the predicted text output by the fully connected layer;

[0029] Decoding is performed based on the result of the character probability statistics to obtain the third predicted text.

[0030] The training method of the Hausa speech model provided in the embodiment of the present invention uses a language model to optimize the decoding process of the prediction probability sequence, because Hausa text resources are easier to obtain than audio resources, thereby further improving the prediction result of the final speech recognition.

[0031] According to a second aspect, an embodiment of the present invention further provides a method for recognizing Hausa speech, comprising:

[0032] Acquire a target Hausa language speech model, wherein the target Hausa language speech model is trained according to the training method for a Hausa language speech model described in the first aspect of the present invention or in any one of the embodiments of the first aspect;

[0033] The target Hausa audio is input into the target Hausa speech model to determine the target Hausa text corresponding to the target Hausa audio.

[0034] The Hausa speech recognition method provided by the embodiment of the present invention, since the characters of Hausa and English differ by only 4 letters, when training the Hausa speech model, the Hausa speech model is first trained with English audio samples to obtain the initial parameters of the convolution unit, and then the initial Hausa speech model can be obtained; the initial Hausa speech model is then trained again with the Hausa audio samples, and the parameters of the convolution unit and the fully connected layer are fine-tuned. That is, the transfer learning method can not only avoid the problem of insufficient Hausa audio samples, but also ensure the accuracy of the trained Hausa speech model. On the basis of the accuracy of the Hausa speech model, the accuracy of the recognition result can be guaranteed.

[0035] According to a third aspect, an embodiment of the present invention further provides a training device for a Hausa language speech model, comprising:

[0036] A first acquisition module, used for acquiring an English audio sample and a first text sample corresponding thereto, and a Hausa audio sample and a second text sample corresponding thereto;

[0037] A first training module is used to train a Hausa speech model using the English audio sample and its corresponding first text sample, and determine initial parameters of a convolution unit in the Hausa speech model to obtain an initial Hausa speech model, wherein the Hausa speech model includes a convolution unit and a fully connected layer connected to the convolution unit;

[0038] The second training module is used to train the initial Hausa speech model using the Hausa audio sample and its corresponding second text sample, and adjust the initial parameters and the parameters of the fully connected layer to determine a target Hausa speech model.

[0039] The training device for the Hausa speech model provided by the embodiment of the present invention, since the characters of Hausa and English differ by only 4 letters, when training the Hausa speech model, the Hausa speech model is first trained with English audio samples to obtain the initial parameters of the convolution unit, and then the initial Hausa speech model can be obtained; the initial Hausa speech model is then trained again with the Hausa audio samples, and the parameters of the convolution unit and the fully connected layer are fine-tuned. That is, the transfer learning method can not only avoid the problem of insufficient Hausa audio samples, but also ensure the accuracy of the trained Hausa speech model.

[0040] According to a fourth aspect, an embodiment of the present invention further provides a Hausa speech recognition device, comprising:

[0041] A second acquisition module is used to acquire a target Hausa language speech model, wherein the target Hausa language speech model is trained according to the training method for the Hausa language speech model described in the first aspect of the present invention or any one of the embodiments of the first aspect;

[0042] The recognition module is used to input the target Hausa audio into the target Hausa speech model to determine the target Hausa text corresponding to the target Hausa audio.

[0043] The Hausa speech recognition device provided by the embodiment of the present invention can ensure the accuracy of the recognition result based on the accuracy of the target Hausa speech model.

[0044] According to a fifth aspect, an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the method for training a Hausa speech model described in the first aspect or any one embodiment of the first aspect, or the method for recognizing Hausa speech described in the second aspect.

[0045] According to a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, wherein the computer instructions are used to enable the computer to execute the training method of the Hausa speech model described in the first aspect or any one of the embodiments of the first aspect, or the Hausa speech recognition method described in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 is a flow chart of a method for training a Hausa language speech model according to an embodiment of the present invention;

[0048] Figure 2 is a flow chart of a method for training a Hausa language speech model according to an embodiment of the present invention;

[0049] Figure 3 is a flow chart of a method for training a Hausa language speech model according to an embodiment of the present invention;

[0050] Figure 4 is a processing architecture diagram of a method for training a Hausa language speech model according to an embodiment of the present invention;

[0051] Figure 5 is a flow chart of a method for recognizing Hausa speech according to an embodiment of the present invention;

[0052] Figure 6 is a structural block diagram of a training device for a Hausa language speech model according to an embodiment of the present invention;

[0053] Figure 7 is a structural block diagram of a Hausa speech recognition device according to an embodiment of the present invention;

[0054] Figure 8 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0056] According to an embodiment of the present invention, an embodiment of a method for training a Hausa speech model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0057] In this embodiment, a method for training a Hausa language speech model is provided, which can be used in electronic devices such as computers, mobile terminals, etc. Figure 1 is a flow chart of a method for training a Hausa language speech model according to an embodiment of the present invention, Figure 1 As shown, the process includes the following steps:

[0058] S11, obtaining an English audio sample and its corresponding first text sample, and a Hausa audio sample and its corresponding second text sample.

[0059] The official orthography of Hausa uses the Latin alphabet called the Boka alphabet, which contains 30 letters and is case-sensitive. Among them, 26 letters are consistent with English letters, including Four unique letters. Therefore, it can be used in the training process of the Hausa speech model, and can be combined with the English audio sample. The English audio sample is in one-to-one correspondence with the first text sample, and the Hausa audio sample is also in one-to-one correspondence with the second text sample.

[0060] The electronic device may obtain the English audio sample and its corresponding first text sample, and the Hausa audio sample and its corresponding second text sample from the outside, or may store them in the electronic device, or the electronic device may first obtain the original audio file, and obtain the audio sample and its corresponding text sample by processing the original audio file and its corresponding original text file.

[0061] Take the example of an electronic device processing a Hausa audio sample and its corresponding second text sample. When processing the original text, the electronic device first traverses each character in the original text, removes characters that do not belong to the 30 letters, and removes redundant spaces and line breaks. For Arabic numerals or abbreviations in the text, they need to be converted into corresponding complete Hausa words. Finally, all characters in the text need to be converted to lowercase. In the Hausa speech model described in the embodiment of the present invention, the uppercase and lowercase letters are not distinguished.

[0062] Furthermore, when electronic devices process raw audio, in order to facilitate the training of the Hausa speech model, the length of the audio needs to be constrained. Among them, according to daily communication habits, the length of a sentence is about 10 seconds. For example, electronic devices can use the VAD tool in the WebRTC library to perform endpoint detection on all audio, divide the continuous audio into smaller audio files containing only one sentence, and then perform duration statistics on each audio, and remove audio that is less than 1 second and greater than 20 seconds. Secondly, there may be long silent segments in the audio, which brings redundant information. For example, the silence removal tool in the Librosa library can be used to remove silence from the audio, leaving only audio segments with human voices.

[0063] After the electronic device has processed the audio and the corresponding text, it needs to align the audio and text. The audio is the input signal, and the text is the supervision signal used to calculate the loss. Therefore, each audio file and text must be ensured to correspond one-to-one to ensure the normal training of the speech model. If there is at least one pronunciation or one word that does not correspond to the content of the audio and the content of the text, the audio and text need to be removed together. You also need to pay attention to the inspection These four special Hausa letters are written to avoid confusion with the English letters b, d, k, and y.

[0064] S12, training a Hausa speech model using the English audio sample and the first text sample corresponding thereto, and determining initial parameters of a convolutional unit in the Hausa speech model to obtain an initial Hausa speech model.

[0065] The Hausa language speech model includes a convolution unit and a fully connected layer connected to the convolution unit.

[0066] As mentioned above, Hausa and English have similarities, and the difference lies in the four special characters in Hausa. Based on this, the electronic device first uses the English audio sample and its corresponding first text sample to train the Hausa speech model, and determines the initial parameters of the convolution unit in the Hausa speech model. Since the convolution unit is used for feature extraction and does not involve the difference in language, the Hausa speech model can be directly trained with the English audio sample and its corresponding first text sample, and the parameters of the convolution unit obtained by training are used as the initial parameters of the convolution unit in the Hausa speech model.

[0067] The convolution unit may include multiple convolution layers, or may be combined with an activation function layer on the basis of a convolution layer, etc. There is no restriction on the specific structure of the convolution unit, and it can be set accordingly according to the actual situation, as long as it can achieve feature extraction.

[0068] The parameters of the initial Hausa speech model include initial parameters of the convolutional unit and preset parameters of the fully connected layer. Since the fully connected layer is used for character classification and is related to the language, the fully connected parameters obtained by training the English audio sample are not suitable for use as the initial parameters of the fully connected layer. In this embodiment, the parameters of the fully connected layer can be preset parameters set by random initialization, so that the preset parameters meet normal distribution or uniform distribution.

[0069] S13, using the Hausa audio sample and the corresponding second text sample to train the initial Hausa speech model, adjusting the initial parameters and the parameters of the fully connected layer to determine the target Hausa speech model.

[0070] The Hausa language speech model includes a convolution unit and a fully connected layer connected to the convolution unit, that is, the Hausa language speech model can be divided into a convolution unit and a fully connected layer. Among them, the fully connected layer is used to classify characters, so the determination of the fully connected layer parameters needs to rely on the Hausa language audio sample and its corresponding second text sample.

[0071] After the electronic device trains the Hausa speech model using the English audio sample and the corresponding first text sample to determine the initial parameters of the convolution unit, the electronic device obtains a model using the initial parameters of the convolution unit and the preset parameters of the fully connected layer, which is called the initial Hausa speech model.

[0072] On the basis of obtaining the initial Hausa speech model, the electronic device uses the Hausa audio sample and the corresponding second text sample to train the initial Hausa speech model, updates the parameters of the initial Hausa speech model (the parameters include the initial parameters of the convolution unit and the parameters of the fully connected layer), so as to finally determine the target Hausa speech model.

[0073] The training method of the Hausa speech model provided in this embodiment, since the characters of Hausa and English differ by only 4 letters, when training the Hausa speech model, the Hausa speech model is first trained with English audio samples to obtain the initial parameters of the convolution unit, and then the initial Hausa speech model can be obtained; the initial Hausa speech model is then trained again with the Hausa audio samples, and the parameters of the convolution unit and the fully connected layer are fine-tuned. That is, the transfer learning method can not only avoid the problem of insufficient Hausa audio samples, but also ensure the accuracy of the trained Hausa speech model.

[0074] In this embodiment, a method for training a Hausa language speech model is provided, which can be used in electronic devices such as computers, mobile terminals, etc. Figure 2 is a flow chart of a method for training a Hausa language speech model according to an embodiment of the present invention, Figure 2As shown, the process includes the following steps:

[0075] S21, obtaining an English audio sample and a first text sample corresponding thereto, and a Hausa audio sample and a second text sample corresponding thereto.

[0076] For details, please see Figure 1 S11 of the illustrated embodiment will not be described in detail here.

[0077] S22, training a Hausa speech model using the English audio sample and the first text sample corresponding thereto, and determining initial parameters of a convolutional unit in the Hausa speech model to obtain an initial Hausa speech model.

[0078] The Hausa language speech model includes a convolution unit and a fully connected layer connected to the convolution unit.

[0079] Specifically, the above S22 may include the following steps:

[0080] S221, inputting the English audio sample into the Hausa language speech model, and using the convolution unit to extract features of the English audio sample to obtain audio features.

[0081] The electronic device inputs the English audio sample into the Hausa speech model, and the convolution unit in the Hausa speech model extracts features of the English audio sample to obtain audio features.

[0082] In some optional implementations of this embodiment, the convolution unit includes at least one convolution sub-unit, and each of the convolution sub-units in the convolution unit is connected in a residual manner; wherein the convolution sub-unit includes a connected depth-separable convolution layer, a normalization layer, and an activation function layer.

[0083] Specifically, the convolution unit is built based on the depthwise separable convolution. For each convolution unit, it includes a depthwise separable convolution layer, a batch normalization layer, and a ReLU nonlinear activation function layer. Each convolution unit is stacked in a residual connection manner, and multiple stacked convolution sub-units constitute a convolution unit for feature extraction in the Hausa speech recognition model, completing the extraction of high-dimensional features containing complex information.

[0084] The depth-wise separable convolutional layer is used for feature extraction in each convolutional sub-unit, which greatly reduces the amount of calculation and improves the efficiency of training.

[0085] S222: Classify the audio features using a fully connected layer to determine a first predicted text corresponding to the English audio sample.

[0086] After the convolution unit extracts audio features from the input English audio sample, the extracted audio features are input into the fully connected layer, and the audio features are classified using the fully connected layer to output the first predicted text. That is, the electronic device inputs the high-dimensional features output by the convolution unit into a fully connected layer to perform character classification prediction, and outputs the probability of letters related to a predefined language. The first predicted text can be obtained by concatenating the letters with the highest probability from each predicted result.

[0087] S223: Based on the first predicted text and the error of the first text sample, update the parameters of the convolution unit to determine the initial parameters of the convolution unit.

[0088] After obtaining the first predicted text, the electronic device calculates the loss function using the first predicted text and the first text sample, and updates the parameters of the convolution unit based on the calculation result of the loss function. After continuous learning and adjustment, the initial parameters of the convolution unit in the Hausa speech model can be determined. For example, the CTC loss function can be used. Of course, other loss functions can also be used, and the embodiment of the present invention does not impose any restrictions on this.

[0089] The Hausa speech model obtained through the above training is based on character modeling and can get rid of the pronunciation dictionary and perform end-to-end training.

[0090] S23, training an initial Hausa speech model using the Hausa audio sample and its corresponding second text sample, adjusting initial parameters and parameters of the fully connected layer to determine a target Hausa speech model.

[0091] Specifically, the above S23 may include the following steps:

[0092] S231, obtaining an initial Hausa language speech model using initial parameters of the convolutional unit and preset parameters of the fully connected layer.

[0093] After the electronic device trains the Hausa speech model using the English audio sample, the parameters of the convolution unit obtained can be directly used as the initial parameters of the convolution unit in the Hausa speech model. For the Hausa speech model, in addition to the convolution unit, it also includes a fully connected layer. Here, the parameters of the fully connected layer can be initialized, that is, the parameters of the fully connected layer are set to preset parameters, and the parameters of the convolution unit obtained by the above S22 training are used as the initial parameters of the convolution unit in the Hausa speech model. After these two parameters are set, the initial Hausa speech model can be obtained.

[0094] S232: Input the Hausa language audio sample into the initial Hausa language speech model to obtain a second predicted text.

[0095] Since the fully connected layer connected to the convolutional unit is language-dependent, the setting of the fully connected layer parameters needs to be trained with the Hausa audio file and its corresponding second text sample. Specifically, after determining the initial Hausa speech model, the electronic device inputs the Hausa audio sample into the initial Hausa speech model, and the fully connected layer outputs the second predicted text.

[0096] S233: Based on the second predicted text and the error of the second text sample, the initial parameters and the parameters of the fully connected layer are updated to determine the target Hausa speech model.

[0097] The electronic device calculates the loss function of the second predicted text and the second text sample, and updates the initial parameters of the convolution unit and the parameters of the full connection in the initial Hausa speech model. After multiple training and parameter updates, the parameters of the convolution unit and the full connection layer can be determined. At this point, the parameters of the convolution unit and the full connection layer are determined, and on this basis, the target Hausa speech model can be determined.

[0098] The training method of the Hausa speech model provided in this embodiment includes a convolution unit and a fully connected layer connected to the convolution unit. The obtained Hausa speech model can also be understood as a fully convolutional network. Since the fully convolutional network can simultaneously observe the historical information and future information of the audio sequence, no additional features are required to record, so the parameters of the network model can also be designed to be relatively small, the computing resources required to be consumed can also be greatly reduced, and it can be run on a mobile terminal.

[0099] In this embodiment, a method for training a Hausa language speech model is provided, which can be used in electronic devices such as computers, mobile terminals, etc. Figure 3 is a flow chart of a method for training a Hausa language speech model according to an embodiment of the present invention, Figure 3 As shown, the process includes the following steps:

[0100] S31, obtaining an English audio sample and its corresponding first text sample, and a Hausa audio sample and its corresponding second text sample.

[0101] For details, please see Figure 2 S21 of the illustrated embodiment will not be described in detail here.

[0102] S32, training a Hausa language speech model using the English audio sample and its corresponding first text sample, determining initial parameters of a convolutional unit in the Hausa language speech model, so as to obtain an initial Hausa language speech model.

[0103] The Hausa language speech model includes a convolution unit and a fully connected layer connected to the convolution unit.

[0104] For details, please see Figure 2 S22 of the illustrated embodiment will not be described in detail here.

[0105] S33, using the Hausa audio sample and the corresponding second text sample to train the initial Hausa speech model, adjusting the initial parameters and the parameters of the fully connected layer to determine the target Hausa speech model.

[0106] For details, please see Figure 2 S23 of the illustrated embodiment will not be described in detail here.

[0107] S34, fixing the adjusted initial parameters and the parameters of the fully connected layer, and initializing the parameters of the language model to obtain a preset Hausa language speech model.

[0108] In order to further improve the accuracy of the trained Hausa speech model, the target Hausa speech model described in this embodiment includes a language model in addition to the convolution unit and the fully connected layer described in S33. Specifically, the language model is connected after the fully connected layer. That is, the target Hausa speech model includes a convolution unit, a fully connected layer and a language model.

[0109] In this embodiment, the parameters of the convolutional unit and the fully connected layer in the target Hausa speech model can be obtained through the above S33 training, and the parameters of the language model can be obtained through initialization settings.

[0110] In the following steps, the language model is trained to determine the parameters in the language model, and then the Hausa speech model is finally determined.

[0111] S35, inputting the Hausa audio sample into a preset Hausa speech model to obtain a third predicted text.

[0112] Specifically, the above S35 may include the following steps:

[0113] S351, using the language units in the language model to perform character probability statistics on the predicted text output by the fully connected layer.

[0114] The electronic device inputs the Hausa audio sample into the Hausa speech model, outputs the predicted text from the fully connected layer of the Hausa speech model, and then uses the language units in the language model to perform character probability statistics.

[0115] The language unit may adopt the N-Gam language model or other models, and no limitation is imposed here. It is only necessary to ensure that the language unit can perform character probability statistics on the predicted text output by the fully connected layer.

[0116] S352, decoding is performed based on the result of character probability statistics to obtain a third predicted text.

[0117] A decoding unit is connected after the language unit to decode the result of character probability statistics output by the language unit to obtain a third predicted text.

[0118] Since Hausa text resources are easier to obtain than audio resources, the language model is used to optimize the decoding process of the prediction probability sequence, further improving the final speech recognition prediction results.

[0119] S36: Based on the third predicted text and the error of the second text sample, adjust the parameters of the language model to determine the target Hausa language speech model.

[0120] After obtaining the third predicted text, the electronic device calculates the loss function using the third predicted text and the second text sample, and adjusts the parameters of the language model based on the calculation result. Through continuous training and parameter adjustment, the parameters in the language model are determined, and then the target Hausa language speech model is finally determined.

[0121] As an optional implementation of this embodiment, the electronic device uses the N-Gam language model to further improve the accuracy of Hausa speech recognition. Specifically, the predicted text output by the fully connected layer is subjected to a sliding window operation of size 4 to form a byte segment sequence of length 4, and then the frequency of occurrence of each byte segment sequence is counted. During the speech recognition process, the character probability sequence output by the N-Gam language model is subjected to Beam Search decoding. During the decoding process, the statistical byte segment frequency in the language unit is introduced to optimize the search process, thereby further improving the accuracy of the recognition result.

[0122] The training method of the Hausa speech model provided in this embodiment further connects a language model after the fully connected layer to further optimize the predicted text output by the fully connected layer, thereby improving the accuracy of the Hausa speech model.

[0123] As a specific implementation of this embodiment, the training of the Hausa language speech model can be carried out as follows: Figure 4 The architecture is represented as follows. Specifically, the training process can be divided into three aspects: data preprocessing to obtain audio samples and their corresponding text samples; an acoustic model, including a convolution unit and a fully connected layer connected to the convolution unit. The training of the model is divided into two parts, namely, English data set pre-training to determine the initial parameters of the convolution unit and Hausa data set transfer learning to determine the parameters of the convolution unit and the fully connected layer; a language model, including a language unit and a decoding unit.

[0124] According to an embodiment of the present invention, an embodiment of a Hausa speech recognition method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0125] In this embodiment, a Hausa language speech recognition method is provided, which can be used in electronic devices such as computers, mobile terminals, etc. Figure 5 : is a flow chart of a Hausa language speech recognition method according to an embodiment of the present invention, Figure 5 As shown, the process includes the following steps:

[0126] S41, obtaining a target Hausa language speech model.

[0127] The target Hausa language speech model is trained according to the training method of the Hausa language speech model described in the embodiment.

[0128] For details, please refer to the description of the training method of the Hausa language speech model described in the above embodiment, which will not be repeated here.

[0129] S42, inputting the target Hausa audio into the target Hausa speech model, and determining the target Hausa text corresponding to the target Hausa audio.

[0130] The electronic device inputs the target Hausa language audio into the target Hausa language speech model, and after being processed by the model, the target Hausa language text can be output.

[0131] The method for recognizing Hausa speech provided in this embodiment, since there is only a difference of 4 letters between Hausa and English characters, when training the Hausa speech model, the Hausa speech model is first trained with English audio samples to obtain the initial parameters of the convolution unit, and then the initial Hausa speech model can be obtained; the initial Hausa speech model is then trained again with the Hausa audio samples, and the parameters of the convolution unit and the fully connected layer are fine-tuned. That is, the transfer learning method can not only avoid the problem of insufficient Hausa audio samples, but also ensure the accuracy of the trained Hausa speech model. On the basis of the accuracy of the Hausa speech model, the accuracy of the recognition result can be guaranteed.

[0132] In the present embodiment, a training device for a Hausa language speech model is also provided. Furthermore, in the present embodiment, a recognition device for Hausa language speech is also provided. The device is used to implement the above-mentioned embodiments and preferred embodiments, and the descriptions thereof will not be repeated. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and contemplated.

[0133] This embodiment provides a training device for a Hausa language speech model. Figure 6 As shown, including:

[0134] A first acquisition module 51 is used to acquire an English audio sample and a first text sample corresponding thereto, and a Hausa audio sample and a second text sample corresponding thereto;

[0135] A first training module 52 is used to train a Hausa speech model using the English audio sample and its corresponding first text sample, determine initial parameters of a convolution unit in the Hausa speech model, and obtain an initial Hausa speech model, wherein the Hausa speech model includes a convolution unit and a fully connected layer connected to the convolution unit;

[0136] The second training module 53 is used to train the initial Hausa speech model using the Hausa audio sample and its corresponding second text sample, and adjust the initial parameters and the parameters of the fully connected layer to determine a target Hausa speech model.

[0137] The training device for the Hausa speech model provided in this embodiment, since the characters of Hausa and English differ by only 4 letters, when training the Hausa speech model, the Hausa speech model is first trained with English audio samples to obtain the initial parameters of the convolution unit, and then the initial Hausa speech model can be obtained; the initial Hausa speech model is then trained again with the Hausa audio samples, and the parameters of the convolution unit and the fully connected layer are fine-tuned. That is, the transfer learning method can not only avoid the problem of insufficient Hausa audio samples, but also ensure the accuracy of the trained Hausa speech model.

[0138] This embodiment also provides a Hausa speech recognition device, such as Figure 7 As shown, including:

[0139] A second acquisition module 61 is used to acquire a target Hausa language speech model, wherein the target Hausa language speech model is trained according to the training method for the Hausa language speech model described in the first aspect of the present invention or any one of the embodiments of the first aspect;

[0140] The recognition module 62 is used to input the target Hausa audio into the target Hausa speech model to determine the target Hausa text corresponding to the target Hausa audio.

[0141] The Hausa speech recognition device provided in this embodiment can ensure the accuracy of the recognition result based on the accuracy of the target Hausa speech model.

[0142] The training device for the Hausa language speech model or the recognition device for the Hausa language speech in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and a memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0143] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0144] The embodiment of the present invention also provides an electronic device having the above Figure 6 The training device of the Hausa speech model shown, or the above Figure 7 The Hausa speech recognition device shown.

[0145] See also Figure 8 , Figure 8 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the electronic device may include: at least one processor 71, such as a CPU (Central Processing Unit), at least one communication interface 73, a memory 74, and at least one communication bus 72. The communication bus 72 is used to realize the connection and communication between these components. The communication interface 73 may include a display screen (Display) and a keyboard (Keyboard), and the optional communication interface 73 may also include a standard wired interface and a wireless interface. The memory 74 may be a high-speed RAM memory (Random Access Memory) or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 74 may optionally be at least one storage device located away from the aforementioned processor 71. The processor 71 may be combined with Figure 6 or Figure 7 In the described device, the memory 74 stores an application program, and the processor 71 calls the program code stored in the memory 74 to execute any of the above method steps.

[0146] The communication bus 72 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The communication bus 72 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0147] Among them, the memory 74 may include a volatile memory (English: volatile memory), such as a random access memory (English: random-access memory, abbreviated: RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk drive (English: hard disk drive, abbreviated: HDD) or a solid-state drive (English: solid-state drive, abbreviated: SSD); the memory 74 may also include a combination of the above types of memory.

[0148] The processor 71 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP.

[0149] The processor 71 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0150] Optionally, the memory 74 is also used to store program instructions. The processor 71 can call the program instructions to implement the present application Figures 1 to 3 The training method of the Hausa language speech model shown in the embodiment, or Figure 4 The Hausa speech recognition method shown in the embodiment.

[0151] The embodiment of the present invention further provides a non-transitory computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the training method of the Hausa language speech model or the Hausa language speech recognition method in any of the above method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memories.

[0152] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for training a Hausa language speech model, characterized in that: include: Obtaining an English audio sample and a first text sample corresponding thereto, and a Hausa audio sample and a second text sample corresponding thereto; The Hausa speech model is trained by using the English audio sample and the first text sample corresponding thereto, and the initial parameters of the convolution unit in the Hausa speech model are determined to obtain an initial Hausa speech model, wherein the Hausa speech model comprises the convolution unit and a fully connected layer connected to the convolution unit, wherein the convolution unit comprises at least one convolution subunit, and each of the convolution subunits in the convolution unit is connected in a residual manner; wherein the convolution subunit comprises a depthwise separable convolution layer, a normalization layer, and an activation function layer connected thereto; Using the Hausa audio sample and its corresponding second text sample to train the initial Hausa speech model, adjusting the initial parameters and the parameters of the fully connected layer to determine a target Hausa speech model; The target Hausa language speech model includes a language model connected to the fully connected layer, and the training method further includes: Fixing the adjusted initial parameters and the parameters of the fully connected layer, and initializing the parameters of the language model to obtain a preset Hausa language speech model; Inputting the Hausa audio sample into the preset Hausa speech model to obtain a third predicted text; Based on the third predicted text and the error of the second text sample, the parameters of the language model are adjusted to determine the target Hausa language speech model.

2. The training method according to claim 1, characterized in that: The step of training the Hausa language speech model by using the English audio sample and the first text sample corresponding thereto to determine the initial parameters of the convolutional unit in the Hausa language speech model comprises: Inputting the English audio sample into a Hausa language speech model, and extracting features from the English audio sample using the convolution unit to obtain audio features; Using the fully connected layer to classify the audio features, and determine a first predicted text corresponding to the English audio sample; Based on the first predicted text and the error of the first text sample, the parameters of the convolution unit are updated to determine the initial parameters of the convolution unit.

3. The training method according to claim 1, characterized in that: The step of training the initial Hausa language speech model by using the Hausa language audio sample and the second text sample corresponding thereto, and adjusting the initial parameters and the parameters of the fully connected layer to determine the target Hausa language speech model includes: Using the initial parameters of the convolution unit and the preset parameters of the fully connected layer, an initial Hausa language speech model is obtained; Inputting the Hausa audio sample into the initial Hausa speech model to obtain a second predicted text; Based on the second predicted text and the error of the second text sample, the initial parameters and the parameters of the fully connected layer are updated to determine the target Hausa speech model.

4. The training method according to claim 1, characterized in that: The step of inputting the Hausa language audio sample into the preset Hausa language speech model to obtain a third predicted text comprises: Using the language units in the language model to perform character probability statistics on the predicted text output by the fully connected layer; Decoding is performed based on the result of the character probability statistics to obtain the third predicted text.

5. A method for recognizing Hausa speech, characterized in that: include: Acquire a target Hausa language speech model, wherein the target Hausa language speech model is trained according to the Hausa language speech model training method according to any one of claims 1 to 4; The target Hausa audio is input into the target Hausa speech model to determine the target Hausa text corresponding to the target Hausa audio.

6. A training device for a Hausa language speech model, based on the training method for a Hausa language speech model according to claim 1, characterized in that: include: A first acquisition module, used for acquiring an English audio sample and a first text sample corresponding thereto, and a Hausa audio sample and a second text sample corresponding thereto; A first training module is used to train a Hausa speech model using the English audio sample and its corresponding first text sample, and determine initial parameters of a convolution unit in the Hausa speech model to obtain an initial Hausa speech model, wherein the Hausa speech model includes a convolution unit and a fully connected layer connected to the convolution unit; The second training module is used to train the initial Hausa speech model using the Hausa audio sample and its corresponding second text sample, and adjust the initial parameters and the parameters of the fully connected layer to determine a target Hausa speech model.

7. A Hausa speech recognition device, characterized in that: include: A second acquisition module is used to acquire a target Hausa language speech model, wherein the target Hausa language speech model is trained according to the training method of the Hausa language speech model according to any one of claims 1 to 4; The recognition module is used to input the target Hausa audio into the target Hausa speech model to determine the target Hausa text corresponding to the target Hausa audio.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for training a Hausa language speech model according to any one of claims 1 to 4, or the method for recognizing Hausa language speech according to claim 5, by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the training method of the Hausa language speech model according to any one of claims 1 to 4, or the Hausa language speech recognition method according to claim 5.

Images