Speech recognition and method for training a model thereof

Abstract

The application relates to a speech recognition method and a model training method thereof. The training method comprises the following steps: generating a first pronunciation dictionary according to a first phoneme set comprising an adult phoneme set and a child phoneme set; training a first acoustic model by using training data and the first pronunciation dictionary to obtain a first state first acoustic model; updating phonemes and a dictionary according to the first state first acoustic model to obtain a second phoneme set and a second pronunciation dictionary; training the first state first acoustic model by using the training data and the second pronunciation dictionary to obtain a second state first acoustic model; generating an initialized second acoustic model according to the second state first acoustic model; and training the initialized second acoustic model by using the training data and the second pronunciation dictionary to obtain a second acoustic model. The second acoustic model can recognize the speech of adults and children, has a high speech recognition rate for the speech of adults and children, and can save system resources.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speech recognition method and a method for training a model thereon. Background Technology

[0002] In recent years, the voice interaction market has received increasing attention, especially for products such as smart speakers with screens in the education sector. Voice interaction is a crucial aspect of using these products, as speech recognition technology can capture the speaker's speech. However, related technologies face the following challenges in user-specific speech recognition: In quiet environments, adult speech recognition rates are generally sufficient for current needs. However, for children, due to their developmental stage, their pronunciation differs significantly from adults, often exhibiting exaggerated pronunciation, large variations in speech rate, elision, and incomplete pronunciation of certain phonemes. This leads to a sharp decline in children's speech recognition rates. Existing technical solutions to address these issues suffer from various problems, such as low speech recognition rates for children and significant waste of system resources. Improving speech recognition rates for both children and adults while conserving system resources is a pressing technical challenge. Summary of the Invention

[0003] In view of this, a speech recognition and model training method is proposed to solve the problems of low speech recognition rate for children and serious waste of system resources in related technologies.

[0004] In a first aspect, embodiments of this application provide a method for training a speech recognition model, the method comprising:

[0005] A first pronunciation dictionary is generated based on a first phoneme set, the first phoneme set including an adult phoneme set and a children's phoneme set, and the first pronunciation dictionary is used to indicate the mapping relationship between characters and phonemes;

[0006] The first acoustic model is trained using training data and the first pronunciation dictionary to obtain the first acoustic model in the first state. The training data includes acoustic features extracted from audio samples and text.

[0007] Based on the first acoustic model of the first state, the first phoneme set and the first pronunciation dictionary are updated to obtain the updated second phoneme set and the second pronunciation dictionary;

[0008] Based on the training data and the second pronunciation dictionary, the first acoustic model of the first state is trained to obtain the first acoustic model of the second state;

[0009] Based on the first acoustic model of the second state, generate an initialized second acoustic model;

[0010] The initial second acoustic model is trained based on the training data and the second pronunciation dictionary to obtain a second acoustic model. The second acoustic model is used to identify the acoustic features extracted from the audio to be identified in conjunction with the second pronunciation dictionary and output the identified text.

[0011] The method provided in the first aspect can train a model that can recognize the speech of both adults and children, with a high speech recognition rate and save system resources, thereby achieving the recognition of the speech of both children and adults.

[0012] According to the first aspect, in a first possible implementation of the method, the first phoneme set and the first pronunciation dictionary are updated according to the first acoustic model of the first state to obtain an updated second phoneme set and a second pronunciation dictionary, including:

[0013] Using the first acoustic model and the first pronunciation dictionary in the first state, the training data is decoded and aligned to obtain the processing result, which includes the decoded phoneme string and the aligned phoneme string for each audio sample.

[0014] Based on the decoded phoneme strings and aligned phoneme strings of all audio samples, determine the replacement frequency of each phoneme in the first phoneme set that is identified as another phoneme.

[0015] Adult and child phonemes whose replacement frequencies meet the phoneme fusion conditions and have the same phoneme meaning are fused to generate fused phonemes.

[0016] Replace the adult and child phonemes corresponding to the fused phonemes in the first phoneme set with the fused phonemes to obtain the second phoneme set;

[0017] The first pronunciation dictionary is updated based on the second phoneme set to obtain the second pronunciation dictionary.

[0018] The first possible implementation method can reduce the number of phonemes in the first phoneme set to obtain the second phoneme set, which simplifies the second pronunciation dictionary based on the second phoneme set, reduces the complexity of model training, and improves the speed of model training.

[0019] According to the first possible implementation of the first aspect, in the second possible implementation of the method, determining the replacement frequency of each phoneme in the first phoneme set that is identified as another phoneme based on the decoded phoneme string and the aligned phoneme string of all audio samples includes:

[0020] Compare the decoded phoneme string and the aligned phoneme string of each audio sample to determine the number of phoneme changes that occur in the decoded phoneme string relative to the aligned phoneme string, wherein the phoneme changes include at least one of insertion, deletion, and replacement;

[0021] Based on the number of phoneme changes in the decoded phoneme string and the total number of phonemes in the aligned phoneme string, determine the minimum edit distance between the decoded phoneme string and the aligned phoneme string for each audio sample;

[0022] Based on the minimized edit distance, the frequency at which each phoneme in the first pronunciation dictionary is identified as a replacement phoneme for another phoneme is determined.

[0023] The second possible implementation method simplifies the process of determining the replacement frequency by minimizing the edit distance, thereby improving the model training speed.

[0024] According to the first possible implementation of the first aspect, in the third possible implementation of the method, adult phonemes and children's phonemes whose replacement frequencies satisfy the phoneme fusion condition and have the same phoneme meaning are fused to generate fused phonemes, including:

[0025] A phoneme confusion matrix is ​​created based on the substitution frequency of each phoneme being identified as another phoneme. The phoneme confusion matrix includes each phoneme in the first phoneme set and the substitution frequency of each phoneme being identified as another phoneme.

[0026] Based on the substitution frequency of adult and child phonemes with the same phoneme meaning in the phoneme confusion matrix, the confusion value between adult and child phonemes with the same phoneme meaning is determined.

[0027] Adult and child phonemes that meet the phoneme fusion conditions and have the same phoneme meaning are fused together to generate a fused phoneme.

[0028] The third possible implementation method utilizes a phoneme confusion matrix and confusion values ​​to determine whether to fuse adult and child phonemes, which can ensure fusion accuracy while improving fusion efficiency.

[0029] According to the first aspect, in a fourth possible implementation of the method, the method further includes:

[0030] After obtaining the second acoustic model, the second pronunciation dictionary is simplified to obtain a simplified second pronunciation dictionary, so that the second acoustic model can be used to identify the acoustic features extracted from the audio to be identified in combination with the simplified second pronunciation dictionary, and output the identified text.

[0031] The fourth possible implementation reduces the number of mapping relationships in the second pronunciation dictionary obtained after simplification, which can improve the speech recognition speed using the second pronunciation model while ensuring the accuracy of speech recognition.

[0032] According to the first aspect, in a fifth possible implementation of the method, the method further includes:

[0033] During training, the model parameters of the second acoustic model are updated based on the loss of the second acoustic model calculated according to the preset loss function.

[0034] The formula for the preset loss function is as follows:

[0035]

[0036] Where f is the loss of the second acoustic model, ∑ w P(O u |S)P(W) is the sum of probabilities that the second acoustic model identifies all possible state sequences S. This represents the maximum probability value obtained by aligning the given state sequence corresponding to sentence u.

[0037] According to the first aspect, in a sixth possible implementation of the method, the first acoustic model is determined based on a hidden Markov model and a Gaussian mixture model, and the initialization of the second acoustic model is determined based on a hidden Markov model and a neural network model.

[0038] Secondly, embodiments of this application provide a speech recognition method applied to a terminal device, the method comprising:

[0039] Receive the acoustic features of the audio to be identified;

[0040] Using a pre-trained second acoustic model and a second pronunciation dictionary, the acoustic features of the audio to be identified are recognized, and the text of the identified audio is output.

[0041] The second acoustic model is trained according to the first aspect and any one or more possible implementation methods of the first aspect.

[0042] The speech recognition method provided in the second aspect of this application uses a pre-trained model that can recognize both adult and children's speech, achieving a high speech recognition rate for both adults and children, and saving system resources.

[0043] Thirdly, embodiments of this application provide a speech recognition method applied to a server, the method comprising:

[0044] Receive audio to be recognized from the terminal device;

[0045] Acoustic features are extracted from the audio to be identified to obtain the acoustic features of the audio to be identified;

[0046] Using a pre-trained second acoustic model and a second pronunciation dictionary, the acoustic features of the audio to be identified are recognized, and the text of the identified audio is sent to the terminal device.

[0047] The second acoustic model is trained according to the first aspect and any one or more possible implementation methods of the first aspect.

[0048] The speech recognition method provided in the third aspect of this application uses a pre-trained model that can recognize both adult and children's speech, achieving a high speech recognition rate for both adults and children, and saving system resources.

[0049] Fourthly, embodiments of this application provide a speech recognition method applied to a server, the method comprising:

[0050] Receive the acoustic features of the audio to be identified from the terminal device;

[0051] Using a pre-trained second acoustic model and a second pronunciation dictionary, the acoustic features of the audio to be identified are recognized, and the text of the identified audio is sent to the terminal device.

[0052] The second acoustic model is trained according to the first aspect and any one or more possible implementation methods of the first aspect.

[0053] The speech recognition method provided in the fourth aspect of this application uses a pre-trained model that can recognize both adult and children's speech, achieving a high speech recognition rate for both adults and children, and saving system resources.

[0054] Fifthly, embodiments of this application provide a speech recognition model training apparatus, the apparatus comprising:

[0055] The dictionary creation module is used to generate a first pronunciation dictionary based on a first phoneme set, which includes an adult phoneme set and a children's phoneme set. The first pronunciation dictionary is used to indicate the mapping relationship between characters and phonemes.

[0056] The first training module is used to train the first acoustic model using training data and the first pronunciation dictionary to obtain the first acoustic model in a first state. The training data includes acoustic features extracted from audio samples and text.

[0057] The dictionary update module is used to update the first phoneme set and the first pronunciation dictionary according to the first acoustic model of the first state, so as to obtain the updated second phoneme set and the second pronunciation dictionary;

[0058] The second training module is used to train the first acoustic model of the first state based on the training data and the second pronunciation dictionary to obtain the first acoustic model of the second state.

[0059] The model generation module is used to generate an initialized second acoustic model based on the first acoustic model in the second state.

[0060] The third training module is used to train the initialized second acoustic model based on the training data and the second pronunciation dictionary to obtain the second acoustic model. The second acoustic model is used to identify the acoustic features extracted from the audio to be identified in combination with the second pronunciation dictionary and output the identified text.

[0061] The device provided in the fifth aspect enables the training of models that can recognize the speech of adults and children, with high speech recognition rate and low system resource consumption, thereby achieving the recognition of speech of both children and adults.

[0062] According to the fifth aspect, in a first possible implementation of the device, the dictionary update module includes:

[0063] The data processing submodule is used to decode and align the training data using the first acoustic model and the first pronunciation dictionary in the first state to obtain the processing result, which includes the decoded phoneme string and the aligned phoneme string for each audio sample.

[0064] The replacement frequency determination submodule is used to determine the replacement frequency of each phoneme in the first phoneme set that is identified as another phoneme based on the decoded phoneme string and aligned phoneme string of all audio samples.

[0065] The phoneme fusion generation submodule is used to fuse adult and child phonemes whose replacement frequencies meet the phoneme fusion conditions and have the same phoneme meaning to generate fused phonemes.

[0066] The set update submodule is used to replace the adult phonemes and child phonemes corresponding to the fused phonemes in the first phoneme set with the fused phonemes to obtain the second phoneme set;

[0067] The dictionary update submodule updates the first pronunciation dictionary based on the second phoneme set to obtain the second pronunciation dictionary.

[0068] The first possible implementation method can reduce the number of phonemes in the first phoneme set to obtain the second phoneme set, which simplifies the second pronunciation dictionary based on the second phoneme set, reduces the complexity of model training, and improves the speed of model training.

[0069] According to the first possible implementation of the fifth aspect, in the second possible implementation of the device, the replacement frequency determining submodule includes:

[0070] The change determination submodule is used to compare the decoded phoneme string and the aligned phoneme string of each audio sample, and determine the number of phoneme changes that occur in the decoded phoneme string relative to the aligned phoneme string, wherein the phoneme changes include at least one of insertion, deletion, and replacement;

[0071] The calculation submodule is used to determine the minimum edit distance between the decoded phoneme string and the aligned phoneme string of each audio sample based on the number of phoneme changes in the decoded phoneme string and the total number of phonemes in the aligned phoneme string.

[0072] The frequency determination submodule is used to determine the replacement frequency of each phoneme in the first pronunciation dictionary that is identified as another phoneme, based on the minimized edit distance.

[0073] The second possible implementation method simplifies the process of determining the replacement frequency by minimizing the edit distance, thereby improving the model training speed.

[0074] According to the first possible implementation of the fifth aspect, in the third possible implementation of the device, the fusion phoneme generation submodule includes:

[0075] The matrix creation submodule is used to create a phoneme confusion matrix based on the substitution frequency of each phoneme being identified as another phoneme. The phoneme confusion matrix includes each phoneme in the first phoneme set and the substitution frequency of each phoneme being identified as another phoneme.

[0076] The confusion value determination submodule is used to determine the confusion value between adult phonemes and children phonemes with the same phonetic meaning based on the substitution frequency of adult phonemes and children phonemes with the same phonetic meaning in the phoneme confusion matrix.

[0077] The phoneme fusion submodule merges adult and child phonemes whose confusion values ​​meet the phoneme fusion conditions and have the same phoneme meaning to generate fused phonemes.

[0078] The third possible implementation method utilizes a phoneme confusion matrix and confusion values ​​to determine whether to fuse adult and child phonemes, which can ensure fusion accuracy while improving fusion efficiency.

[0079] According to the fifth aspect, in a fourth possible implementation of the device, the third training module includes:

[0080] The parameter update module is used to update the model parameters of the second acoustic model during training based on the loss of the second acoustic model calculated by the preset loss function.

[0081] The formula for the preset loss function is as follows:

[0082]

[0083] Where f is the loss of the second acoustic model, ∑ w P(O u |S)P(W) is the sum of probabilities that the second acoustic model identifies all possible state sequences S. This represents the maximum probability value obtained by aligning the given state sequence corresponding to sentence u.

[0084] According to the fifth aspect, in a fifth possible implementation of the device, the device further includes:

[0085] The dictionary simplification module is used to simplify the second pronunciation dictionary after obtaining the second acoustic model, so as to obtain a simplified second pronunciation dictionary, so that the second acoustic model can be used to identify the acoustic features extracted from the audio to be identified in combination with the simplified second pronunciation dictionary, and output the identified text.

[0086] The fifth possible implementation reduces the number of mapping relationships in the second pronunciation dictionary obtained after simplification, which can both improve the speech recognition speed using the second pronunciation model and ensure the accuracy of speech recognition.

[0087] According to the fifth aspect and any possible implementation of the fifth aspect, in a sixth possible implementation of the device, the first acoustic model is determined based on a hidden Markov model and a Gaussian mixture model, and the initialization of the second acoustic model is determined based on a hidden Markov model and a neural network model.

[0088] Sixthly, embodiments of this application provide a voice recognition device applied to a terminal device, the device comprising:

[0089] The receiving module is used to receive the acoustic features of the audio to be identified;

[0090] The speech recognition module is used to identify the acoustic features of the audio to be recognized using a pre-trained second acoustic model and a second pronunciation dictionary, and output the text of the recognized audio.

[0091] The second acoustic model is trained according to the first aspect and any one or more possible implementation methods of the first aspect.

[0092] The speech recognition device provided in the sixth aspect of this application uses a pre-trained model that can recognize both adult and child speech for speech recognition. It has a high speech recognition rate for both adults and children and can save system resources.

[0093] Seventhly, embodiments of this application provide a voice recognition device applied to a server, the device comprising:

[0094] The receiving module is used to receive the audio to be identified from the terminal device;

[0095] The feature extraction module is used to extract acoustic features from the audio to be identified, thereby obtaining the acoustic features of the audio to be identified.

[0096] The speech recognition module is used to identify the acoustic features of the audio to be recognized using a pre-trained second acoustic model and a second pronunciation dictionary, and to send the identified text of the audio to the terminal device.

[0097] The second acoustic model is trained according to the first aspect and any one or more possible implementation methods of the first aspect.

[0098] The speech recognition device provided in the seventh aspect of this application uses a pre-trained model that can recognize both adult and child speech for speech recognition. It has a high speech recognition rate for both adults and children and can save system resources.

[0099] Eighthly, embodiments of this application provide a voice recognition device applied to a server, the device comprising:

[0100] The receiving module is used to receive the acoustic features of the audio to be identified from the terminal device;

[0101] The speech recognition module is used to identify the acoustic features of the audio to be recognized using a pre-trained second acoustic model and a second pronunciation dictionary, and to send the identified text of the audio to the terminal device.

[0102] The second acoustic model is trained according to the first aspect and any one or more possible implementation methods of the first aspect.

[0103] The speech recognition device provided in the eighth aspect of this application uses a pre-trained model that can recognize both adult and child speech for speech recognition. It has a high speech recognition rate for both adults and children and can save system resources.

[0104] In a ninth aspect, embodiments of this application provide an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the methods provided by the first aspect and any one or more possible implementations of the first aspect when executing the instructions.

[0105] In a tenth aspect, embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in an electronic device, a processor in the electronic device executes one or more methods of the first aspect or a variety of possible implementations of the first aspect described above.

[0106] Eleventhly, embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that the computer program instructions, when executed by a processor, implement the method provided by the first aspect and any one or more possible implementations of the first aspect.

[0107] These and other aspects of this application will become more apparent in the description of the following embodiments(s). Attached Figure Description

[0108] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0109] Figure 1 This diagram illustrates the structure of a server according to an embodiment of the present application.

[0110] Figure 2 A schematic diagram of an application scenario according to an embodiment of this application is shown.

[0111] Figure 3A , Figure 3B A flowchart and a schematic diagram of a speech recognition model training method according to an embodiment of this application are shown.

[0112] Figure 4 The flowchart illustrates step S13 of a speech recognition model training method according to an embodiment of this application.

[0113] Figure 5 A schematic diagram of a phoneme confusion matrix according to an embodiment of this application is shown.

[0114] Figure 6 A schematic diagram of a speech recognition model training apparatus according to an embodiment of this application is shown. Detailed Implementation

[0115] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0116] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0117] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0118] To address the aforementioned technical problems, this application provides a speech recognition method and its model training method. The speech recognition method and its model training method of this application can train models to recognize the speech of both adults and children, achieving high recognition rates for both adult and child speech and saving system resources, thereby realizing the recognition of both children's and adults' speech. This speech recognition method and its model training method can be applied to terminal devices or servers.

[0119] The terminal device involved in this application can refer to a device with wireless connectivity, which means that it can connect to other terminal devices via wireless connection methods such as Wi-Fi and Bluetooth. The terminal device of this application can also have wired connection communication capabilities. The terminal device of this application can be touchscreen, non-touchscreen, or screenless. Touchscreen devices can be controlled by clicking or swiping on the display screen with fingers or styluses. Non-touchscreen devices can be connected to input devices such as mice, keyboards, and touch panels to control the terminal device. Screenless devices can be, for example, screenless Bluetooth speakers.

[0120] For example, the terminal device of this application may be a smartphone, netbook, tablet computer, laptop computer, wearable electronic device (such as smart bracelet, smartwatch, etc.), TV, virtual reality device, speaker, smart speaker, robot, etc.

[0121] Terminal devices may include processors. A processor may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, memory, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). Different processing units may be independent devices or integrated into one or more processors. The controller may serve as the central nervous system and command center of the terminal device. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0122] The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces processor waiting time, and thus improves system efficiency.

[0123] The processor can run the speech recognition method and speech recognition model training method provided in the embodiments of this application to facilitate the recognition of children's and adults' speech, thereby improving the speech recognition rate and reducing the redundancy and waste of system resources. The processor 210 may include different devices, such as a CPU and a GPU, in which case the CPU and GPU can work together to execute the speech recognition method and speech recognition model training method provided in the embodiments of this application.

[0124] The terminal device may also include a display screen. The display screen is used to display images, videos, etc. After the processor runs the speech recognition method and speech recognition model training method provided in the embodiments of this application, the terminal device can establish a connection with other terminal devices through an antenna or USB interface, and transmit data and control the display screen 294 to display the corresponding graphical user interface according to the speech recognition method and speech recognition model training method provided in the embodiments of this application.

[0125] The terminal device may also include internal memory. The internal memory can be used to store computer-executable program code, which includes instructions. It may store one or more computer programs corresponding to the speech recognition method and speech recognition model training method provided in the embodiments of this application. These one or more computer programs are stored in the aforementioned memory and configured to be executed by the one or more processors. The one or more computer programs include instructions that can be used to execute the speech recognition method and speech recognition model training method provided in this application.

[0126] In addition, the internal memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0127] Of course, the code for the speech recognition method and the speech recognition model training method provided in this application embodiment can also be stored in external memory. In this case, the processor can run the code for the speech recognition method and the speech recognition model training method stored in external memory through the external memory interface.

[0128] It should be understood that in practical applications, terminal devices may include more or fewer components than those described in the exemplary embodiments above, and this application embodiment does not limit the scope of such devices.

[0129] Figure 1 This diagram illustrates the structure of a server according to an embodiment of the present application, as shown below. Figure 1 As shown, the server may include at least one processor 1001, a communication line 1002, a memory 1003, and at least one communication interface 1004.

[0130] The processor 1001 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0131] The communication line 1002 may include a path for transmitting information between the aforementioned components.

[0132] Communication interface 1004 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, RAN, wireless local area networks (WLAN), etc.

[0133] The memory 1003 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via communication line 1002. The memory may also be integrated with the processor. The memory provided in this application embodiment is generally non-volatile. The memory 1003 is used to store computer execution instructions for executing the scheme of this application and is controlled by the processor 1001 for execution. The processor 1001 is used to execute computer execution instructions stored in the memory 1003, thereby implementing the speech recognition and model training method provided in the above embodiments of this application.

[0134] Optionally, the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0135] In a specific implementation, as one example, the processor 1001 may include one or more CPUs, for example... Figure 1 CPU0 and CPU1 in the CPU.

[0136] In a specific implementation, as one example, the server may include multiple processors, for example... Figure 1 Processors 1001 and 1007 are mentioned. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0137] In a specific implementation, as one embodiment, the server may further include an output device 1005 and an input device 1006. The output device 1005 communicates with the processor 1001 and can display information in various ways. For example, the output device 1005 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 1006 communicates with the processor 1001 and can receive user input in various ways. For example, the input device 1006 may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0138] In related technologies, the recognition rate of adult speech in certain scenarios has reached or even exceeded 95%, but speech recognition for children has not yet been well addressed. In recent years, children's voice interaction has received increasing attention, but because children are in different stages of growth and development, their pronunciation differs significantly from that of adults, and their speech also exhibits different characteristics at different developmental stages, posing obstacles to children's speech recognition. Effective children's speech recognition is a problem that urgently needs to be solved. The following methods exist to achieve speech recognition for both adults and children:

[0139] The first type uses a pre-trained classifier model to identify the received audio and determine whether it belongs to a child or an adult. For audio identified as adult, a pre-trained adult model is used to identify the audio and obtain the corresponding text; for audio identified as child, a pre-trained child model is used to identify the audio and obtain the corresponding text. However, this method suffers from high speech recognition rates due to the accuracy of the classifier model, and because it requires setting up classifier, adult, and child models on the terminal or server, it significantly increases the system's computational efficiency and storage requirements.

[0140] The second approach leverages the inherent differences between adult and children's speech. Adult audio is adjusted to create a simulated child's audio, which is then used to train a speech recognition model. This model is then used to recognize the child's audio to be recognized. Alternatively, for the child's audio to be recognized, a simulated adult audio is first generated based on the differences between adult and children's speech, and then an existing adult speech recognition model is used for recognition. However, this approach requires recognition based on the inherent differences between adult and children's speech and the corresponding speech recognition model. Whether training the speech recognition model based on the inherent differences between adult and children's speech or generating a simulated adult audio and then using an existing adult speech recognition model, both methods distort the child's audio to some extent, reducing the speech recognition rate. Furthermore, the need to recognize the inherent differences between adult and children's speech increases the complexity of speech recognition and reduces efficiency.

[0141] To address the aforementioned technical problems, this application provides a speech recognition method and its model training method. The speech recognition method and its model training method of this application can train models to recognize the speech of both adults and children, achieving high speech recognition rates and saving system resources, thereby realizing the recognition of both children's and adults' speech. This speech recognition method and its model training method can be applied to terminal devices or servers.

[0142] Figure 2 This illustration shows an application scenario diagram according to an embodiment of the present application, such as... Figure 2 As shown, the speech recognition model training method provided in this application can be implemented in terminal device 20 or server 40. Terminal device 20 may include robots, smartphones, smart speakers, computer devices (not shown in the figure), etc. The speech recognition method can also be implemented in terminal device 20 or server 40.

[0143] Alternatively, terminal device 20 can collect the speech emitted by user 10 to obtain the audio to be recognized, and then send the audio to be recognized to server 40 through router 30. Server 40 will extract the acoustic features of the audio to be recognized, and use a pre-trained second acoustic model and second pronunciation dictionary to recognize the acoustic features of the audio to be recognized, and send the recognized text of the audio to be recognized to terminal device 20, so that terminal device 20 can perform corresponding operations based on the text of the audio to be recognized.

[0144] Alternatively, after acquiring the audio to be recognized, the terminal device 20 extracts acoustic features from the audio data and sends these acoustic features to the server 40. The server 40 uses a pre-trained second acoustic model and a second pronunciation dictionary to recognize the acoustic features of the audio to be recognized and sends the recognized text of the audio to be recognized to the terminal device 20, so that the terminal device 20 can perform corresponding operations based on the text of the audio to be recognized.

[0145] Before performing speech recognition, it is necessary to train the model in advance to obtain a second acoustic model for speech recognition and a second pronunciation dictionary required for speech recognition. Figure 3A , Figure 3B A flowchart and schematic diagram illustrating a speech recognition model training method according to an embodiment of this application are shown. Figure 3A , Figure 3B As shown, the method may include steps S11 to S16.

[0146] In step S11, a first pronunciation dictionary is generated based on a first phoneme set, the first phoneme set including an adult phoneme set and a children's phoneme set, and the first pronunciation dictionary is used to indicate the mapping relationship between characters and phonemes.

[0147] In this embodiment, a children's phoneme set corresponding to children can be created in advance based on an existing adult phoneme set. The adult phoneme set can be pre-collected or an existing adult phoneme set. The adult phoneme set can include subsets corresponding to different language types to meet the usage needs of users of different language types. For example:

[0148] Assuming the set of adult phonemes is {a, b, c...}, then the set of children's phonemes {a_child, b_child, c_child...} can be expanded based on {a, b, c...}. Then, the adult and children's phoneme sets are merged to form the first phoneme set {a, b, c..., a_child, b_child, c_child...}.

[0149] Those skilled in the art can set the method of obtaining the adult phoneme set according to actual needs, and this application does not impose any restrictions on this.

[0150] In this embodiment, after determining the first phoneme set, a first pronunciation dictionary is established according to the mapping relationship between phonemes and the pronunciations of characters. Here, characters can be the basic components of the corresponding language type. For example, characters can be Chinese characters, such as "北" (běi), "华" (huá), etc. Characters can also be English words, such as "can", "what", etc. This enables subsequent determination of the phoneme string corresponding to the determined text (including multiple characters) based on the determined text and the first pronunciation dictionary, or determination of the corresponding text based on the determined phoneme string and the first pronunciation dictionary.

[0151] In this embodiment, the first pronunciation dictionary can include the mapping relationship between adult phonemes and adult characters, as well as the mapping relationship between child phonemes and child characters. For example:

[0152] The adult phoneme "b ei3" corresponds to the adult character "北京" (Běijīng).

[0153] The adult phoneme "m a1" corresponds to the adult character "妈" (mā).

[0154] The child phoneme "b_child ei3_child" corresponds to the child character "北京_child" (Běijīng_child).

[0155] The child phoneme "m_child a1_child" corresponds to the child character "妈_child" (mā_child).

[0156] In step S12, the first acoustic model is trained using the training data and the first pronunciation dictionary to obtain the first acoustic model in the first state. The training data includes the acoustic features and text extracted from the audio samples.

[0157] In this embodiment, a first acoustic model can be created in advance based on the Hidden Markov Model (HMM) and the Gaussian Mixture Model (GMM). The obtained first acoustic model has an HMM-GMM network architecture. Then, the "acoustic features extracted from the audio samples" in the training data are input into the first acoustic model to obtain the corresponding decoded phoneme string, and the recognized text corresponding to the decoded phoneme string is determined based on the first pronunciation dictionary. The loss of the first acoustic model is determined according to the recognized text and the "text in the training data", and the model parameters of the first acoustic model are updated according to the loss of the first acoustic model until the loss of the first acoustic model reaches a preset threshold, at which point the training stops, and the first acoustic model in the first state is obtained.

[0158] In this embodiment, the acoustic features and text included in the training data are extracted from audio samples. These audio samples can be pre-collected user speech, which can be speech produced by adults and children of different ages and dialects with different pronunciation characteristics. The acoustic features can include at least one of the following: fundamental frequency (FO), spectrum envelope, and Mel-frequency cepstral coefficients (MFCCs).

[0159] In step S13, the first phoneme set and the first pronunciation dictionary are updated according to the first acoustic model of the first state to obtain the updated second phoneme set and the second pronunciation dictionary.

[0160] Figure 4 The flowchart illustrates step S13 of a speech recognition model training method according to an embodiment of this application. In one possible implementation, such as... Figure 4 As shown, step S13 may include steps S131 to S135. This can reduce the number of phonemes in the first phoneme set to obtain a second phoneme set, thereby simplifying the second pronunciation dictionary obtained based on the second phoneme set, reducing the complexity of model training, and increasing the speed of model training.

[0161] In step S131, the training data is decoded and aligned using the first acoustic model of the first state and the first pronunciation dictionary to obtain the processing result, which includes the decoded phoneme string and the aligned phoneme string for each audio sample.

[0162] In this implementation, the acoustic features extracted from audio samples in the training data are first input into the first acoustic model in the first state to obtain the decoded phoneme string. Furthermore, the first acoustic model in the first state is used to align the phonemes in the training data to obtain the aligned phoneme string.

[0163] In step S132, based on the decoded phoneme strings and aligned phoneme strings of all audio samples, the replacement frequency of each phoneme in the first phoneme set that is identified as another phoneme is determined.

[0164] In one possible implementation, step S132 may include: comparing the decoded phoneme string and the aligned phoneme string of each audio sample to determine the number of phoneme changes that occur in the decoded phoneme string relative to the aligned phoneme string, wherein the phoneme changes include at least one of insertion, deletion, and replacement; determining the minimum edit distance between the decoded phoneme string and the aligned phoneme string of each audio sample based on the number of phoneme changes in the decoded phoneme string and the total number of phonemes in the aligned phoneme string; and determining the replacement frequency of each phoneme in the first pronunciation dictionary being identified as another phoneme based on the minimum edit distance. This simplifies the process of determining the replacement frequency and improves the model training speed.

[0165] The minimum edit distance is the ratio between the number of phoneme changes in the decoded phoneme string relative to the aligned phoneme string and the total number of phonemes in the aligned phoneme string, i.e.:

[0166]

[0167] In this implementation, audio samples can include multiple audio samples, each of which is a sentence with a certain meaning. After determining the minimum edit distance corresponding to each audio sample, the replacement frequency of each phoneme in the first pronunciation dictionary being identified as another phoneme can be further determined based on the number of replacements in the minimum edit distance. For example, the replacement frequency of a phoneme being identified as another phoneme can include: the probability of an adult phoneme 'a' being replaced by the adult phoneme 'ao', the probability of an adult phoneme 'f' being replaced by the child phoneme 'f_child', etc.

[0168] Those skilled in the art can determine the replacement frequency of each phoneme in the first pronunciation dictionary as being identified as another phoneme according to actual needs, and this application does not impose any restrictions on this.

[0169] In step S133, adult phonemes and children's phonemes that meet the phoneme fusion conditions and have the same phoneme meaning are fused to generate fused phonemes.

[0170] In one possible implementation, step S133 may include:

[0171] A phoneme confusion matrix is ​​created based on the frequency at which each phoneme is identified as another phoneme. The phoneme confusion matrix may include each phoneme in the first phoneme set and the frequency at which each phoneme is identified as another phoneme.

[0172] Based on the substitution frequency of adult and child phonemes with the same phoneme meaning in the phoneme confusion matrix, the confusion value between adult and child phonemes with the same phoneme meaning is determined.

[0173] Adult and child phonemes that meet the phoneme fusion conditions and have the same phoneme meaning are fused together to generate a fused phoneme.

[0174] In this way, by using the phoneme confusion matrix and confusion value to determine whether to fuse adult and child phonemes, the accuracy of fusion can be guaranteed while improving the efficiency of fusion.

[0175] In this implementation, Figure 5 A schematic diagram of a phoneme confusion matrix according to an embodiment of this application is shown. Figure 5 As shown, the phoneme confusion matrix can record the substitution probability of each phoneme in the first phoneme set being replaced by another phoneme. Each substitution probability in the phoneme confusion matrix represents the probability that "the phoneme at the corresponding position in the row corresponding to this substitution probability" is replaced by "the phoneme at the corresponding position in the column corresponding to this substitution probability". Based on Figure 5 The phoneme confusion matrix shown can determine the probability that each phoneme is replaced by another phoneme.

[0176] In this implementation, it is necessary to traverse the phoneme confusion matrix to determine the substitution frequency P(substituted phoneme - substituting phoneme) of adult and child phonemes with the same phoneme meaning (i.e., the phonemes themselves have the same meaning, such as adult phoneme m and child phoneme m_child, which are two phonemes with the same phoneme meaning). For example, the substitution probability of adult phoneme m being replaced by child phoneme m_child is P(m-m_child), and the substitution probability of child phoneme m_child being replaced by adult phoneme m is P(m_child-m). This allows for the determination of the confusion value between adult and child phonemes with the same phoneme meaning.

[0177] The confusion value for mutual substitution between adult and child phonemes with the same phonetic meaning can be defined as the ratio of the probability of an adult phoneme with the same phonetic meaning being substituted for a child phoneme to the probability of a child phoneme with the same phonetic meaning being substituted for an adult phoneme. For example, the confusion value between the adult phoneme m and the child phoneme m_child is α(m) = P(m - m_child) / P(m_child - m). The phoneme fusion condition can be that the confusion value is greater than or equal to a preset confusion threshold, which can be 0.7. For example, such as... Figure 5As shown, P(m - m_child) is 0.243 and P(m_child - m) is 0.310. Then α(m) = P(m - m_child) / P(m_child - m) = 0.243 / 0.310 = 0.78 > 0.7. Thus, the adult phoneme m and the child phoneme m_child can be mixed to generate a fused phoneme mM. The larger the value of the preset confusion threshold, the higher the similarity between the generated fused phoneme and the adult and child phonemes with the same phoneme meaning, and the better the accuracy of the fused phoneme.

[0178] In step S134, replace the adult phoneme and the child phoneme corresponding to the fused phoneme in the first phoneme set with the fused phoneme to obtain a second phoneme set.

[0179] For example:

[0180] Assume the generated fused phoneme is mM.

[0181] The first phoneme set is {a, b, c…m…, a_child, b_child, c_child…m_child…}.

[0182] Then replace m and m_child in the first phoneme set with the fused phoneme mM to obtain the second phoneme set {a, b, c……, a_child, b_child, c_child……, mM}, and there is no m or m_child in the second phoneme set. After traversing the phoneme confusion matrix, all fused phonemes can be obtained.

[0183] The phoneme replacement between the first phoneme set and the second phoneme set can be seen in Table 1 below.

[0184] Table 1 Phoneme replacement between the first phoneme set and the second phoneme set

[0185] First phoneme set F f_child G g_child K k_child m m_child s Second phoneme set fF fF gG gG kK kK mM mM sS First phoneme set sh sh_child t t_child w w_child y y_child s_child Second phoneme set shSH shSH tT tT wW wW yY yY sS

[0186] In step S135, update the first pronunciation dictionary according to the second phoneme set to obtain a second pronunciation dictionary.

[0187] In this implementation, update the mapping relationship of the part corresponding to the fused phoneme in the first pronunciation dictionary to obtain the second pronunciation dictionary. Taking the following mapping relationships included in the first pronunciation dictionary as an example:

[0188] The adult phoneme "b ei3" corresponds to the adult character "北京".

[0189] The adult phoneme "m a1" corresponds to the adult character "妈".

[0190] The child phoneme "b_child ei3_child" corresponds to the child character "北京_child".

[0191] The child phoneme "m_child a1_child" corresponds to the child character "妈_child".

[0192] Then, after the change of the second pronunciation dictionary for the above mapping relationship:

[0193] The adult phoneme "b ei3" corresponds to the adult character "北京".

[0194] "b_child ei3_child" corresponds to the child character "北京_child".

[0195] The "mM a1_child" and "mM a1" with fused phonemes correspond to the child character "妈_child" and the adult character "妈" respectively.

[0196] In step S14, according to the training data and the second pronunciation dictionary, the first acoustic model in the first state is trained to obtain the first acoustic model in the second state. The implementation of training the first acoustic model in the first state is the same as that of "training the first acoustic model to obtain the first acoustic model in the first state" above. For relevant descriptions, refer to step S12 above and will not be elaborated here.

[0197] In step S15, according to the first acoustic model in the second state, an initial second acoustic model is generated.

[0198] In this embodiment, the initial second acoustic model can be determined according to the hidden Markov model and the neural network model. Then, according to the model parameters corresponding to the hidden Markov model in the first acoustic model in the second state and the model parameters of the preset neural network model, the initial second acoustic model can be created.

[0199] In step S16, according to the training data and the second pronunciation dictionary, the initial second acoustic model is trained to obtain the second acoustic model. The second acoustic model is used to identify the acoustic features extracted from the audio to be recognized in combination with the second pronunciation dictionary and output the recognized text. Since the second acoustic model can recognize adult and child voices, it avoids occupying system resources.

[0200] In a possible implementation manner, the method further includes: during the training process, updating the model parameters of the second acoustic model according to the loss of the second acoustic model calculated by the preset loss function.

[0201] The training can be stopped based on the model accuracy requirements. A loss threshold can be set, and training is stopped when the loss is less than or equal to the loss threshold to obtain the second acoustic model.

[0202] In one possible implementation, the formula for the preset loss function can be determined based on the LF-CMMI loss strategy, where the LF-CMMI loss is obtained by modifying LF-MMI (Lattice-Free Maximum Mutual Information). The formula for the preset loss function can be:

[0203]

[0204] Where f is the loss of the second acoustic model, O u W represents the feature sequence (i.e., acoustic features) of sentence u. u S represents the word sequence of sentence u. u Let represent the state sequence corresponding to the words in sentence u, and Q represent the set of adult and child sequences in which sentence u is located. The denominator represents all possible state sequences S identified by the second acoustic model and the sum of the probabilities of the sequences; the numerator represents the probability value P(O) calculated by aligning the given state sequences. u |S u )P(W u The maximum value in ) Wu∈Q P(O u |S u )P(W u ).

[0205] That is, ∑ w P(O u |S)P(W) represents the sum of probabilities of the second acoustic model identifying all possible state sequences S. Wu∈Q P(O u |S u )P(W u P(O) represents the maximum probability value obtained by aligning the given state sequence corresponding to sentence u. u |S u P(W) represents the probability that the state sequence corresponding to a word in sentence u corresponds to the feature sequence of sentence u. u P(O) represents the probability of the word sequence in sentence u. u |S) represents the probability that the state sequence corresponding to the word in all sentences corresponds to the feature sequence of sentence u. P(W) represents the probability of the word sequence in all sentences.

[0206] For example, suppose Q represents the set of adult-child sequences containing sentence u, which is {“Beijing is great”, “Beijing_child is great_child”}.

[0207] Then calculate the probability value corresponding to the sentence "Beijing is so nice" u1 as P(adult) = P(O) u1 |S u1 )P(W u1 ).

[0208] The probability value corresponding to the sentence "Beijing_child is so good_child" u2 is calculated as P(child) = P(O) u2 |S u2 )P(W u2 ).

[0209] max Wu∈Q P(O u |S u )P(W u =MAX(P(adult):P(child)).

[0210] In this way, for the identification of non-adults who are children going through puberty, it can be based on max... Wu∈Q P(O u |S u )P(W u This determines the phonemes that will be identified using the second acoustic model, ensuring the accuracy of speech recognition.

[0211] In one possible implementation, the method further includes:

[0212] After obtaining the second acoustic model, the second pronunciation dictionary is simplified to obtain a simplified second pronunciation dictionary, so that the second acoustic model can be used to identify the acoustic features extracted from the audio to be identified in combination with the simplified second pronunciation dictionary, and output the identified text.

[0213] The simplified processing reduces the number of mapping relationships in the second pronunciation dictionary, which in turn reduces the amount of dictionary data. This improves the speech recognition speed using the second pronunciation model while ensuring the accuracy of speech recognition.

[0214] Simplification could mean no longer differentiating between adult and child characters in some or all of the characters in the second pronunciation dictionary. Alternatively, it could mean no longer differentiating between adult and child characters for certain characters, for example:

[0215] In the second pronunciation dictionary, the adult phoneme "b ei3" corresponds to the adult text "北京" (Beijing), and the child phoneme "b_child ei3_child" corresponds to the child text "北京_child". It is simplified to the adult phoneme "b ei3" and the child phoneme "b_child ei3_child" corresponding to the text "北京" (Beijing).

[0216] For the remaining text, the distinction between adult text and child text can still be made. For example:

[0217] The adult phoneme "e4" corresponds to the adult text "饿" (hungry), and the child phoneme "e4_chil" corresponds to the child text "饿_child".

[0218] The training data in the above steps S12, step S14, and step S16 can be partially or completely the same, or completely different. Moreover, the richer the audio samples of the training data, the higher the speech recognition rate of the finally obtained second acoustic model. Since the audio samples are the voices of adults and children, the greater the difference in the pronunciation characteristics of adults and children corresponding to the audio samples and the richer the audio samples. The pronunciation characteristics of adults and children are related to gender, age, physical health status (such as physical health status that can change pronunciation characteristics like having a cold, etc.), and whether it is the voice change period, etc.

[0219] In the above steps S11 - S16, the first acoustic model in the second state is obtained through two trainings based on the HMM - GMM network architecture. Actually, after step S14, the second acoustic dictionary can be further updated based on the first acoustic model in the second state, and then the third model training based on the HMM - GMM network architecture can be carried out. By analogy, the fourth, fifth... can also be carried out. Those skilled in the art can set the number of times of model training based on the HMM - GMM network architecture according to actual needs, and this application does not limit this.

[0220] The present disclosure also provides a speech recognition method applied to a terminal device. The method includes:

[0221] Receiving the acoustic features of the audio to be recognized;

[0222] Using the pre - trained second acoustic model and the second pronunciation dictionary to recognize the acoustic features of the audio to be recognized, and outputting the text of the recognized audio to be recognized.

[0223] The pre - trained second acoustic model is the second acoustic model trained in the above steps S11 - S16. The second pronunciation dictionary can be the second pronunciation dictionary generated in the above step S135 or the simplified second pronunciation dictionary above.

[0224] In one possible implementation, the method may further include:

[0225] The system receives the audio to be identified, extracts its acoustic features, and obtains the acoustic features of the audio to be identified.

[0226] The speech recognition method in this application uses a pre-trained model that can recognize both adult and children's speech, achieving a high speech recognition rate for both adults and children, and saving system resources.

[0227] This disclosure also provides another speech recognition method applied to a server, the method comprising:

[0228] Receive audio to be recognized from the terminal device;

[0229] Acoustic features are extracted from the audio to be identified to obtain the acoustic features of the audio to be identified;

[0230] Using a pre-trained second acoustic model and a second pronunciation dictionary, the acoustic features of the audio to be identified are identified, and the text of the identified audio is sent to the terminal device.

[0231] The pre-trained second acoustic model is the second acoustic model trained in steps S11-S16 above. The second pronunciation dictionary can be the second pronunciation dictionary generated in step S135 above or the simplified second pronunciation dictionary above.

[0232] The speech recognition method in this application uses a pre-trained model that can recognize both adult and children's speech, achieving a high speech recognition rate for both adults and children, and saving system resources.

[0233] This disclosure also provides another speech recognition method applied to a server, the method comprising:

[0234] Receive the acoustic features of the audio to be identified from the terminal device;

[0235] Using a pre-trained second acoustic model and a second pronunciation dictionary, the acoustic features of the audio to be identified are identified, and the text of the identified audio is sent to the terminal device.

[0236] The pre-trained second acoustic model is the second acoustic model trained in steps S11-S16 above. The second pronunciation dictionary can be the second pronunciation dictionary generated in step S135 above or the simplified second pronunciation dictionary above.

[0237] The speech recognition method in this application uses a pre-trained model that can recognize both adult and children's speech, achieving a high speech recognition rate for both adults and children, and saving system resources.

[0238] Figure 6 A schematic diagram of a speech recognition model training apparatus according to an embodiment of this application is shown. Figure 6 As shown, the device includes a dictionary creation module 51, a first training module 52, a dictionary update module 53, a second training module 54, a model generation module 55, and a third training module 56. This device can be applied to terminal devices, servers, etc.

[0239] The dictionary creation module 51 is used to generate a first pronunciation dictionary based on a first phoneme set, the first phoneme set including an adult phoneme set and a children's phoneme set, and the first pronunciation dictionary is used to indicate the mapping relationship between characters and phonemes.

[0240] The first training module 52 is used to train the first acoustic model using training data and the first pronunciation dictionary to obtain the first acoustic model in a first state. The training data includes acoustic features extracted from audio samples and text.

[0241] The dictionary update module 53 is used to update the first phoneme set and the first pronunciation dictionary according to the first acoustic model of the first state, so as to obtain the updated second phoneme set and the second pronunciation dictionary.

[0242] The second training module 54 is used to train the first acoustic model of the first state based on the training data and the second pronunciation dictionary to obtain the first acoustic model of the second state.

[0243] The model generation module 55 is used to generate an initialized second acoustic model based on the first acoustic model in the second state.

[0244] The third training module 56 is used to train the initialized second acoustic model based on the training data and the second pronunciation dictionary to obtain the second acoustic model. The second acoustic model is used to identify the acoustic features extracted from the audio to be identified in combination with the second pronunciation dictionary and output the identified text.

[0245] The device may include a dictionary creation module 51, a first training module 52, a dictionary update module 53, a second training module 54, a model generation module 55, and a third training module 56. Alternatively, some modules may be combined to form a new module. For example, a new module can be constructed based on the first training module 52 and the second training module 54, which can be used to implement the operations of the first training module 52 and the second training module 54. A new module can also be constructed based on the dictionary creation module 51, the dictionary update module 53, and the dictionary simplification module (described below), which can be used to implement the operations of the dictionary creation module 51, the dictionary update module 53, and the dictionary simplification module (described below). Furthermore, a new module can be constructed based on the model generation module 55, the third training module 56, and the recognition module, loss determination module, and parameter update module (described below), which can be used to implement the operations of the model generation module 55, the third training module 56, and the recognition module, loss determination module, and parameter update module (described below). Those skilled in the art can divide the modules of the device according to actual needs; this application does not impose any restrictions on this.

[0246] The implementation methods of the dictionary creation module 51, the first training module 52, the dictionary update module 53, the second training module 54, the model generation module 55, and the third training module 56 can be referred to the relevant descriptions of steps S11-S16 above, and will not be repeated here.

[0247] The speech recognition model training device of this application embodiment can train a model for recognizing the speech of adults and children, and has a high speech recognition rate for adults and children and can save system resources, thereby realizing the recognition of speech of children and adults.

[0248] In one possible implementation, the dictionary update module 53 may include: a data processing submodule, a replacement frequency determination submodule, a fusion phoneme generation submodule, a set update submodule, and a dictionary update submodule.

[0249] The data processing submodule is used to decode and align the training data using the first acoustic model and the first pronunciation dictionary in the first state to obtain the processing result, which includes the decoded phoneme string and the aligned phoneme string for each audio sample.

[0250] The replacement frequency determination submodule is used to determine the replacement frequency of each phoneme in the first phoneme set that is identified as another phoneme, based on the decoded phoneme string and aligned phoneme string of all audio samples.

[0251] The phoneme fusion generation submodule is used to fuse adult and child phonemes that meet the phoneme fusion conditions and have the same phoneme meaning, and generate fused phonemes.

[0252] The set update submodule is used to replace the adult phonemes and child phonemes corresponding to the fused phonemes in the first phoneme set with the fused phonemes to obtain the second phoneme set.

[0253] The dictionary update submodule updates the first pronunciation dictionary based on the second phoneme set to obtain the second pronunciation dictionary.

[0254] In one possible implementation, the replacement frequency determination submodule may include: a change determination submodule, a calculation submodule, and a frequency determination submodule.

[0255] The change determination submodule is used to compare the decoded phoneme string and the aligned phoneme string of each audio sample to determine the number of phoneme changes that occur in the decoded phoneme string relative to the aligned phoneme string, wherein the phoneme changes include at least one of insertion, deletion, and replacement.

[0256] The calculation submodule is used to determine the minimum edit distance between the decoded phoneme string and the aligned phoneme string for each audio sample based on the number of phoneme changes in the decoded phoneme string and the total number of phonemes in the aligned phoneme string.

[0257] The frequency determination submodule is used to determine the replacement frequency of each phoneme in the first pronunciation dictionary that is identified as another phoneme, based on the minimized edit distance.

[0258] In one possible implementation, the fused phoneme generation submodule may include: a matrix creation submodule, an obfuscation value determination submodule, and a phoneme fusion submodule.

[0259] The matrix creation submodule is used to create a phoneme confusion matrix based on the substitution frequency of each phoneme being identified as another phoneme. The phoneme confusion matrix includes each phoneme in the first phoneme set and the substitution frequency of each phoneme being identified as another phoneme.

[0260] The confusion value determination submodule is used to determine the confusion value between adult and child phonemes with the same phonetic meaning based on the substitution frequency of adult and child phonemes with the same phonetic meaning in the phoneme confusion matrix.

[0261] The phoneme fusion submodule merges adult and child phonemes whose confusion values ​​meet the phoneme fusion conditions and have the same phoneme meaning to generate fused phonemes.

[0262] In one possible implementation, the device may further include a parameter update module.

[0263] The parameter update module is used to update the model parameters of the second acoustic model during training based on the loss of the second acoustic model calculated by a preset loss function.

[0264] In one possible implementation, the formula for the preset loss function is:

[0265]

[0266] Where f is the loss of the second acoustic model, O u W represents the feature sequence (i.e., acoustic features) of sentence u. u S represents the word sequence of sentence u. u Let represent the state sequence corresponding to the words in sentence u, and Q represent the set of adult and child sequences in which sentence u is located. The denominator represents all possible state sequences S identified by the second acoustic model and the sum of the probabilities of the sequences; the numerator represents the probability value P(O) calculated by aligning the given state sequences. u |S u )P(W u The maximum value in )

[0267]

[0268] That is, ∑ w P(O u |S)P(W) is the sum of probabilities that the second acoustic model identifies all possible state sequences S. P(O) represents the maximum probability value obtained by aligning the given state sequence corresponding to sentence u. u |S u P(W) represents the probability that the state sequence corresponding to a word in sentence u corresponds to the feature sequence of sentence u. u P(O) represents the probability of the word sequence in sentence u. u |S) represents the probability that the state sequence corresponding to the word in all sentences corresponds to the feature sequence of sentence u. P(W) represents the probability of the word sequence in all sentences.

[0269] In one possible implementation, the device may further include a dictionary simplification module.

[0270] The dictionary simplification module is used to simplify the second pronunciation dictionary after obtaining the second acoustic model, so as to obtain a simplified second pronunciation dictionary, so that the second acoustic model can be used to identify the acoustic features extracted from the audio to be identified in combination with the simplified second pronunciation dictionary, and output the identified text.

[0271] In one possible implementation, the first acoustic model is determined based on a Hidden Markov Model and a Gaussian Mixture Model, and the initialization of the second acoustic model is determined based on a Hidden Markov Model and a Neural Network Model.

[0272] This application provides a voice recognition device for use in a terminal device, the device comprising:

[0273] The receiving module is used to receive the acoustic features of the audio to be identified;

[0274] The speech recognition module is used to identify the acoustic features of the audio to be recognized using a pre-trained second acoustic model and a second pronunciation dictionary, and output the text of the recognized audio.

[0275] The pre-trained second acoustic model is the second acoustic model trained in steps S11-S16 above, and the second pronunciation dictionary can be the second pronunciation dictionary generated in step S135 above or the simplified second pronunciation dictionary above.

[0276] The speech recognition device in this application uses a pre-trained model that can recognize both adult and children's speech, achieving a high speech recognition rate for both adults and children, and saving system resources.

[0277] This application embodiment provides another speech recognition device applied to a server, the device comprising:

[0278] The receiving module is used to receive the audio to be identified from the terminal device;

[0279] The feature extraction module is used to extract acoustic features from the audio to be identified, thereby obtaining the acoustic features of the audio to be identified.

[0280] The speech recognition module is used to identify the acoustic features of the audio to be recognized using a pre-trained second acoustic model and a second pronunciation dictionary, and send the text of the identified audio to the terminal device.

[0281] The pre-trained second acoustic model is the second acoustic model trained in steps S11-S16 above, and the second pronunciation dictionary can be the second pronunciation dictionary generated in step S135 above or the simplified second pronunciation dictionary above.

[0282] The speech recognition device in this application uses a pre-trained model that can recognize both adult and children's speech, achieving a high speech recognition rate for both adults and children, and saving system resources.

[0283] This application provides yet another speech recognition device applied to a server, the device comprising:

[0284] The receiving module is used to receive the acoustic features of the audio to be identified from the terminal device;

[0285] The speech recognition module is used to identify the acoustic features of the audio to be recognized using a pre-trained second acoustic model and a second pronunciation dictionary, and to send the identified text of the audio to the terminal device.

[0286] The pre-trained second acoustic model is the second acoustic model trained in steps S11-S16 above, and the second pronunciation dictionary can be the second pronunciation dictionary generated in step S135 above or the simplified second pronunciation dictionary above.

[0287] The speech recognition device in this application uses a pre-trained model that can recognize both adult and children's speech, achieving a high speech recognition rate for both adults and children, and saving system resources.

[0288] Embodiments of this application provide an electronic device, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing the instructions.

[0289] Embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0290] Embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0291] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital video disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing.

[0292] The computer-readable program instructions or code described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0293] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this application.

[0294] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0295] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0296] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0297] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0298] It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented using hardware (such as circuits or ASICs (Application Specific Integrated Circuits)) that performs the corresponding function or action, or using a combination of hardware and software, such as firmware.

[0299] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0300] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for model training of speech recognition, characterized in that, The method comprises: generating a first pronunciation dictionary according to a first phoneme set, the first phoneme set comprising an adult phoneme set and a child phoneme set, the first pronunciation dictionary being used to indicate a mapping relationship between a character and a phoneme; training a first acoustic model using training data and the first pronunciation dictionary to obtain a first state of the first acoustic model, the training data comprising acoustic features extracted from audio samples and texts; updating the first phoneme set and the first pronunciation dictionary according to the first state of the first acoustic model to obtain an updated second phoneme set and a second pronunciation dictionary; training the first state of the first acoustic model according to the training data and the second pronunciation dictionary to obtain a second state of the first acoustic model; generating an initialized second acoustic model according to the second state of the first acoustic model; training the initialized second acoustic model according to the training data and the second pronunciation dictionary to obtain a second acoustic model, the second acoustic model being used to recognize acoustic features extracted from to-be-recognized audio in combination with the second pronunciation dictionary and output recognized texts.

2. The method of claim 1, wherein, updating the first phoneme set and the first pronunciation dictionary according to the first state of the first acoustic model to obtain an updated second phoneme set and a second pronunciation dictionary comprises: decoding and aligning the training data using the first state of the first acoustic model and the first pronunciation dictionary to obtain a processing result, the processing result comprising a decoded phoneme string and an aligned phoneme string of each audio sample; determining a replacement frequency of each phoneme in the first phoneme set being recognized as another phoneme according to the decoded phoneme string and the aligned phoneme string of all audio samples; fusing adult phonemes and child phonemes that meet a phoneme fusion condition and have the same phoneme meaning to generate a fused phoneme; replacing adult phonemes and child phonemes corresponding to the fused phoneme in the first phoneme set with the fused phoneme to obtain a second phoneme set; updating the first pronunciation dictionary according to the second phoneme set to obtain a second pronunciation dictionary.

3. The method of claim 2, wherein, determining a replacement frequency of each phoneme in the first phoneme set being recognized as another phoneme according to the decoded phoneme string and the aligned phoneme string of all audio samples comprises: comparing the decoded phoneme string and the aligned phoneme string of each audio sample to determine a number of phoneme changes in the decoded phoneme string relative to the aligned phoneme string, the phoneme changes comprising at least one of insertion, deletion, and replacement; determining a minimum edit distance between the decoded phoneme string and the aligned phoneme string of each audio sample according to the number of phoneme changes in the decoded phoneme string and a total number of phonemes in the aligned phoneme string; determining a replacement frequency of each phoneme in the first pronunciation dictionary being recognized as another phoneme according to the minimum edit distance.

4. The method of claim 2, wherein, fusing adult phonemes and child phonemes that meet a phoneme fusion condition and have the same phoneme meaning to generate a fused phoneme comprises: creating a phoneme confusion matrix including each phoneme in the first phoneme set and a substitution frequency of each phoneme being recognized as other phonemes according to the substitution frequency of each phoneme being recognized as other phonemes; determining a confusion value of the adult phoneme and the child phoneme with the same phoneme meaning being substituted with each other according to the substitution frequency of the adult phoneme and the child phoneme with the same phoneme meaning in the phoneme confusion matrix; fusing the adult phoneme and the child phoneme with the same phoneme meaning and satisfying a phoneme fusion condition to generate a fused phoneme.

5. The method of claim 1, wherein, The method further comprises: updating the model parameters of the second acoustic model according to a loss of the second acoustic model calculated by a preset loss function during the training process; wherein a formula of the preset loss function is: wherein, is a loss for the second acoustic model, is a probability sum of all possible state sequences S identified by the second acoustic model, is a maximum probability value obtained by aligning a given state sequence corresponding to the sentence u. a sequence of features representing sentence u, a sequence of words representing sentence u, a sequence of states corresponding to the words in sentence u.

6. The method of claim 1, wherein, The method comprises: After obtaining the second acoustic model, performing a simplification processing on the second pronunciation dictionary to obtain a simplified second pronunciation dictionary, so that the second acoustic model is used to recognize acoustic features extracted from the to-be-recognized audio in combination with the simplified second pronunciation dictionary, and output a recognized text.

7. The method according to any one of claims 1 to 6, characterized in that, The first acoustic model is determined according to a hidden Markov model and a Gaussian mixture model, and the initialized second acoustic model is determined according to a hidden Markov model and a neural network model.

8. A voice recognition method, characterized by, The method is applied to a terminal device and comprises: receiving acoustic features of to-be-recognized audio; recognizing the acoustic features of the to-be-recognized audio by using a pre-trained second acoustic model and a second pronunciation dictionary, and outputting a recognized text of the to-be-recognized audio, wherein the second acoustic model is trained according to the method in any one of claims 1-7.

9. A voice recognition method, characterized by, The method is applied to a server and comprises: receiving to-be-recognized audio from a terminal device; extracting acoustic features of the to-be-recognized audio to obtain acoustic features of the to-be-recognized audio; recognizing the acoustic features of the to-be-recognized audio by using a pre-trained second acoustic model and a second pronunciation dictionary, and sending a recognized text of the to-be-recognized audio to the terminal device, wherein the second acoustic model is trained according to the method in any one of claims 1-7.

10. A speech recognition method, characterized by, The method is applied to a server and comprises: receiving acoustic features of to-be-recognized audio from a terminal device; recognizing the acoustic features of the to-be-recognized audio by using a pre-trained second acoustic model and a second pronunciation dictionary, and sending a recognized text of the to-be-recognized audio to the terminal device, wherein the second acoustic model is trained according to the method in any one of claims 1-7. 11.A model training apparatus for speech recognition, comprising: The apparatus comprises: a dictionary creation module configured to generate a first pronunciation dictionary according to a first phoneme set, the first phoneme set comprising an adult phoneme set and a child phoneme set, and the first pronunciation dictionary being used to indicate a mapping relationship between a character and a phoneme; a first training module configured to train a first acoustic model by using training data and the first pronunciation dictionary to obtain a first acoustic model in a first state, the training data comprising acoustic features and a text extracted from an audio sample; The dictionary updating module is configured to update the first phoneme set and the first pronunciation dictionary according to the first acoustic model of the first state, to obtain a second phoneme set and a second pronunciation dictionary after updating. The second training module is configured to train the first acoustic model of the first state according to the training data and the second pronunciation dictionary, to obtain a first acoustic model of a second state. The model generating module is configured to generate an initialized second acoustic model according to the first acoustic model of the second state. The third training module is configured to train the initialized second acoustic model according to the training data and the second pronunciation dictionary, to obtain a second acoustic model, which is used to recognize acoustic features extracted from to-be-recognized audio in combination with the second pronunciation dictionary, and output recognized text.

12. The apparatus of claim 11, wherein, The dictionary updating module comprises: The data processing submodule is configured to decode and align the training data by using the first acoustic model of the first state and the first pronunciation dictionary, to obtain a processing result, wherein the processing result comprises a decoded phoneme string and an aligned phoneme string of each audio sample. The replacement frequency determining submodule is configured to determine, according to the decoded phoneme string and the aligned phoneme string of all audio samples, a replacement frequency of each phoneme in the first phoneme set that is recognized as another phoneme. The fused phoneme generating submodule is configured to fuse adult phonemes and child phonemes that have the same phoneme meaning and meet a phoneme fusion condition, to generate a fused phoneme. The set updating submodule is configured to replace adult phonemes and child phonemes corresponding to the fused phoneme in the first phoneme set with the fused phoneme, to obtain a second phoneme set. The dictionary updating submodule is configured to update the first pronunciation dictionary according to the second phoneme set, to obtain a second pronunciation dictionary.

13. The apparatus of claim 12, wherein, The replacement frequency determining submodule comprises: The change determining submodule is configured to compare the decoded phoneme string and the aligned phoneme string of each audio sample, to determine a number of phoneme changes in the decoded phoneme string relative to the aligned phoneme string, wherein the phoneme changes comprise at least one of insertion, deletion, and replacement. The calculation submodule is configured to determine, according to the number of phoneme changes in the decoded phoneme string and a total number of phonemes in the aligned phoneme string, a minimum edit distance between the decoded phoneme string and the aligned phoneme string of each audio sample. The frequency determining submodule is configured to determine, according to the minimum edit distance, a replacement frequency of each phoneme in the first pronunciation dictionary that is recognized as another phoneme.

14. The apparatus of claim 12, wherein, The fused phoneme generating submodule comprises: The matrix creating submodule is configured to create a phoneme confusion matrix according to the replacement frequency of each phoneme that is recognized as another phoneme, wherein the phoneme confusion matrix comprises each phoneme in the first phoneme set and the replacement frequency of each phoneme that is recognized as another phoneme. The confusion value determination submodule is configured to determine a confusion value of the adult phoneme and the child phoneme with the same phoneme meaning according to the substitution frequency of the adult phoneme and the child phoneme with the same phoneme meaning in the phoneme confusion matrix. The phoneme fusion submodule is configured to fuse the adult phoneme and the child phoneme with the same phoneme meaning and satisfying a phoneme fusion condition to generate a fused phoneme.

15. The apparatus of claim 11, wherein, The third training module comprises: The parameter updating module is configured to update the model parameters of the second acoustic model according to the loss of the second acoustic model calculated by the preset loss function during the training process. The formula of the preset loss function is as follows: wherein, is a loss for the second acoustic model, is a probability sum of all possible state sequences S identified by the second acoustic model, is a maximum probability value obtained by aligning a given state sequence corresponding to the sentence u. a sequence of features representing sentence u, a sequence of words representing sentence u, a sequence of states corresponding to the words in sentence u.

16. The apparatus of claim 11, wherein, The device further comprises: The dictionary simplification module is configured to simplify the second pronunciation dictionary after obtaining the second acoustic model to obtain a simplified second pronunciation dictionary, so that the second acoustic model is used to recognize the acoustic features extracted from the audio to be recognized in combination with the simplified second pronunciation dictionary and output the recognized text.

17. The apparatus of any of claims 11-16, wherein, The first acoustic model is determined according to a hidden Markov model and a Gaussian mixture model, and the initialized second acoustic model is determined according to a hidden Markov model and a neural network model.

18. A speech recognition apparatus, characterized by comprising: The device is applied to a terminal device and comprises: The receiving module is configured to receive acoustic features of audio to be recognized. The speech recognition module is configured to recognize the acoustic features of the audio to be recognized by using the pre-trained second acoustic model and the second pronunciation dictionary and output the recognized text of the audio to be recognized. The second acoustic model is trained according to the method in any one of claims 1-7.

19. A speech recognition apparatus, characterized by The device is applied to a server and comprises: The receiving module is configured to receive audio to be recognized from a terminal device. The feature extraction module is configured to extract acoustic features of the audio to be recognized to obtain acoustic features of the audio to be recognized. The speech recognition module is configured to recognize the acoustic features of the audio to be recognized by using the pre-trained second acoustic model and the second pronunciation dictionary and send the recognized text of the audio to be recognized to the terminal device. The second acoustic model is trained according to the method in any one of claims 1-7.

20. A speech recognition apparatus, characterized by The device is applied to a server and comprises: The receiving module is configured to receive acoustic features of audio to be recognized from a terminal device. The speech recognition module is configured to recognize the acoustic features of the audio to be recognized by using the pre-trained second acoustic model and the second pronunciation dictionary and send the recognized text of the audio to be recognized to the terminal device. The second acoustic model is trained according to the method in any one of claims 1-7.

21. An electronic device, comprising: The device comprises: A processor; A memory for storing processor-executable instructions; The processor is configured to implement the method in any one of claims 1-7 or the method in any one of claims 8-10 when executing the instructions.

22. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by a processor, implement the method of any one of claims 1-7, or implement the method of any one of claims 8-10.

23. A computer program product, characterised in that, The computer program product, when running on a computer, causes the computer to perform the method of any one of claims 1-7, or implement the method of any one of claims 8-10.

Images