A speech recognition method and related device

By using delay neural networks and syllable modeling units in speech recognition technology and combining the front and back syllable information for recognition, the problem of excessive granularity of phoneme modeling units in the prior art is solved, and the accuracy and applicable scenarios of speech recognition are improved.

CN114360510BActive Publication Date: 2024-12-06TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210042387.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-14
Publication Date
2024-12-06
Estimated Expiration
2042-01-14

AI Technical Summary

Technical Problem

The existing speech recognition technology uses phonemes as modeling units, and the granularity is too fine, resulting in high requirements for speech data quality, difficulty in adapting to complex scenarios, low recognition accuracy, and limited applicable scenarios.

Method used

The delay neural network is used as the acoustic model and the syllables are used as the modeling unit. The syllable probability distribution of speech frames is determined through multiple syllable modeling units of the output layer, and the recognition is combined with the front and rear syllable information to improve the recognition accuracy and reduce the speech data. Quality requirements.

Benefits of technology

It improves the accuracy and robustness of speech recognition, expands the applicable scenarios of speech recognition technology, and can obtain more accurate recognition results under low-quality speech data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114360510B_ABST
    Figure CN114360510B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses a speech recognition method and related devices, which at least relate to speech recognition technology in artificial intelligence. The speech data to be recognized is used as the input data of the time-delay neural network in the acoustic model. Since the output layer of the time-delay neural network includes acoustic modeling units corresponding to multiple syllables, the time-delay neural network can use syllables as the recognition granularity to obtain the syllable probability distribution corresponding to the speech frames included in the speech data. When the output layer recognizes syllables, the syllable information before and after the speech frame can be combined to assist in judging the syllable to which the speech frame belongs based on the pronunciation rules to output a more accurate syllable probability distribution. Moreover, since a syllable is generally composed of one or more phonemes and has a higher fault tolerance, not only a more accurate speech recognition result can be obtained based on the syllable probability distribution, but also the quality requirements for the speech data to be recognized are not high, which effectively expands the applicable scenarios of speech recognition technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of speech recognition, and in particular to a speech recognition method and related devices. Background Art

[0002] Speech recognition technology can provide users with speech content recognition services, and can be applied to various scenarios, such as speech-to-text, voice wake-up, human-computer interaction, etc. In specific implementations, the acoustic features of the speech data to be recognized can be extracted through the acoustic model, and the corresponding speech recognition results can be determined based on the acoustic features.

[0003] The related technology mainly uses phonemes as the modeling units of acoustic models. Phonemes are the smallest speech units divided according to the natural properties of speech. They are analyzed based on the pronunciation actions in the syllables, and one action constitutes a phoneme.

[0004] However, the modeling granularity of phonemes is relatively fine, and this fine-grained speech recognition method has very high requirements for the quality of speech data to be recognized. Slight pronunciation errors may directly affect the recognition results. As a result, speech recognition technology is difficult to adapt to some speech recognition scenarios. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a speech recognition method and related devices for improving the accuracy of speech recognition results and expanding the application scenarios of speech recognition technology.

[0006] On the one hand, an embodiment of the present application provides a speech recognition method, the method comprising:

[0007] Acquiring an acoustic model and speech data to be recognized, wherein the acoustic model includes a time-delay neural network, and an output layer of the time-delay neural network includes acoustic modeling units corresponding to a plurality of syllables respectively;

[0008] Using the speech data as input data of the time-delay neural network, determining the syllable probability distribution corresponding to the speech frames included in the speech data through the time-delay neural network, wherein the syllable probability distribution is used to identify the probability that the speech frames correspond to the multiple syllables respectively;

[0009] A speech recognition result corresponding to the speech data is determined according to the syllable probability distribution.

[0010] On the other hand, an embodiment of the present application provides a speech recognition device, the device comprising an acquisition unit, a syllable probability distribution determination unit, and a speech recognition result determination unit;

[0011] The acquisition unit is used to acquire an acoustic model and speech data to be recognized, wherein the acoustic model includes a time-delay neural network, and an output layer of the time-delay neural network includes acoustic modeling units corresponding to a plurality of syllables respectively;

[0012] The syllable probability distribution determination unit is used to use the speech data as input data of the time-delay neural network, and determine the syllable probability distributions corresponding to the speech frames included in the speech data through the time-delay neural network, wherein the syllable probability distributions are used to identify the probabilities that the speech frames correspond to the multiple syllables respectively;

[0013] The speech recognition result determination unit is used to determine the speech recognition result corresponding to the speech data according to the syllable probability distribution.

[0014] On the other hand, an embodiment of the present application provides a computer device, the device comprising a processor and a memory:

[0015] The memory is used to store program code and transmit the program code to the processor;

[0016] The processor is used to execute the method described in the above aspects according to the instructions in the program code.

[0017] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method described in the above aspects.

[0018] On the other hand, an embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method described in the above aspects.

[0019] It can be seen from the above technical solution that, for the speech data to be recognized, an acoustic model including a time-delay neural network is obtained. The speech data is used as the input data of the time-delay neural network. Since the output layer of the time-delay neural network includes acoustic modeling units corresponding to multiple syllables, the time-delay neural network can use syllables as the recognition granularity, and obtain the syllable probability distribution corresponding to the speech frames included in the speech data through each acoustic modeling unit of the output layer. In the process of speech feature extraction and transmission to the output layer, the time-delay neural network will carry rich contextual information of the speech frame in the speech data. The contextual information can reflect the relevant information of the syllables before and after the speech frame in the speech data. Therefore, when the output layer is used for syllable recognition, the syllable to which the speech frame belongs can be auxiliary judged based on the pronunciation rules in combination with the information of the syllables before and after the speech frame, so as to output a more accurate syllable probability distribution. Moreover, since a syllable is generally composed of one or more phonemes, it has a higher fault tolerance. Even if there is a pronunciation error of individual phonemes in a syllable in the speech data, the impact on the recognition result of the entire syllable is smaller than that of the phonemes. Therefore, even if the quality of the speech data to be recognized is not high, a more accurate speech recognition result can be obtained based on the syllable probability distribution, which effectively expands the applicable scenarios of speech recognition technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A schematic diagram of an application scenario of a speech recognition method provided in an embodiment of the present application;

[0022] Figure 2 A flow chart of a speech recognition method provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of a TDNN provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of an acoustic model provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of an application scenario embodiment of a speech recognition system provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of an acoustic model provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of a speech recognition device provided in an embodiment of the present application;

[0028] Figure 8 A structural diagram of a terminal device provided in an embodiment of the present application;

[0029] Fig. 9 A structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0031] When recognizing speech data, related technologies use phonemes as the modeling units of acoustic models. For example, the phoneme sequence corresponding to the keyword "hello" is "n iy3 hh aw3". The keyword "hello" can only be recognized if the phoneme sequence "n iy3 hhaw3" is recognized in the speech data. Since the modeling granularity of phonemes is too fine, it has high requirements on the quality of speech data. If one factor in a keyword is not pronounced standardly, the keyword recognition fails, resulting in a low accuracy rate of the speech recognition result. Even in some complex speech recognition scenarios such as noisy background sound and non-standard user speech, the robustness of speech recognition is low, which makes speech recognition technology less applicable.

[0032] Based on this, the embodiment of the present application provides a speech recognition method, which not only improves the accuracy of the speech recognition results, but also has lower requirements on the quality of speech data, effectively expanding the applicable scenarios of speech recognition technology.

[0033] The speech recognition method provided in the embodiment of the present application is based on artificial intelligence. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0034] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0035] In the embodiments of the present application, the artificial intelligence technologies mainly involved include the above-mentioned speech processing technologies and other directions.

[0036] The speech recognition method provided in this application can be applied to speech recognition devices with data processing capabilities, such as terminal devices and servers. Among them, the terminal devices involved in this application can specifically be smart phones, tablet computers, laptops, PDAs, personal computers, smart watches, smart speakers, vehicle-mounted devices, wearable devices, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc., but are not limited to this. The server involved in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this. The number of servers and terminal devices is also not limited.

[0037] The aforementioned speech recognition devices may have speech processing technology. The key technologies of speech technology include automatic speech recognition technology, speech synthesis technology and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the development direction of human-computer interaction in the future, among which speech has become one of the most promising human-computer interaction methods in the future.

[0038] In the speech recognition method provided in the embodiment of the present application, the artificial intelligence model adopted mainly involves the application of speech recognition technology, and the speech recognition result corresponding to the speech data to be recognized is determined through the acoustic model.

[0039] In order to facilitate understanding of the technical solution of the present application, the voice recognition method provided in the embodiment of the present application is introduced below in combination with actual application scenarios, taking the terminal device as the voice recognition device.

[0040] See also Figure 1, which is a schematic diagram of an application scenario of a speech recognition method provided by an embodiment of the present application. Figure 1 In the application scenario shown, the terminal device is a smart speaker 100, which is used to recognize the keyword "hello" in the voice data to be recognized.

[0041] The smart speaker 100 obtains an acoustic model. Among them, the acoustic model includes a time-delay neural network, and the output layer of the time-delay neural network includes acoustic modeling units corresponding to multiple syllables. In the embodiment of the present application, phonemes are no longer used as acoustic modeling units, but syllables are used as acoustic modeling units. Syllables are the smallest speech units in a language that are pronounced by a combination of a single vowel phoneme and a consonant phoneme. A single vowel phoneme can also form a syllable by itself, that is, a syllable can include one or more phonemes. Taking the keyword "hello" as an example, the corresponding syllable sequence is "ni3 hao3" and the phoneme sequence is "n iy3hh aw3". Therefore, compared with phonemes, syllables have higher fault tolerance. Even if there is a pronunciation error of an individual phoneme in a syllable in the voice data, the impact on the recognition result of the entire syllable is smaller than that of the phoneme, thereby reducing the requirements for the quality of the voice data to be recognized.

[0042] After the smart speaker 100 obtains the voice data to be recognized, it inputs the voice data into the time-delay neural network. The time-delay neural network includes a multi-layer network, and each layer of the network has a strong abstract ability for voice features. Moreover, the time-delay neural network has a wider contextual field of view and can capture a wider range of contextual information. The contextual information can reflect the relevant information of the syllables before and after the voice frame in the voice data. Therefore, when the syllables are recognized through the output layer, the information of the previous and next syllables of the voice frame can be combined to assist in judging the syllable to which the voice frame belongs based on the pronunciation rules, so that the output syllable probability distribution for the syllable sequence of "ni3 hao3" is more accurate.

[0043] The smart speaker 100 determines the speech recognition result corresponding to the speech data more accurately based on the more accurate syllable probability distribution. As a result, the accuracy of the speech recognition result is improved, the requirements for the quality of the speech data are reduced, and the applicable scenarios of the speech recognition technology are effectively expanded.

[0044] In conjunction with the accompanying drawings, a speech recognition method provided in an embodiment of the present application is introduced below using a server as a speech recognition device.

[0045] See also Figure 2 , which is a flow chart of a speech recognition method provided by an embodiment of the present application. Figure 2 As shown, the speech recognition method comprises the following steps:

[0046] S201: Acquire an acoustic model and speech data to be recognized.

[0047] In the related technology, the Gaussian Mixed Mode (GMM)-Hidden Markov Model (HMM), which uses phonemes as modeling units, has become the mainstream system for automatic speech recognition (ASR) tasks. However, with the accumulation of annotated corpora and the rapid growth of computer computing power, the effects of many newly proposed deep neural network models have greatly surpassed the GMM-HMM model and have been further improved. Due to the powerful modeling capabilities of these deep neural networks, the output labels of the acoustic model no longer need to be subdivided into phonemes. Many research works have found that acoustic models can also use larger modeling granularity, and in most cases can achieve better results.

[0048] Based on this, the embodiment of the present application no longer uses phonemes as acoustic modeling units, but syllables as acoustic modeling units. Among them, syllables are mainly composed of initial consonants, finals and tones, and each syllable can be decomposed into one or more phonemes. See Table 1, taking the keywords "hello" and "world" as examples to illustrate the difference between syllables and phonemes.

[0049] Table 1 The difference between syllables and phonemes

[0050] Keywords Syllable sequence Phoneme sequence Hello ni3 hao3 n iy3 hh aw3 world shi4 jie4 sh iy4 j iy4 eh4

[0051] Compared with phonemes, syllables can carry more contextual information, thereby improving the contextual learning ability of the acoustic model and increasing the coverage of voice keywords. In addition, syllables are more consistent with the human speaking process and are easier to understand. Moreover, since syllables are generally composed of one or more phonemes, they have higher fault tolerance. Even if there is a pronunciation error of individual phonemes in a syllable in the voice data, the impact on the recognition result of the entire syllable is smaller than that of phonemes, thereby reducing the requirements for the quality of voice data and effectively expanding the applicable scenarios of voice recognition technology.

[0052] Among them, the acoustic model (AM) includes a time delay neural network (TDNN), which will be explained later. Voice data, also known as sound files, is data recorded by voice, such as the sound of waking up a smart speaker can be called voice data.

[0053] It can be understood that in the specific implementation of the present application, user-related data such as voice data is involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0054] It should be noted that the acoustic model and the speech data to be recognized can be obtained at the same time, or the acoustic model can be obtained in advance and then the speech data to be recognized can be obtained, or the acoustic model can be obtained after the speech data to be recognized is obtained. This application does not make any specific limitations on this.

[0055] S202: Using the speech data as input data of the time-delay neural network, and determining the syllable probability distribution corresponding to the speech frames included in the speech data through the time-delay neural network.

[0056] The acoustic model can convert speech input into syllable output represented by acoustics. In the embodiment of the present application, the probability distribution of the syllable to which each speech frame belongs is calculated. The acoustic model includes TDNN, such as Figure 3 As shown, this figure is a schematic diagram of a TDNN provided in an embodiment of the present application.

[0057] like Figure 3 As shown in the figure, TDNN includes multiple layers of networks, each of which has a strong ability to abstract speech features. Moreover, TDNN has a wider contextual vision, can capture a wider range of contextual information, and has a stronger ability to model speech time-dependent information.

[0058] In the embodiment of the present application, the output layer of TDNN includes acoustic modeling units corresponding to multiple syllables respectively. After the speech data is input into TDNN, TDNN can use syllables as the recognition granularity to determine the syllable probability distribution corresponding to the speech frames included in the speech data. Among them, the syllable probability distribution is used to identify the probability that the speech frame corresponds to multiple syllables respectively. Figure 3 As shown in the figure, the output layer of TDNN can output the probability corresponding to each syllable. It should be noted that Figure 3 Only 4 syllables are used as an example, but not limited to 4 syllables. For example, all syllables required for Chinese can be counted, and all syllables are represented by the output layer, so as to determine the recognition result corresponding to the speech data to be recognized input by the Chinese user.

[0059] Since the time-delay neural network carries rich contextual information of the speech frame in the speech data during the process of speech feature extraction and transmission to the output layer, the contextual information can reflect the relevant information of the syllables before and after the speech frame in the speech data. Therefore, when the syllables are recognized through the output layer, the information of the preceding and following syllables of the speech frame can be combined to assist in judging the syllable to which the speech frame belongs based on the pronunciation rules, so as to output a more accurate syllable probability distribution.

[0060] S203: Determine a speech recognition result corresponding to the speech data according to the syllable probability distribution.

[0061] Among them, the syllable probability distribution is used to identify the probability that a speech frame corresponds to multiple syllables. For example, the output layer of TDNN will output the probabilities corresponding to 100 syllables respectively, then the syllable probability distribution obtained by TDNN is a 1×100 row vector, each element in the row vector represents the probability corresponding to the syllable, and the sum of the probabilities corresponding to the 100 elements is 1. It can be understood that the greater the probability of a syllable, the greater the possibility that the actual pronunciation of the speech frame is the same as the pronunciation of the syllable. Continuing with the above example, if the syllable probability distribution obtained by speech frame A is [0.1, 0.8, 0.1, 0, ..., 0] (95 0s are omitted in the middle), then the actual pronunciation of speech frame A is more likely to be the pronunciation composed of the first three syllables, or the actual pronunciation of speech frame A is more likely to be the pronunciation of the second syllable. This application does not make specific restrictions on this, and technicians in this field can set it according to the actual application scenario.

[0062] After obtaining the syllable probability distribution corresponding to each speech frame in the speech data, the speech recognition result corresponding to the speech data can be further determined according to the syllable probability distributions corresponding to the multiple speech frames. For example, after the user inputs a speech, it is used as the speech data to be recognized. Through TDNN, the syllable probability distribution corresponding to the speech frames included in the speech data can be obtained, and then the speech recognition result can be determined.

[0063] It can be seen from the above technical solution that, for the speech data to be recognized, an acoustic model including a time-delay neural network is obtained. The speech data is used as the input data of the time-delay neural network. Since the output layer of the time-delay neural network includes acoustic modeling units corresponding to multiple syllables, the time-delay neural network can use syllables as the recognition granularity, and obtain the syllable probability distribution corresponding to the speech frames included in the speech data through each acoustic modeling unit of the output layer. In the process of speech feature extraction and transmission to the output layer, the time-delay neural network will carry rich contextual information of the speech frame in the speech data. The contextual information can reflect the relevant information of the syllables before and after the speech frame in the speech data. Therefore, when the output layer is used for syllable recognition, the syllable to which the speech frame belongs can be auxiliary judged based on the pronunciation rules in combination with the information of the syllables before and after the speech frame, so as to output a more accurate syllable probability distribution. Moreover, since a syllable is generally composed of one or more phonemes, it has a higher fault tolerance. Even if there is a pronunciation error of individual phonemes in a syllable in the speech data, the impact on the recognition result of the entire syllable is smaller than that of the phonemes. Therefore, even if the quality of the speech data to be recognized is not high, a more accurate speech recognition result can be obtained based on the syllable probability distribution, which effectively expands the applicable scenarios of speech recognition technology.

[0064] It should be noted that the present application can be applied to voice keyword detection (Keyword spotting, KWS) tasks such as wake-up interaction, audio and video file voice keyword detection in smart devices. Due to the limitations of annotated corpus and computing power, the acoustic models such as Gaussian Mixed Mode (GMM), Deep Neural Networks (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) are not strong enough, resulting in poor results in smart devices. In order to cope with the limited computing resource constraints of smart devices and the processing requirements of massive audio and video files, not only does the speed of keyword detection need to be fast enough, but the detection results also need to be accurate enough.

[0065] Based on this, see Figure 4 , which is a schematic diagram of an acoustic model provided in an embodiment of the present application. Figure 4As shown in the figure, the acoustic model includes not only TDNN but also a decoder. Since TDNN has a wider contextual vision and can capture wider contextual information, it can further increase the acoustic model's modeling ability for speech timing information, that is, TDNN can process timing information more efficiently, thereby improving the accuracy of keyword detection. The decoder is used to determine whether the speech data includes the speech recognition result of the keyword based on the syllable probability distribution, so as to determine whether to wake up the corresponding terminal device based on the speech recognition result.

[0066] For the convenience of explanation, the following takes the wake-up interaction scenario of a smart device as an example, see S2031-S2033:

[0067] S2031: Determine a keyword corresponding to the wake-up scene according to the wake-up scene corresponding to the voice data.

[0068] It is understandable that due to different wake-up scenarios, the corresponding keywords may be different, and those skilled in the art can set them according to actual needs. For example, the keyword corresponding to the smart home wake-up scenario is "open XX (home device)", and the keyword corresponding to the virtual assistant wake-up scenario is "XXX (virtual assistant name)", etc. According to the setting of keywords by those skilled in the art, after obtaining the voice data to be recognized, the keyword corresponding to the wake-up scenario is determined according to the corresponding wake-up scenario.

[0069] It should be noted that S2031 can be executed after obtaining the syllable probability distribution, or can be executed after obtaining the voice data to be recognized, and this application does not make any specific limitation on this.

[0070] S2032: Determine, through a decoder, a speech recognition result for identifying whether the speech data includes a keyword according to the syllable probability distribution.

[0071] Among them, the decoder is used to decompress the syllable probability distribution into a speech recognition result. Therefore, after obtaining the syllable probability distribution through TDNN, not only can the speech recognition result corresponding to the speech data be determined by the decoder based on the syllable probability distribution, but the decoder can also be used to determine the speech recognition result used to identify whether the speech data includes keywords.

[0072] As a possible implementation method, for a given keyword, a decoder based on a weighted finite state transducer (WFST) can be used to directly output the speech recognition result of whether the keyword is included. Among them, WFST provides a unified framework that can integrate information from acoustic models, pronunciation dictionaries, and language models in a unified way to find the optimal path in a huge search space, thereby improving the search speed of the model.

[0073] As a possible implementation method, during the decoding process, a WFST-based Lattice static decoding strategy can be further adopted to find the optimal path in the search space and improve the search speed of the model.

[0074] As a possible implementation method, during the decoding process, the decoding speed can also be improved by methods such as frame pruning. For example, the generated syllable probability distribution is a 1×100 vector, the first three probabilities are 0.1, 0.8 and 0.1 respectively, and the last 97 probabilities are all 0. Through frame pruning, the last 97 are pruned, and the 1×100 vector is changed to a 1×3 vector, so that only the 1×3 vector needs to be decoded, thereby improving the decoding speed.

[0075] S2033: If the voice recognition result indicates that the voice data includes keywords, the terminal device corresponding to the wake-up scene is woken up.

[0076] It can be seen from the above technical solution that by using syllables instead of phonemes as the modeling unit of the acoustic model, the granularity becomes coarser, which not only reduces the memory occupied, but also requires fewer computing resources. For voice keywords, syllables have higher coverage, higher fault tolerance, and higher accuracy. At the same time, for voice data to be recognized of the same length, the speed of detecting keywords based on syllables is faster. When dealing with limited computing resource constraints and the processing requirements of massive audio and video files, since the acoustic model includes TDNN and decoder, TDNN can further increase the modeling ability of the acoustic model for speech timing information, and the decoder can determine whether the voice data includes the speech recognition results of keywords, thereby improving the accuracy of keyword detection. Therefore, the acoustic model provided in the embodiment of the present application not only requires faster keyword detection, but also higher accuracy of the detection results, and its performance and speed have achieved a better balance, and the effect in smart devices will be better.

[0077] With respect to the aforementioned S2032, the embodiment of the present application does not specifically limit the manner in which the speech recognition result is determined by the decoder. For example, the speech recognition result for identifying whether the speech data includes keywords can be determined by the decoder based on the syllable probability distribution and the matching vocabulary.

[0078] For example, in a virtual assistant wake-up scenario, the corresponding keyword is "Hello World", and a corresponding keyword table is established based on this keyword. After a user inputs a piece of speech, this speech is used as the speech data to be recognized. The acoustic model provided by the embodiments of the present application is used to determine the syllable probability distribution. The decoder searches the keyword table corresponding to "Hello World" and combines the syllable probability distribution to determine whether the speech data input by the user includes "Hello World". If the speech data includes "Hello World", the decoder can output a result of "including the keyword". Further, the virtual assistant can be woken up based on this speech recognition result to provide services to the user, as shown in S2033. Among them, the terminal device corresponding to the wake-up scenario can be terminal devices such as smart speakers, virtual assistants, and smart furniture, and the present application does not make specific limitations on this.

[0079] Among them, the keyword table is established with the keyword as the core and includes the possible pronunciations corresponding to the keyword. The embodiments of the present application do not specifically limit the determination method of the keyword table. For example, a matching word table for the keyword can be constructed with syllables as the division granularity according to similar syllables and / or polyphonic syllables, as well as the syllables corresponding to the keyword. The matching word table is enriched through similar syllables and / or polyphonic syllables, effectively improving the coverage of the keyword, thereby avoiding missed wake-up and enhancing the user experience. The following will separately explain similar syllables and polyphonic syllables.

[0080] Similar syllables: Similar syllables are the similar syllables corresponding to the syllables included in the keyword determined according to the similarity of pronunciation. For example, the syllable sequence of the keyword "Hello" is "ni3 hao3". When the keyword is pronounced loudly and drawn out, "ni" may be pronounced as the first tone instead of the third tone. Therefore, the syllable sequence "ni1 hao3" can be added to the matching keyword table as a similar syllable.

[0081] Polyphonic syllables: A polyphonic character has more than one pronunciation. If there is a polyphonic character in the keyword, the corresponding polyphonic syllables of the polyphonic character are determined, and all its pronunciations should be included in the keyword table to prevent missed wake-up caused by the user's accent. Continuing with the keyword "Hello" as an example, the syllable sequence "ni3 hao4" can be added to the matching keyword table as a similar syllable.

[0082] As a possible implementation, some error-prone pronunciations will also be added to the keyword table. For example, the keyword "傥" is easily pronounced as "dang (the third tone)", so the syllable "dang3" will be included in the keyword table corresponding to the keyword "傥".

[0083] As a possible implementation method, TDNN can not only combine the syllable information before and after the speech frame, but also combine the speech frame features corresponding to at least one frame before and after the i-th speech frame in the speech data to further improve the accuracy of the syllable probability distribution. Taking TDNN including N layers of feature extraction layers as an example, for the i-th speech frame in the speech data, S202 can be implemented through S2021-S2023, as follows:

[0084] S2021: According to the output features of the j-1th feature extraction layer for the i-th speech frame, the speech frame features of the i-th speech frame are determined through the j-th feature extraction layer.

[0085] Each layer of the network included in TDNN can be called a feature extraction layer, and each feature extraction layer will extract features for each speech frame included in the speech data to be recognized. The following is an example of taking any two feature extraction layers in the N feature extraction layers, i.e., the j-1 feature extraction layer and the j feature extraction layer, for the i-th speech frame. Where j∈N.

[0086] For the i-th speech frame, the j-1-th feature extraction layer will perform feature extraction to output the output features for the i-th speech frame, which is then input into the j-th feature extraction layer. The j-th feature extraction layer performs feature extraction to determine the speech frame features of the i-th speech frame.

[0087] S2022: Determine the output feature of the i-th speech frame at the j-th feature extraction layer by using the speech frame feature and the speech frame feature corresponding to at least one frame before and after the i-th speech frame in the speech data.

[0088] In order to further improve the contextual learning ability of the acoustic model, after determining the speech frame features of the i-th speech frame, not only the preceding and following syllable information of the speech frame can be combined, but also the speech frame features corresponding to at least one frame before and after the i-th speech frame in the speech data can be combined, that is, more contextual information is further carried by the preceding and following contextual information of the speech frame to determine the output features of the i-th speech frame at the j-th feature extraction layer.

[0089] Continue to see Figure 3 In this figure, each upper node is connected to three lower nodes, and each node corresponds to a speech frame feature. S2022 can determine the output feature of the i-th speech frame in the j-th feature extraction layer through the speech frame feature, the speech frame feature corresponding to the previous frame of the i-th speech frame in the speech data, and the speech frame feature corresponding to the next frame. It can be understood that the speech frame features corresponding to the first three frames of the current frame and the speech frame features corresponding to the next frame can also be considered. Those skilled in the art can make settings according to actual needs, and this application does not make specific limitations on this.

[0090] As a possible implementation method, the N feature extraction layers included in TDNN can all implement the above S2021-S2022 until all feature extraction layers perform feature extraction on the i-th speech frame, and then determine the syllable probability distribution corresponding to the i-th speech frame.

[0091] S2023: Determine the syllable probability distribution corresponding to the i-th speech frame according to the output features of the i-th speech frame at the j-th feature extraction layer.

[0092] Therefore, after the speech data to be recognized is input into TDNN, the syllable probability distribution corresponding to the i-th speech frame can be obtained, thereby obtaining the syllable probability distribution corresponding to the speech frames included in the speech data, and then determining the speech recognition result corresponding to the speech data.

[0093] As a possible implementation method, the training process of the acoustic model provided in the embodiment of the present application is described below.

[0094] S301: Acquire a speech sample of the same language as the speech data.

[0095] Speech data belonging to the same language have basically the same pronunciation rules and corresponding syllables. For example, the syllables corresponding to Chinese are different from those corresponding to English. Therefore, speech samples can be obtained based on the language to which the speech data belongs to train the acoustic model.

[0096] S302: Using the speech sample as input data of the initial acoustic model, a prediction result is obtained through an initial time delay neural network included in the initial acoustic model.

[0097] The speech sample is input into the initial time delay neural network included in the initial acoustic model to obtain a prediction result. Compared with the speech sample label, the prediction result may include two types, a correct prediction result and an incorrect prediction result. It should be noted that the correct prediction result is that the error between the prediction result and the corresponding sample label is less than a threshold value, and the incorrect prediction result is that the error between the prediction result and the corresponding sample label is greater than or equal to the threshold value. The embodiment of the present application does not specifically limit the threshold value, and those skilled in the art can determine it according to actual needs.

[0098] S303: Determine a loss function according to the prediction result and the sample label of the speech sample.

[0099] The loss function includes guidance weights, which are used to increase the influence of correct prediction results on the recognition path in the initial time-delay neural network and reduce the influence of incorrect prediction results on the recognition path in the initial time-delay neural network.

[0100] As a possible implementation method, the number of speech samples input into the initial TDNN can be multiple. For example, if the speech samples include 100, 20 of them can be used as a batch and input into the initial TDNN 5 times to determine the loss function. If 20 speech samples are input into the initial TDNN, 5 correct prediction results and 15 wrong prediction results are obtained. When determining the loss function, the influence of the recognition path where the 5 correct prediction results are located is increased through the guidance weights, and the influence of the recognition path where the 15 wrong prediction results are located is reduced, so that subsequent training can trust the recognition paths with higher influences more, thereby making the influence of the recognition paths corresponding to the correct prediction results higher and higher, and the influence of the recognition paths corresponding to the wrong prediction results lower and lower, avoiding falling into the local optimum.

[0101] The initial TDNN includes multiple layers of networks, each layer of the network includes multiple nodes, and each node obtains different output features from the previous layer of the network, so the learned content is different, and finally multiple recognition paths are formed. Figure 3 As shown, if the TDNN model is the initial TDNN model, the first node of the second layer can obtain different output features from the first node, the second node and the third node from the left of the first layer, thereby forming three paths, and continuing upward to the last layer, thereby forming multiple recognition paths between nodes from the first layer to the last layer. Different recognition paths may correspond to different final prediction results, and then according to the prediction results and sample labels, different recognition paths have different corresponding influences, such as different recognition paths have different corresponding weights.

[0102] As a possible implementation method, the Lattic-free Maximum Mutual Information (LF-MMI) loss function can be added to the frame-level cross-entropy (CE) loss function, that is, the bi-phone-based LF-MMI model is used to replace the mono-phone-based nnet3 model to guide the error propagation of the initial TDNN model. Compared with only using the CE loss function, this method can make the training faster and more stable, thereby further improving the robustness and coverage of keyword detection through TDNN. The LF-MMI model introduces blanks to absorb uncertain boundaries, that is, to absorb overlapping and meaningless outputs, such as "ni3 hao3 hao3" or "ni3ni3 hao3".

[0103] S304: Adjusting the model parameters of the initial time-delay neural network in the initial acoustic model according to the loss function to obtain the acoustic model.

[0104] The model parameters of the initial time-delay neural network in the initial acoustic model are adjusted according to the loss function, thereby obtaining an acoustic model including the adjusted time-delay neural network.

[0105] As a possible implementation method, if the initial acoustic model includes an initial time delay neural network and an initial decoder, the model parameters of the initial time delay neural network and the initial decoder in the initial acoustic model can be adjusted according to the loss function to obtain an acoustic model including the adjusted time delay neural network and decoder.

[0106] Therefore, through the guidance weight included in the loss function, the influence of the correct prediction result on the recognition path in the initial time-delay neural network is increased, and the influence of the wrong prediction result on the recognition path in the initial time-delay neural network is reduced, so that through differentiated training, the recognition probability of the correct prediction result is made greater, and the other probabilities are as small as possible, which not only improves the recognition rate of the trained acoustic model, but also makes the training of the acoustic model faster and more stable.

[0107] As a possible implementation method, if the number of speech samples used for training is small, it may affect the accuracy of the acoustic model. Therefore, for scenarios where the number of speech samples is insufficient, such as for minority languages, the number of speech samples can be increased based on data augmentation. The details are as follows:

[0108] S3031: Obtain a speech sample to be processed that belongs to the same language as the speech data.

[0109] S3032: Perform data augmentation on the speech sample to be processed to obtain an augmented sample.

[0110] The sample label of the augmented sample is determined based on the sample label of the corresponding speech sample to be processed. The embodiment of the present application does not specifically limit the method of data augmentation, and three methods are used as examples for description below.

[0111] Method 1: Through speech coding, augmented samples with different sampling rates, different channels, and different coding formats are generated for the speech samples to be processed.

[0112] Method 2: By means of speech rate and volume disturbance, a multiplication factor is randomly generated in a preset interval to change the speed and volume of the speech sample to be processed to obtain an augmented sample. The preset interval can be between 0.9 and 1.1, and those skilled in the art can add it according to actual needs, and this application does not make specific limitations on this.

[0113] Method 3: By adding background music and noise, that is, using various background music and noise of various scenes, adding them to relatively clean speech samples to obtain augmented samples.

[0114] It should be noted that augmented samples can be generated by combining one or more of the above three methods.

[0115] S3033: Obtain a speech sample according to the speech sample to be processed and the augmented sample.

[0116] It can be seen from the above technical solutions that due to the great differences between voices, such as the sampling rate, encoding rate, channel differences of audio and video, as well as the speaker's speaking speed, intonation, environmental interference, etc., these will lead to a sharp increase in the difficulty of voice keyword detection. If you want to collect a large amount of voice data of a certain scene and annotate it, and then obtain a large number of voice samples, it is an extremely costly thing, and even in some low-resource, small language scenarios, the number of voice samples is even smaller. Therefore, in scenarios with a small number of voice samples, by expanding the number of voice samples in methods such as S3031-S3033, it is possible to improve the robustness of the acoustic model in different scenarios with limited voice samples.

[0117] When training an acoustic model, each frame needs to have an output as a label. However, after data augmentation, the speech frames of the augmented samples do not correspond to the sample labels. Based on this, the speech frames of the augmented samples can be aligned to the sample labels one by one, that is, a model is used to match the output sequence to the input features one by one, as follows:

[0118] The sample label of the speech sample to be processed corresponding to the augmented sample is used as the pending label of the augmented sample;

[0119] Label alignment is performed on the speech frame of the augmented sample and the pending label to obtain the sample label of the augmented sample.

[0120] In related technologies, this is usually achieved through GMM, which calculates the score of each frame on the potential label and then outputs the final alignment label in combination with dynamic programming. However, with the increase of annotated corpus and computing power, the effect of deep neural network model has gradually surpassed the GMM model. In order to obtain better results, the acoustic model based on ChainModel can be used instead of the GMM model to make the output label sequence more accurate.

[0121] Next, we will combine Figure 5 and Figure 6 For the scenario of keyword recognition, the speech recognition method provided in the embodiment of the present application is explained by taking the acoustic model including TDNN and a decoder as an example.

[0122] See also Figure 5 , which is a schematic diagram of an application scenario embodiment of a speech recognition system provided by an embodiment of the present application. The speech recognition system includes a feature extraction module 501, a TDNN module 502, a decoding module 503 and a vocabulary generation module 504, which are described below respectively.

[0123] The feature extraction module 501 converts the continuous speech data to be recognized into discrete vector representations through a digital signal processing algorithm. These vectors can effectively represent the speech features corresponding to the speech data to be recognized, thereby facilitating subsequent speech tasks.

[0124] As a possible implementation method, fast fixed-point processing can also be performed during speech feature extraction, that is, using integers to simulate floating-point numbers. Specifically, through linear mapping, floating-point values ​​are mapped to integer values, such as int8, int16 or int32, according to the dynamic range of the data. Through testing, this method is nearly twice as fast as the Kaldi tool (an open source speech recognition tool).

[0125] The TDNN module 502 is used to determine the syllable probability distribution corresponding to each speech frame included in the speech data according to the speech features.

[0126] The decoding module 503 determines a speech recognition result for determining whether a keyword is included in the speech data.

[0127] Among them, the decoder and TDNN can be composed as follows Figure 6 The acoustic model shown in FIG. 1 inputs the extracted speech features into TDNN to obtain the syllable probability distribution, and then inputs the syllable probability distribution into the decoder. At the same time, the speech features are data aligned and then input into the decoder. The decoder determines the speech recognition result used to identify whether the speech data includes keywords based on the syllable probability distribution and the matching word list.

[0128] The vocabulary generation module 504 is used to generate syllable sequences corresponding to keywords according to the dictionary.

[0129] Therefore, by cascading the above four modules, the speech recognition system can efficiently detect all the keywords and their start and end positions in the speech data to be recognized.

[0130] In the scenario of keyword recognition, 27.5 hours of speech data to be recognized is used as a test set, and the effects of the two keyword detection methods are listed in Table 2. From the results, it can be seen that compared with the traditional keyword detection system based on keywords / filling, the keyword detection proposed in the embodiment of the present application has obvious improvements in accuracy and coverage, and the overall F1 has increased from 65.99% to 75.57%. This result shows that the speech recognition method provided in the embodiment of the present application can effectively improve the number of keyword recalls, and can also more effectively reduce the number of false alarms of keywords.

[0131] Table 2 Performance comparison experimental results

[0132] method Accuracy Recall F1 Keyword detection based on keywords / stuffing 86.76% 53.24% 65.99% Keyword detection based on the embodiment of the present application 92.00% 64.12% 75.57% Improvement effect +5.23% +10.89% +9.58%

[0133] Among them, F1 is an indicator used in statistics to measure the accuracy of a binary classification model. It takes into account both the precision and recall of the classification model. The F1 score can be regarded as a harmonic average of the model's precision and recall, with a maximum value of 1 and a minimum value of 0.

[0134] In view of the speech recognition method provided in the above embodiment, the embodiment of the present application also provides a speech recognition device.

[0135] See also Figure 7 , which is a schematic diagram of a speech recognition device provided in an embodiment of the present application. Figure 7 As shown, the speech recognition device 700 includes: an acquisition unit 701, a syllable probability distribution determination unit 702 and a speech recognition result determination unit 703;

[0136] The acquisition unit 701 is used to acquire an acoustic model and speech data to be recognized, wherein the acoustic model includes a time-delay neural network, and an output layer of the time-delay neural network includes acoustic modeling units corresponding to a plurality of syllables respectively;

[0137] The syllable probability distribution determining unit 702 is used to use the speech data as input data of the time-delay neural network, and determine the syllable probability distributions corresponding to the speech frames included in the speech data through the time-delay neural network, wherein the syllable probability distributions are used to identify the probabilities that the speech frames correspond to the multiple syllables respectively;

[0138] The speech recognition result determination unit 703 is used to determine the speech recognition result corresponding to the speech data according to the syllable probability distribution.

[0139] As a possible implementation, the acoustic model further includes a decoder, and the speech recognition result determination unit 703 is used to:

[0140] Determining a keyword corresponding to the wake-up scene according to the wake-up scene corresponding to the voice data;

[0141] Determining, by the decoder, a speech recognition result for identifying whether the speech data includes the keyword according to the syllable probability distribution;

[0142] The method further comprises:

[0143] If the voice recognition result indicates that the voice data includes the keyword, the terminal device corresponding to the wake-up scene is woken up.

[0144] As a possible implementation, the device further includes a matching vocabulary building unit, which is used to:

[0145] Constructing a matching word list for the keyword based on syllables;

[0146] The speech recognition result determination unit 703 is used to:

[0147] According to the syllable probability distribution and the matching word list, a speech recognition result for identifying whether the speech data includes the keyword is determined by the decoder.

[0148] As a possible implementation, the time-delay neural network includes N layers of feature extraction layers, j∈N, and for the i-th speech frame in the speech frames included in the speech data, the syllable probability distribution determination unit 702 is used to:

[0149] According to the output features of the j-1th feature extraction layer for the i-th speech frame, the speech frame features of the i-th speech frame are determined by the j-th feature extraction layer;

[0150] Determine the output feature of the i-th speech frame at the j-th feature extraction layer by using the speech frame feature and the speech frame feature corresponding to at least one frame before and after the i-th speech frame in the speech data;

[0151] According to the output features of the i-th speech frame at the j-th feature extraction layer, a syllable probability distribution corresponding to the i-th speech frame is determined.

[0152] As a possible implementation, the matching word list construction unit is used to:

[0153] Determining similar syllables of the syllables included in the keyword according to the pronunciation similarity;

[0154] If the keyword contains a polyphone, determining the polyphone syllable corresponding to the polyphone;

[0155] According to at least one of the similar syllables and the polyphone syllables, and the syllables corresponding to the keyword, a matching word list for the keyword is constructed with the syllable as the division granularity.

[0156] As a possible implementation manner, the device further includes a training unit, which is used to:

[0157] Acquire a speech sample in the same language as the speech data;

[0158] Using the speech sample as input data of an initial acoustic model, obtaining a prediction result through an initial time-delay neural network included in the initial acoustic model;

[0159] Determining a loss function according to the prediction result and the sample label of the speech sample, wherein the loss function includes a guidance weight, and the guidance weight is used to increase the influence of a correct prediction result on a recognition path of the initial time-delay neural network and reduce the influence of an incorrect prediction result on a recognition path of the initial time-delay neural network;

[0160] The model parameters of the initial time-delay neural network in the initial acoustic model are adjusted according to the loss function to obtain the acoustic model.

[0161] As a possible implementation manner, the training unit is used to:

[0162] Acquire a speech sample to be processed that belongs to the same language as the speech data;

[0163] Performing data augmentation on the speech sample to be processed to obtain an augmented sample, wherein the sample label of the augmented sample is determined based on the sample label of the corresponding speech sample to be processed;

[0164] The speech sample is obtained according to the speech sample to be processed and the augmented sample.

[0165] As a possible implementation manner, the training unit is further used for:

[0166] Using the sample label of the speech sample to be processed corresponding to the augmented sample as the pending label of the augmented sample;

[0167] Label alignment processing is performed on the speech frame of the augmented sample and the undetermined label to obtain a sample label of the augmented sample.

[0168] It can be seen from the above technical solution that, for the speech data to be recognized, an acoustic model including a time-delay neural network is obtained. The speech data is used as the input data of the time-delay neural network. Since the output layer of the time-delay neural network includes acoustic modeling units corresponding to multiple syllables, the time-delay neural network can use syllables as the recognition granularity, and obtain the syllable probability distribution corresponding to the speech frames included in the speech data through each acoustic modeling unit of the output layer. In the process of speech feature extraction and transmission to the output layer, the time-delay neural network will carry rich contextual information of the speech frame in the speech data. The contextual information can reflect the relevant information of the syllables before and after the speech frame in the speech data. Therefore, when the output layer is used for syllable recognition, the syllable to which the speech frame belongs can be auxiliary judged based on the pronunciation rules in combination with the information of the syllables before and after the speech frame, so as to output a more accurate syllable probability distribution. Moreover, since a syllable is generally composed of one or more phonemes, it has a higher fault tolerance. Even if there is a pronunciation error of individual phonemes in a syllable in the speech data, the impact on the recognition result of the entire syllable is smaller than that of the phonemes. Therefore, even if the quality of the speech data to be recognized is not high, a more accurate speech recognition result can be obtained based on the syllable probability distribution, which effectively expands the applicable scenarios of speech recognition technology.

[0169] The aforementioned speech recognition device may be a computer device, which may be a server or a terminal device. The aforementioned speech recognition device may be built into a server or a terminal device. The following will introduce the computer device provided in the embodiment of the present application from the perspective of hardware entity. Figure 8 The following is a schematic diagram of the server structure. Fig. 9 Shown is a schematic diagram of the structure of the terminal equipment.

[0170] See also Figure 8 , Figure 8 14 is a schematic diagram of a server structure provided in an embodiment of the present application. The server 1400 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 1422 (for example, one or more processors) and a memory 1432, and one or more storage media 1430 (for example, one or more mass storage devices) storing application programs 1442 or data 1444. Among them, the memory 1432 and the storage medium 1430 may be short-term storage or permanent storage. The program stored in the storage medium 1430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the CPU 1422 may be configured to communicate with the storage medium 1430 to execute a series of instruction operations in the storage medium 1430 on the server 1400.

[0171] The server 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458, and / or one or more operating systems 1441, such as Windows Server 2003 or Windows Server 2003R. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0172] The steps performed by the server in the above embodiment can be based on the Figure 8 The server structure shown.

[0173] The CPU 1422 is used to perform the following steps:

[0174] Acquiring an acoustic model and speech data to be recognized, wherein the acoustic model includes a time-delay neural network, and an output layer of the time-delay neural network includes acoustic modeling units corresponding to a plurality of syllables respectively;

[0175] Using the speech data as input data of the time-delay neural network, determining the syllable probability distribution corresponding to the speech frames included in the speech data through the time-delay neural network, wherein the syllable probability distribution is used to identify the probability that the speech frames correspond to the multiple syllables respectively;

[0176] A speech recognition result corresponding to the speech data is determined according to the syllable probability distribution.

[0177] Optionally, CPU 1422 may also execute the method steps of any specific implementation of the speech recognition method in the embodiments of the present application.

[0178] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. Fig. 9 The block diagram shows a partial structure of a smart phone related to a terminal device provided in an embodiment of the present application, and the smart phone includes: a radio frequency (RF) circuit 1510, a memory 1520, an input unit 1530, a display unit 1540, a sensor 1550, an audio circuit 1560, a wireless fidelity (WiFi) module 1570, a processor 1580, and a power supply 1590. Those skilled in the art can understand that Fig. 9The structure of the smartphone shown in the figure does not constitute a limitation of the smartphone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0179] Combine the following Fig. 9 A detailed introduction to the various components of a smartphone:

[0180] The RF circuit 1510 can be used for receiving and sending signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1580 for processing; in addition, the designed uplink data is sent to the base station. Usually, the RF circuit 1510 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, referred to as LNA), a duplexer, etc. In addition, the RF circuit 1510 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to the Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0181] The memory 1520 can be used to store software programs and modules. The processor 1580 implements various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 1520. The memory 1520 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, a phone book, etc.), etc. In addition, the memory 1520 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0182] The input unit 1530 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the smart phone. Specifically, the input unit 1530 may include a touch panel 1531 and other input devices 1532. The touch panel 1531, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1531 or near the touch panel 1531 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1531 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 1580, and can receive and execute the command sent by the processor 1580. In addition, the touch panel 1531 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1531, the input unit 1530 may also include other input devices 1532. Specifically, other input devices 1532 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0183] The display unit 1540 may be used to display information input by the user or information provided to the user and various menus of the smartphone. The display unit 1540 may include a display panel 1541. Optionally, the display panel 1541 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Further, the touch panel 1531 may cover the display panel 1541. When the touch panel 1531 detects a touch operation on or near it, it is transmitted to the processor 1580 to determine the type of touch event. Subsequently, the processor 1580 provides a corresponding visual output on the display panel 1541 according to the type of touch event. Although in Fig. 9 In the embodiment, the touch panel 1531 and the display panel 1541 are used as two independent components to implement the input and output functions of the smartphone, but in some embodiments, the touch panel 1531 and the display panel 1541 can be integrated to implement the input and output functions of the smartphone.

[0184] The smartphone may also include at least one sensor 1550, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1541 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1541 and / or the backlight when the smartphone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the smartphone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in smartphones, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0185] The audio circuit 1560, the speaker 1561, and the microphone 1562 can provide an audio interface between the user and the smartphone. The audio circuit 1560 can transmit the received audio data to the speaker 1561 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1560 and converted into audio data, and then the audio data is output to the processor 1580 for processing, and then sent to another smartphone through the RF circuit 1510, or the audio data is output to the memory 1520 for further processing.

[0186] WiFi is a short-range wireless transmission technology. Smartphones can help users send and receive emails, browse web pages, and access streaming media through WiFi modules 1570. It provides users with wireless broadband Internet access. Fig. 9 A WiFi module 1570 is shown, but it is understandable that it is not an essential component of the smartphone and can be omitted as needed without changing the essence of the invention.

[0187] The processor 1580 is the control center of the smartphone, which uses various interfaces and lines to connect various parts of the entire smartphone, and executes various functions of the smartphone and processes data by running or executing software programs and / or modules stored in the memory 1520, and calling data stored in the memory 1520. Optionally, the processor 1580 may include one or more processing units; preferably, the processor 1580 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1580.

[0188] The smart phone also includes a power supply 1590 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1580 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption management.

[0189] Although not shown, the smartphone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0190] In the embodiment of the present application, the memory 1520 included in the smart phone can store program codes and transmit the program codes to the processor.

[0191] The processor 1580 included in the smart phone can execute the speech recognition method provided in the above embodiment according to the instructions in the program code.

[0192] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program, wherein the computer program is used to execute the speech recognition method provided in the above embodiment.

[0193] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the speech recognition method provided in various optional implementations of the above aspects.

[0194] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium can be at least one of the following media: read-only memory (English: Read-only Memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0195] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0196] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a technician familiar with the technical field within the technical scope disclosed in the present application should be included in the protection scope of the present application. Moreover, based on the implementation methods provided in the above aspects, the present application can also be further combined to provide more implementation methods. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A speech recognition method, characterized in that: The method comprises: Acquiring an acoustic model and speech data to be recognized, wherein the acoustic model includes a time-delay neural network, and an output layer of the time-delay neural network includes acoustic modeling units corresponding to a plurality of syllables respectively; The speech data is used as input data of the time-delay neural network, and the syllable probability distribution corresponding to the speech frames included in the speech data is determined by the time-delay neural network, wherein the syllable probability distribution is used to identify the probability that the speech frame corresponds to the plurality of syllables, and the time-delay neural network carries the context information of the speech frame in the speech data during speech extraction and transmission to the output layer; Determine a speech recognition result corresponding to the speech data according to the syllable probability distribution; The time-delay neural network includes N layers of feature extraction layers, j∈N, and for the i-th speech frame in the speech frames included in the speech data, the speech data is used as input data of the time-delay neural network, and the syllable probability distribution corresponding to the speech frames included in the speech data is determined by the time-delay neural network, including: According to the output features of the j-1th feature extraction layer for the i-th speech frame, the speech frame features of the i-th speech frame are determined by the j-th feature extraction layer; Determine the output feature of the i-th speech frame at the j-th feature extraction layer by using the speech frame feature and the speech frame feature corresponding to at least one frame before and after the i-th speech frame in the speech data; According to the output features of the i-th speech frame at the j-th feature extraction layer, a syllable probability distribution corresponding to the i-th speech frame is determined.

2. The method according to claim 1, characterized in that: The acoustic model further includes a decoder, and the determining of the speech recognition result corresponding to the speech data according to the syllable probability distribution includes: Determining a keyword corresponding to the wake-up scene according to the wake-up scene corresponding to the voice data; Determining, by the decoder, a speech recognition result for identifying whether the speech data includes the keyword according to the syllable probability distribution; The method further comprises: If the voice recognition result indicates that the voice data includes the keyword, the terminal device corresponding to the wake-up scene is woken up.

3. The method according to claim 2, characterized in that The method further comprises: Constructing a matching word list for the keyword based on syllables; The step of determining, by the decoder, a speech recognition result for identifying whether the speech data includes the keyword according to the syllable probability distribution comprises: According to the syllable probability distribution and the matching word list, a speech recognition result for identifying whether the speech data includes the keyword is determined by the decoder.

4. The method according to claim 3, characterized in that The method further comprises: Determining similar syllables of the syllables included in the keyword according to the pronunciation similarity; If the keyword contains a polyphone, determining the polyphone syllable corresponding to the polyphone; The step of constructing a matching word list for the keyword based on syllables as the granularity includes: According to at least one of the similar syllables and the polyphone syllables, and the syllables corresponding to the keyword, a matching word list for the keyword is constructed with the syllable as the division granularity.

5. The method according to claim 1, characterized in that The method further comprises: Acquire a speech sample in the same language as the speech data; Using the speech sample as input data of an initial acoustic model, obtaining a prediction result through an initial time-delay neural network included in the initial acoustic model; Determining a loss function according to the prediction result and the sample label of the speech sample, wherein the loss function includes a guidance weight, and the guidance weight is used to increase the influence of a correct prediction result on a recognition path of the initial time-delay neural network and reduce the influence of an incorrect prediction result on a recognition path of the initial time-delay neural network; The model parameters of the initial time-delay neural network in the initial acoustic model are adjusted according to the loss function to obtain the acoustic model.

6. The method according to claim 5, characterized in that The acquiring of a speech sample belonging to the same language as the speech data comprises: Acquire a speech sample to be processed that belongs to the same language as the speech data; Performing data augmentation on the speech sample to be processed to obtain an augmented sample, wherein the sample label of the augmented sample is determined based on the sample label of the corresponding speech sample to be processed; The speech sample is obtained according to the speech sample to be processed and the augmented sample.

7. The method according to claim 6, characterized in that The method further comprises: Using the sample label of the speech sample to be processed corresponding to the augmented sample as the pending label of the augmented sample; A label alignment process is performed on the speech frame of the augmented sample and the undetermined label to obtain a sample label of the augmented sample.

8. A speech recognition device, characterized in that: The device comprises an acquisition unit, a syllable probability distribution determination unit and a speech recognition result determination unit; The acquisition unit is used to acquire an acoustic model and speech data to be recognized, wherein the acoustic model includes a time-delay neural network, and an output layer of the time-delay neural network includes acoustic modeling units corresponding to a plurality of syllables respectively; The syllable probability distribution determination unit is used to use the speech data as input data of the time-delay neural network, and determine the syllable probability distributions corresponding to the speech frames included in the speech data through the time-delay neural network, wherein the syllable probability distributions are used to identify the probabilities that the speech frames correspond to the multiple syllables respectively, and the time-delay neural network carries the context information of the speech frames in the speech data during speech extraction and transmission to the output layer; The speech recognition result determination unit is used to determine the speech recognition result corresponding to the speech data according to the syllable probability distribution; The time-delay neural network includes N layers of feature extraction layers, j∈N, and for the i-th speech frame in the speech frames included in the speech data, the syllable probability distribution determination unit is used to: According to the output features of the j-1th feature extraction layer for the i-th speech frame, the speech frame features of the i-th speech frame are determined by the j-th feature extraction layer; Determine the output feature of the i-th speech frame at the j-th feature extraction layer by using the speech frame feature and the speech frame feature corresponding to at least one frame before and after the i-th speech frame in the speech data; According to the output features of the i-th speech frame at the j-th feature extraction layer, a syllable probability distribution corresponding to the i-th speech frame is determined.

9. The device according to claim 8, characterized in that The acoustic model further includes a decoder, and the speech recognition result determination unit is used to: Determining a keyword corresponding to the wake-up scene according to the wake-up scene corresponding to the voice data; Determining, by the decoder, a speech recognition result for identifying whether the speech data includes the keyword according to the syllable probability distribution; If the voice recognition result indicates that the voice data includes the keyword, the terminal device corresponding to the wake-up scene is woken up.

10. The device according to claim 9, characterized in that The device also includes a matching vocabulary building unit, which is used to: Constructing a matching word list for the keyword based on syllables; The speech recognition result determination unit is used to: According to the syllable probability distribution and the matching word list, a speech recognition result for identifying whether the speech data includes the keyword is determined by the decoder.

11. The device according to claim 10, characterized in that The matching word list construction unit is used to: Determining similar syllables of the syllables included in the keyword according to the pronunciation similarity; If the keyword contains a polyphone, determining the polyphone syllable corresponding to the polyphone; According to at least one of the similar syllables and the polyphone syllables, and the syllables corresponding to the keyword, a matching word list for the keyword is constructed with the syllable as the division granularity.

12. The device according to claim 8, characterized in that The device also includes a training unit, which is used to: Acquire a speech sample in the same language as the speech data; Using the speech sample as input data of an initial acoustic model, obtaining a prediction result through an initial time-delay neural network included in the initial acoustic model; Determining a loss function according to the prediction result and the sample label of the speech sample, wherein the loss function includes a guidance weight, and the guidance weight is used to increase the influence of a correct prediction result on a recognition path of the initial time-delay neural network and reduce the influence of an incorrect prediction result on a recognition path of the initial time-delay neural network; The model parameters of the initial time-delay neural network in the initial acoustic model are adjusted according to the loss function to obtain the acoustic model.

13. The device according to claim 12, characterized in that The training unit is used to: Acquire a speech sample to be processed that belongs to the same language as the speech data; Performing data augmentation on the speech sample to be processed to obtain an augmented sample, wherein the sample label of the augmented sample is determined based on the sample label of the corresponding speech sample to be processed; The speech sample is obtained according to the speech sample to be processed and the augmented sample.

14. The device according to claim 13, characterized in that The training unit is further used for: Using the sample label of the speech sample to be processed corresponding to the augmented sample as the pending label of the augmented sample; A label alignment process is performed on the speech frame of the augmented sample and the undetermined label to obtain a sample label of the augmented sample.

15. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the speech recognition method described in any one of claims 1 to 7 according to the instructions in the program code.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the speech recognition method according to any one of claims 1 to 7.

17. A computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the speech recognition method according to any one of claims 1 to 7.