A speech recognition method and apparatus

Abstract

The application provides a speech recognition method and device, which is applied to the field of speech recognition technology in the field of artificial intelligence. The scheme comprises the following steps: obtaining a speech signal to be recognized; a speech recognition model determines a first candidate text based on the speech signal, and determines a candidate phoneme of the speech signal based on the phoneme of the first candidate text; and determining a recognition result of the speech signal based on the first candidate text and the candidate phoneme. The speech recognition model simultaneously outputs the candidate text and the candidate phoneme of the speech signal, so that the recognition result of the speech signal obtained is more accurate, the OOV problem that may occur in the prior art is solved, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of automatic speech recognition (ASR) technology in the field of artificial intelligence (AI), and particularly to a speech recognition method and apparatus. Background Technology

[0002] AI (Artificial Intelligence) is the theory, methods, technology, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] ASR technology has evolved from using traditional model architectures (such as Hidden Markov Models and Deep Neural Networks) to end-to-end model architectures. Taking Chinese as an example, speech recognition typically uses models that model Chinese characters. While these character-based speech recognition models can achieve good recognition results in general scenarios, they often fail to recognize out-of-vocabulary (OOV) words in certain niche scenarios, leading to OOV problems and a poor user experience in those scenarios. Summary of the Invention

[0004] This application provides a speech recognition method and apparatus that utilizes a speech recognition model to simultaneously output candidate characters and candidate phonemes of the speech signal to be recognized, thereby solving the OOV problem in the prior art and improving the user experience.

[0005] In a first aspect, embodiments of this application provide a speech recognition method, the method comprising: acquiring a speech signal to be recognized; a speech recognition model determining a first candidate character based on the speech signal, and determining candidate phonemes of the speech signal based on the phonemes of the first candidate character; and determining the recognition result of the speech signal based on the first candidate character and the candidate phonemes.

[0006] This implementation scheme outputs candidate text and candidate phonemes simultaneously through the speech recognition model, enabling subsequent decision-making based on the first candidate text and candidate phonemes to obtain more accurate recognition results. It solves the OOV problem that easily occurs when the speech recognition model uses only text modeling in the existing technology, and improves the user experience.

[0007] In one possible implementation, the speech recognition model determines a first candidate character based on a speech signal, and determines candidate phonemes of the speech signal based on the phonemes of the first candidate character, including: determining the probability of the first candidate character based on the speech signal, and determining the probability of the phonemes of the first candidate character based on a phoneme mapping matrix, wherein the phoneme mapping matrix indicates the phonemes corresponding to each candidate character; and determining candidate phonemes and their probabilities based on the probability of the first candidate character and the probability of the phonemes of the first candidate character.

[0008] In this implementation scheme, the speech recognition model obtains the probability of candidate phonemes through a phoneme mapping matrix. Compared with using two sub-models to output candidate text and candidate phonemes, this speech recognition model occupies less space and requires less computing power, making its application scope more extensive. It can be applied not only to edge devices in the cloud, such as servers, but also to local edge devices with small storage space and low computing power, such as smartphones with small storage space and low processor performance.

[0009] In one possible implementation, the phoneme mapping matrix includes the probabilities of phonemes corresponding to each candidate character. The phonemes of a candidate character may include one or more phonemes.

[0010] In one possible implementation, the phoneme mapping matrix can obtain the probability of phonemes corresponding to each candidate character based on a character dictionary. For example, when a phoneme belongs to the pronunciation of a candidate character, the probability of that candidate character corresponding to that phoneme is set to 1; otherwise, it is set to 0. The phoneme mapping matrix can also obtain the probability of phonemes corresponding to each candidate character by statistically analyzing corpus samples. For example, when a phoneme belongs to the pronunciation of a candidate character, the frequency of occurrence of that candidate character corresponding to that phoneme in the corpus samples is counted, and the probability of that candidate character corresponding to that phoneme is determined based on the frequency of occurrence of that candidate character and the frequency of occurrence of the candidate character.

[0011] In one possible implementation, the method further includes: compressing the phoneme mapping matrix using sparse row compression.

[0012] This implementation scheme uses sparse row compression to process the phoneme mapping matrix, retaining only the non-zero elements in the matrix, which further reduces the size of the speech recognition model, making the speech recognition model occupy less storage space on the application device and reducing the storage pressure on the speech recognition model pair.

[0013] In one possible implementation, determining the recognition result of the speech signal based on the first candidate character and the candidate phoneme includes: a decoding model decoding the candidate phoneme and its probability, and outputting the second candidate character and its probability; the decoding model is obtained based on a vertical character set; and the recognition result of the speech signal is determined based on the first candidate character and its probability and the second candidate character and its probability.

[0014] This implementation scheme uses a vertical character set to obtain a decoding model to decode candidate phonemes, obtain a second candidate character in the vertical scene, and then makes a decision on the first and second candidate characters, so that the final recognition result can be more accurate. When the user is in the vertical scene, the recognition result obtained in the vertical scene is obtained, thereby avoiding the OOV problem.

[0015] Secondly, embodiments of this application also provide a speech recognition device, which includes: an acquisition unit for acquiring a speech signal to be recognized; a recognition unit for a speech recognition model to determine a first candidate character based on the speech signal and to determine candidate phonemes of the speech signal based on the phonemes of the first candidate character; and a decision unit for determining the recognition result of the speech signal based on the first candidate character and the candidate phonemes.

[0016] In one possible implementation, the recognition unit is specifically used to: determine the probability of a first candidate character based on a speech signal, and determine the probability of the phoneme of the first candidate character based on a phoneme mapping matrix, wherein the phoneme mapping matrix indicates the phoneme corresponding to each candidate character; and determine the candidate phoneme and its probability based on the probability of the first candidate character and the probability of the phoneme of the first candidate character.

[0017] In one possible implementation, the phoneme mapping matrix includes the probability of the phonemes corresponding to each candidate character.

[0018] In one possible implementation, the recognition unit is further configured to: compress the phoneme mapping matrix using sparse row compression.

[0019] In one possible implementation, the decision unit is specifically used for: decoding the candidate phonemes and their probabilities using a decoding model, and outputting the second candidate characters and their probabilities; the decoding model is obtained based on a vertical character set; and based on the first candidate characters and their probabilities and the second candidate characters and their probabilities, the recognition result of the speech signal is determined.

[0020] Thirdly, embodiments of this application also provide a computing device, which includes a processor and a transmission interface, wherein the processor is configured to invoke program instructions stored in a memory to enable the computing device to implement any of the methods described in the first aspect and its optional embodiments.

[0021] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer program code, which, when executed by a computer or processor, enables the computer or processor to implement any of the methods described in the first aspect and its optional embodiments.

[0022] Fifthly, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer or processor, cause the computer or processor to implement any of the methods described in the first aspect and its optional embodiments.

[0023] The apparatus, computer storage medium, or computer program product provided above are all used to execute the methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the corresponding solutions in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of a speech recognition scenario provided in an embodiment of this application;

[0026] Figure 2a This is a schematic diagram of the output result of a speech recognition model using Chinese character modeling provided in an embodiment of this application;

[0027] Figure 2b This is a schematic diagram illustrating the output result of a speech recognition model using Pinyin modeling provided in an embodiment of this application;

[0028] Figure 3 This is a flowchart illustrating a related technology for speech recognition provided in this application;

[0029] Figure 4 This is a schematic diagram of a system architecture for speech recognition provided in an embodiment of this application;

[0030] Figure 5 This is a flowchart of a method for constructing a speech recognition model provided in an embodiment of this application;

[0031] Figure 6 This is a schematic diagram of the structure of an RNN-T model provided in this application;

[0032] Figure 7 and Figure 8This is a schematic diagram of a phoneme mapping matrix provided in an embodiment of this application;

[0033] Figure 9 This is a schematic diagram of the structure of a speech recognition model provided in an embodiment of this application;

[0034] Figure 10 This is a schematic diagram illustrating the acquisition of candidate phonemes based on a phoneme mapping matrix, provided in an embodiment of this application.

[0035] Figure 11 This is a flowchart of a speech recognition method provided in an embodiment of this application;

[0036] Figure 12 This is a flowchart illustrating the speech recognition process performed by the speech recognition model provided in this application embodiment;

[0037] Figure 13 This is a schematic diagram of candidate characters and candidate phonemes for speech recognition provided in an embodiment of this application;

[0038] Figure 14 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this application;

[0039] Figure 15 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0041] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0042] In the description of the embodiments in this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals.

[0043] In addition, the terms "first" and "second" are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the technical features indicated. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. The terms "comprise", "include", "have" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0044] Figure 1 is a schematic diagram of a speech recognition scenario provided by an embodiment of the present application. As Figure 1 shown, in this scenario, the user can send a voice signal of "Call Zhang Jie" to the terminal device 100 (taking a mobile phone as an example). The terminal device 100 collects the voice signal and recognizes the voice signal through a built-in speech recognition model. Then, the terminal device 100 responds to the voice signal and performs the operation of making a call.

[0045] Continuing to refer to Figure 1 , the terminal device 100 can also display the result recognized by the speech recognition model to the user through a display interface, and the user can determine whether the terminal device 100 correctly recognizes the voice signal through the display interface. It can be understood that the scenario of speech recognition is not limited to Figure 1 the application scenario shown. In other examples, the user can also send a voice signal of "Turn on the light" to the terminal device 100. The terminal device 100 recognizes the voice signal and then responds to the voice signal to perform the operation of turning on the light. Optionally, the terminal device 100 can be any one of intelligent devices with speech recognition functions such as mobile phones, tablet computers, laptop computers, smart speakers, and in-vehicle speakers. The embodiment of the present application does not limit the specific category of the terminal device 100.

[0046] In the above Figure 1 shown application scenario, the speech recognition model built in the terminal device 100 generally uses Chinese character modeling. As mentioned in the background technology, in some vertical scenarios, this kind of model will have the OOV problem. The OOV problem existing in the separate modeling of Chinese characters refers to that the speech recognition model of Chinese character modeling is difficult to recognize words (i.e., rare characters) that do not appear in the corpus of the training model. For example, when recognizing the name of a contact "Zhang Jie", since the Chinese character "Jie" does not appear or appears with a low probability in the corpus, the modeling method of Chinese characters makes it difficult for the model to rank the rare character at the top, and thus the Chinese character "Jie" cannot be output, that is, the OOV problem occurs. When the speech recognition model of Chinese character modeling recognizes "Zhang Jie", it may output a result as Figure 2a shown. As can be seen from Figure 2a , the probability of "Zhang" is 0.77, and the probability of "Jie" is 0.95. The model is very likely to output "Zhang Jie" as the final result.

[0047] To address the out-of-voice (OOV) problem inherent in separate Chinese character modeling, the relevant technology employs a two-stage modeling approach to construct the speech recognition model built into the aforementioned terminal device 100. This speech recognition model includes a model for Pinyin modeling and a model for Pinyin-to-Chinese character conversion. The Pinyin-to-Chinese character conversion model is further divided into a general decoding model for Pinyin-to-Chinese character conversion and a vertical decoding model for Pinyin-to-Chinese character conversion.

[0048] The following is combined with Figure 3 Briefly describe the speech recognition process and existing problems of related technologies.

[0049] like Figure 3 As shown, to the Figure 1 Taking the voice signal recognition of "calling Zhang Jie" in the illustrated scenario as an example, the voice signal is first recognized by a Pinyin model, outputting the corresponding Pinyin "da3 / dian4 / hua4 / gei3 / zhang1 / jie2". Then, this Pinyin is input into a general Pinyin-to-Chinese character decoding model and a vertical Pinyin-to-Chinese character decoding model. The general and vertical decoding models perform general domain decoding and vertical domain decoding on the Pinyin, respectively, outputting the Chinese character recognition result in the general domain and the Chinese character sequence in the vertical domain. Finally, a decision is made based on the two Chinese character recognition results to obtain the final recognition result of the voice signal. It is understood that Chinese Pinyin includes syllables and tones. A syllable consists of one or more phonemes, and tones can include four tones: high level tone, rising tone, falling-rising tone, and falling tone. In this embodiment, the tone is represented by a number after the syllable; for example, "1" is high level tone, "2" is rising tone, "3" is falling-rising tone, and "4" is falling tone. In other examples, the tone may also include a neutral tone.

[0050] While the aforementioned technologies can address the Out-of-Voice (OOV) problem inherent in separate Chinese character modeling and the poor readability issue in separate Pinyin modeling, they also introduce new challenges. For example, end-to-end models and general Pinyin-to-Chinese character models require separate training, increasing training difficulty; a joint inference process between sub-models needs to be designed, complicating the speech recognition process; simultaneous decoding of the general and vertical models places high demands on device computational resources; and the resulting speech recognition model, formed by the Pinyin model and the Pinyin-to-Chinese character model, is large, requiring significant memory and computing power from the device, making it unsuitable for space-constrained or low-computing-power devices, especially when the terminal device's memory is limited. The poor readability of separate Pinyin modeling refers to the situation where, when one Pinyin corresponds to multiple Chinese characters, the user may not know which Chinese character the output Pinyin corresponds to in the current context. For example… Figure 2bShows a result of the model output of pinyin modeling. Taking the pinyin "jie2" as an example, the Chinese characters corresponding to this pinyin can be "节", "杰", "结", and "劼". For users, it is impossible to determine which Chinese character the pinyin "jie2" corresponds to from the pinyin.

[0051] Based on the above description, an embodiment of the present application provides a speech recognition model applied on the terminal side for recognizing the speech of users. In application, the speech recognition model can output the first candidate characters and candidate phonemes of the speech to be recognized, so that in the subsequent decision-making process, the recognition result of the speech to be recognized can be determined according to the candidate characters and candidate phonemes, thereby solving the OOV problem that is prone to occur in the model of text alone, and solving the problem of high demand for the space and computing power of the terminal device by the two-stage model in the above related technology.

[0052] Figure 4 Is a schematic diagram of the system architecture for training a speech model provided by an embodiment of the present application. The system architecture is used to train a pre-built model to obtain the speech recognition model in the embodiment of the present application. As Figure 4 Shown, the system architecture 400 includes: a data acquisition device 410 for collecting data for training the model and storing it in the database 420. For example, speech signal samples and a candidate character set can be collected and stored in the database 420; a training device 430 that generates a speech recognition model 101 based on the speech signal samples and the candidate character set maintained in the database 420. The speech recognition model 101 can output candidate characters and candidate phonemes of the speech signal to be recognized. Among them, the training process of the speech recognition model 101 will be introduced below in combination with Figure 5 And will not be elaborated here.

[0053] In one example, the speech recognition model 101 obtained by the training device 430 can be applied in different systems or devices, such as being applied in the Figure 4 Shown execution device 450.

[0054] In Figure 4 The execution device 450 is configured with a calculation module 451 and an I / O interface 452. The I / O interface 452 interacts with external devices for data. Among them, the user can input data to the I / O interface 452 through the client device 440. The input data can include the speech signal to be recognized or the audio feature data of the speech signal to be recognized. The I / O interface 452 is also used to feedback the processed data to the client device 440. The I / O interface 452 can feedback the recognition result of the speech signal to the client device 240, and the user can directly see the recognition result through the interface of the client device 440.

[0055] The calculation module 451 is used to process the input data using the speech recognition model 101, that is, to perform speech recognition on the recognized speech signal. Specifically, the calculation module 451 obtains candidate characters and candidate phonemes of the speech signal to be recognized through the speech recognition model 101, and performs decoding and decision-making based on the candidate characters and candidate phonemes to obtain the recognition result of the speech signal to be recognized; wherein, when the input data is the speech signal to be recognized, the calculation module 451 can first perform frame segmentation and audio feature extraction on the speech signal to be recognized, and then input the audio feature data into the speech recognition model 101.

[0056] The computing module 451 can also call up data, code, etc. in the data storage system 460, and can also store data, instructions, etc. in the storage system 460. For example, when performing speech recognition, it can call the speech framing algorithm and audio feature extraction algorithm stored in the data storage system 460 to process the speech to be recognized, thereby obtaining audio feature data.

[0057] Understandable, attached Figure 4 This is merely a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationships between the devices, components, modules, etc. shown in the diagram do not constitute any limitation. For example, in the attached diagram... Figure 4 In this context, the data storage system 460 is an external memory relative to the execution device 450; however, in other cases, the data storage system 460 may be placed within the execution device 450. Furthermore, in the aforementioned... Figure 1 In the speech recognition scenario shown, both the client device 440 and the execution device 450 can be Figure 1 This is a functional module of the terminal device 100, meaning that the terminal device 100 can simultaneously perform the functions of the client device 440 and the execution device 450. In another scenario, the client device 440 and the execution device 450 can be two different devices. For example, a user can input voice signals on a smartphone (client device 440), which then sends the voice signals to a smart speaker (execution device 450). The smart speaker (execution device 450) recognizes the voice and responds to the user's control.

[0058] Figure 5This application provides a method for establishing a speech recognition model. This method is applied to a training device 430 to construct a speech recognition model that can be applied to an execution device 450, enabling the execution device 450 to recognize speech. The speech recognition model employs an end-to-end model, a type of machine learning model. An end-to-end model specifically uses a single model to model and solve a multi-step or multi-module task, addressing the drawbacks of inconsistent training objectives between steps or modules when using multiple steps or models to solve a complex task. This improves the efficiency of model construction and the accuracy of model application. Optionally, the end-to-end model can be one of a connectionist temporal classification model, an attention-based encoder-decoder model, or a recurrent neural network transducer (RNN-T) model. The following will combine... Figure 5 This section describes how the training device 430 obtains the speech recognition model 101. It should be noted that the embodiments in this application are all illustrated using Chinese characters as the text and Chinese pinyin as the phonemes.

[0059] See Figure 5 Taking the RNN-T model as an example, the method for constructing the speech recognition model in this application embodiment includes the following steps S501-S503.

[0060] In step S501, the RNN-T model is trained using speech signal samples and candidate text.

[0061] In this embodiment, the training device 430 obtains speech signal samples and candidate text sets from the database 420, performs frame segmentation and audio feature extraction on the speech signal samples, obtains audio feature data corresponding to each frame of speech signal samples, and trains an RNN-T model based on the audio feature data and Chinese character tags corresponding to each frame of speech signal samples until the output of the RNN-T model meets the training objective. In one example, when training the model, a loss function can be used to calculate the loss value of the RNN-T, and then the gradient descent method can be used to update the parameters of the RNN-T model until the RNN-T model meets the set loss target value. The loss function can be the Transducer Loss. Optionally, the audio feature data can be the audio power spectrum or Mel power spectrum of the speech signal samples. The RNN-T model can be as follows: Figure 6 The model architecture shown.

[0062] like Figure 6As shown, the RNN-T model consists of three parts: an Encoder, a Predict Network, and a Joint Network. Both the Encoder and the Predict Network are composed of multiple layers of recurrent neural networks (RNNs). The Joint Network is generally composed of several layers of fully connected layers and also includes a Softmax layer. Among them, the audio feature data input to the Encoder is generally the audio power spectrum or Mel power spectrum of the speech signal to be recognized, and the output is a feature vector Fencoder of a fixed dimension; the input to the Predict Network is the decoded text, and the output is a feature vector FpredNet of a fixed dimension; the input to the Joint Network is Fencoder and FpredNet, and the output of the Softmax layer is the predicted value of the next text, and this predicted value includes the probabilities of all candidate texts in the candidate text set. Optionally, the Encoder and PredictNet can also be composed of a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0063] In step S502, a phoneme mapping matrix is constructed.

[0064] In this embodiment, the phoneme mapping matrix includes the probabilities of the phonemes corresponding to each candidate text in the candidate text set.

[0065] In one example, a 0-1 mapping can be used to construct the phoneme mapping matrix. Taking Chinese as an example, the phoneme can be the pinyin of the candidate text, and one candidate text can correspond to one or more pinyins, that is, one candidate text can correspond to one or more phonemes. Specifically, the correspondence table between the candidate text and the phoneme can be obtained through a Chinese character dictionary first. If the phoneme belongs to the pronunciation of the candidate text, the probability that the candidate text corresponds to this phoneme is set to 1, otherwise, the probability that the candidate text corresponds to this phoneme is set to 0; for example, the pinyin of the Chinese character "型" is the phoneme "xing2", then the probability that "型" corresponds to the phoneme "xing2" is 1, and the probability that "型" corresponds to the phoneme "xing4" is 0. Repeat this process until the probabilities of all candidate texts corresponding to all phonemes in the correspondence table are obtained, so as to obtain the phoneme mapping matrix. Refer to Figure 7 , the probability values of the phonemes corresponding to each candidate text in the phoneme mapping matrix can be Figure 7For the values in the shown table, the probability values in the phoneme mapping matrix obtained by this method are only 1 and 0. It can be understood that if a candidate character is a polyphonic character, the probability that the candidate character corresponds to multiple phonemes will be 1. For example Figure 7 the Chinese character "行" in Figure 7 has two pronunciations, "xing2" and "hang2". Therefore, the probabilities of the phonemes "xing2" and "hang2" corresponding to the Chinese character "行" are both 1.

[0066] In one example, a probability statistical mapping can be used to construct a phoneme mapping matrix. Specifically, taking Chinese as an example again, each candidate character appearing in a large number of corpus samples can be phonetically annotated according to a Chinese character dictionary to obtain the phonemes of each candidate character; then, the phonemes of the candidate characters are statistically analyzed to obtain the number of occurrences of each candidate character corresponding to each phoneme, and then the probability of each candidate character corresponding to each phoneme is obtained according to formula (1). As Figure 8 shown, "型" has only one pronunciation, so the phoneme obtained from the corpus sample statistics is 1. Assuming that "行" appears with two pronunciations, "xing2" and "hang2", in the corpus sample, the probabilities can be determined to be 0.7 and 0. !

[0067]

[0068] In formula (1), N(pinyin j |hanzi i ) is the number of occurrences of the phoneme j corresponding to the candidate character i, and P(pinyin j |hanzi i ) is the probability of the phoneme j corresponding to the candidate character i. [[ID=!]]<!

[0069] In one example, in the scenario of recognizing a Chinese-English mixed speech signal, a Chinese-English mixed phoneme mapping matrix can be constructed. Among them, when constructing the English part in the phoneme mapping matrix, each candidate English word in the English word list can be annotated with word piece (word segment) or phoneme, and the word segment or phoneme of each candidate English word is added to the phonemes in the phoneme mapping matrix, and the probability of the word segment or phoneme at the corresponding position in the matrix is set to 1. That is, when the matrix is converted, the word segment or phoneme of English is not converted. In one example, the above two methods for constructing a phoneme mapping matrix can also be used to construct a phoneme mapping matrix in a pure English scenario, or a phoneme mapping matrix in other languages (such as Japanese and Korean).

[0070] In this embodiment, after obtaining the phoneme mapping matrix, compressed sparse row (CSR) can be used to compress the phoneme mapping matrix, making it occupy less space on the edge and requiring less computing power from the edge device. If the phoneme mapping matrix is ​​stored directly in normal density matrix form, it requires a large amount of space on the edge device. For example, assuming there are 4900 candidate characters and 1600 phonemes, the size of the matrix is ​​4900*1600, requiring approximately 7M floating-point numbers to be stored on the edge. Furthermore, if the probability of the candidate character is directly multiplied by the matrix using a general density matrix, the computational cost is approximately 15M floating-point operations per second (FLOPS). However, since the vast majority of values ​​in this matrix are 0, a sparse matrix CSR storage method can be used.

[0071] In one example, matrix operations can also employ CSR+ block operations instead of direct density matrix operations. Taking the total number of candidate characters and phonemes as an example, using sparse storage requires only 100KB of space. Using sparse matrix operations reduces computation to approximately 0.9M FLOPS, a reduction of about 94%, ensuring that the phoneme mapping matrix does not add storage or computational pressure to the edge device.

[0072] In step S503, a speech recognition model is obtained based on the RNN-T model and the phoneme mapping matrix.

[0073] In this embodiment, the matrix operation unit is determined based on the phoneme mapping matrix, and the matrix operation unit is connected to the Softmax layer in the RNN-T model to obtain, as shown below. Figure 9 The speech recognition model shown uses a phoneme mapping matrix to convert the probabilities of candidate characters output by the Softmax layer into probabilities of candidate phonemes. From... Figure 9 As can be seen, the speech recognition model can simultaneously output candidate characters and their probabilities, as well as candidate phonemes and their probabilities, corresponding to the speech signal. The candidate phonemes and their probabilities are used for vertical-category decoding to obtain recognition results within that specific domain.

[0074] In one example, the matrix operation unit calculates the probability of the candidate phonemes corresponding to the speech signal based on the phoneme mapping matrix and the probability of the candidate characters output by the Softmax layer. Figure 10 This is a schematic diagram of the process for obtaining candidate phonemes provided in an embodiment of this application. For example... Figure 10 As shown, assume the audio features x of the signal in frame t are... tThe probabilities of the corresponding candidate Chinese characters are "now: 0.5, gold: 0.3, catty: 0.1, enter: 0.05, refined: 0.02". The probabilities of the candidate Chinese characters are multiplied by the corresponding phoneme mapping matrix to obtain the probabilities of the corresponding candidate phonemes "jin1: 0.9, jin4: 0.05, jing1: 0.02". Taking "jin1: 0.9" as an example, the probability 0.9 can be obtained according to formula (2). Substituting the relevant data into the right side of the equal sign in formula (2), we can get "0.5×1 + 0.3×1 + 0.1×1", and then calculate the probability 0.9.

[0075] P(pinyin j |x t )=∑P(pinyin j |hanzi i )*P(hanzi i |x t ) (2)

[0076] In formula (2), x t is the t-th frame of speech signal, P(hanzi i |x t ) is the probability of the candidate Chinese character i corresponding to x t , and P(pinyin j |x t ) is the probability of the phoneme j corresponding to x t .

[0077] In the above method embodiment of the present application, a speech recognition model obtained by connecting a phoneme mapping matrix after the Softmax layer of the RNN-T model can output the candidate Chinese characters corresponding to the speech signal, their probabilities, the candidate phonemes corresponding to the speech signal, and their probabilities at the same time. Compared with the prior art, the speech recognition model of the embodiment of the present application only adds matrix operations on the basis of RNN-T. The overall model occupies a small amount of memory space on the terminal side, has low computing power requirements for the terminal side, and is applicable to a wider range of terminal devices. The phoneme mapping matrix is stored on the terminal side after CSR processing, so that the phoneme mapping matrix does not add too much storage pressure and computing pressure to the terminal device. In addition, setting the matrix operation unit obtained according to the phoneme mapping matrix after the RNN-T model does not affect the decoding result of the probability of the candidate Chinese character. Moreover, the training of the phoneme mapping matrix and the RNN-T model is completely decoupled. Only by inserting the phoneme mapping matrix after the RNN-T model, it can be used immediately, which is convenient to use.

[0078] Generally speaking, the speech recognition model established by combining the phoneme mapping matrix and the RNN-T model in the embodiment of the present application meets the requirements of the terminal side for the speech recognition model, such as small occupied space, high accuracy, low computing power, and support for vertical hot word recognition.

[0079] Figure 11 This application provides a speech recognition method, which is applied in an execution device 450. It is understood that the client device 440 and the execution device 450 can be modules within the aforementioned terminal device 100; that is, the functions implemented by the client device 440 and the execution device 450 can be uniformly implemented by the terminal device 100. For example... Figure 11 As shown, the method includes the following steps S1101-S1103.

[0080] In step S1101, the speech signal to be recognized is acquired.

[0081] In this embodiment, the user, within a certain range of the client device 440, emits a voice signal to control the execution device 450. The client device 440 can acquire the voice signal emitted by the user by collecting voice signals from the surrounding environment through its built-in microphone. After acquiring the voice signal, the client device 440 sends it to the execution device 450. The execution device 450 performs framing and audio feature extraction on the voice signal to obtain audio feature data corresponding to each frame. Optionally, the audio feature data may include the power spectrum and Mel-index power spectrum of the voice signal. It can be understood that after acquiring the voice signal, the client device 440 can also process the voice signal to obtain audio feature data and then send the audio feature data to the execution device 450.

[0082] In step S1102, the speech recognition model determines the first candidate characters and candidate phonemes based on the speech signal.

[0083] In this embodiment, the execution device 450 inputs the audio feature data corresponding to each frame of the speech signal into the speech recognition model for speech recognition. The speech recognition model in this step can be the one described above. Figure 4 The construction method shown is used, and will not be elaborated further here.

[0084] In one example, when the execution device 450 recognizes the voice signal, such as Figure 12As shown, the audio feature data (Mel power spectrum) of the speech signal is input into the Encoder, and the word vectors corresponding to the candidate characters obtained in the previous decoding are input into the Predict Network. The Encoder outputs the Encoder feature vector, and the Predict Network outputs the PredictNetwork feature vector. The Joint Network outputs the currently decoded candidate characters and their probabilities based on the input Encoder feature vector and PredictNetwork feature vector. This process is repeated until all candidate characters in the speech signal are obtained. During the first decoding, specific symbols can be input into the Predict Network to obtain the probabilities of the first decoded candidate characters. Optionally, when using the Beam search algorithm for decoding, the Joint Network can use the N candidate characters with the highest probabilities among all candidate characters as the current decoding output; when using a greedy algorithm, the Joint Network can use the candidate character with the highest probability among all candidate characters as the current decoding output.

[0085] In this embodiment, after obtaining the probability of the candidate characters, the matrix operation unit determines the candidate phonemes of the speech signal based on the phoneme mapping matrix. In one example, the phonemes of the candidate characters are determined in the phoneme mapping matrix to obtain the probability of the phonemes of the candidate characters. Then, the probability of the candidate characters and the probability of the phonemes of the candidate characters are multiplied to obtain the probability of the candidate phonemes (i.e., the calculation is performed according to the aforementioned formula (2)). Finally, the phoneme with the highest probability or N phonemes among the candidate phonemes are output. Figure 13 As shown, taking the user's voice signal "Call Zhang Jie" as an example, when the speech recognition model recognizes the voice signal, it outputs the first candidate text "Call Zhang Jie" and candidate phonemes, where the probability of the candidate phonemes can be... Figure 13 The probabilities are shown in the table below.

[0086] In step S1103, the recognition result of the speech signal is determined based on the first candidate character and the candidate phoneme.

[0087] In this embodiment, a decoding model is used to decode the probability of candidate phonemes to obtain a second candidate character. Then, a decision is made between the first and second candidate characters to determine the final recognition result of the speech signal.

[0088] In one example, see below. Figure 13The decoding model decodes the candidate phonemes and their probabilities to obtain the second candidate text "call Zhang Jie". Optionally, the vertical decoding model can be a weighted finite state transducer (WFST) model, which can be obtained based on a user-defined vertical text set, which includes user hot words in a preset vertical scenario.

[0089] In one example, when determining the recognition result of a speech signal, a decision can be made based on the first candidate text and its probability, and the second candidate text and its probability. Taking the user's speech signal "Call Zhang Jie" as an example, the final recognition result of the current speech signal can be determined based on the first candidate text "Call Zhang Jie" and its probability, and the second candidate text "Call Zhang Jie" and its probability.

[0090] Based on the above Figure 11 The method embodiment shown in this application also provides a speech recognition device. This speech recognition device is applied in an execution device 450 to achieve... Figure 11 The steps in the method embodiment shown are as follows. It can be understood that the client device 440 and the execution device 450 can be a module of the aforementioned terminal device 100, that is, the functions implemented by the client device 440 and the execution device 450 can be uniformly implemented by the terminal device 100.

[0091] like Figure 14 As shown, the speech recognition device 200 may include an acquisition unit 201, a recognition unit 202, and a decision unit 203. It can be understood that... Figure 13 This application only provides an exemplary division of the speech recognition device 200 in its embodiments. In other possible embodiments, the speech recognition device 200 may include other numbers of functional modules for implementation. Figure 10 The steps are shown.

[0092] The acquisition unit 201 is used to acquire the collected speech signal from the client device 440, thereby obtaining the speech signal to be recognized, and to perform frame segmentation and audio feature extraction on the speech signal to obtain the audio feature data corresponding to each frame of the speech signal; wherein the client device 440 can acquire the speech signal through a built-in microphone; the recognition unit 202 is used to obtain the first candidate characters and candidate phonemes of the speech signal using a speech recognition model; the decision unit 203 is used to decode the candidate phonemes to obtain the second candidate characters of the speech signal, and to determine the recognition result of the speech signal from the first candidate characters and the second candidate characters. It can be understood that the client device 440 and the execution device 450 can be a module of the aforementioned terminal device 100, that is, the functions implemented by the client device 440 and the execution device 450 can be uniformly implemented by the terminal device 100.

[0093] The specific implementation process of each of the above units can be found in the description of the invention and the foregoing method embodiments, and will not be repeated here.

[0094] This application embodiment also provides a computing device, which may include one or more processors and a transmission interface; wherein, the one or more processors are configured to invoke computer instructions stored in memory, so that the computing device can perform the functions implemented by the aforementioned execution device 450 and training device 440, that is, execute this application. Figure 4 The method steps in the embodiment of the speech recognition model construction method shown, or Figure 11 The illustrated speech recognition method embodiment includes steps and optional steps. Optionally, the terminal device 100 may include an NPU coprocessor mounted on the main CPU for implementing... Figure 4 and Figure 11 The method steps in the illustrated method embodiment.

[0095] Figure 15 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. This computing device can be used to implement the functions of the aforementioned execution device 450 and training device 440. For example... Figure 14 As shown, the computing device includes: at least one central processing unit (CPU), memory (the type of memory may include, for example, static random-access memory (SRAM) and read-only memory (ROM), a microcontroller unit (MCU), a wireless local area network (WLAN) subsystem, a bus, a transmission interface, etc. Although Figure 15 As not shown in the diagram, the computing device may also include an application processor (AP), an NPU, and other dedicated processors, as well as other subsystems such as a power management subsystem, a clock management subsystem, and a power consumption management subsystem.

[0096] The various parts of the computing device are coupled together by connectors. For example, connectors include various interfaces, transmission lines or buses, etc. These interfaces are usually electrical communication interfaces, but may also be mechanical interfaces or other forms of interfaces. This embodiment does not limit them.

[0097] Optionally, the CPU can be a single-core or multi-core processor; alternatively, the CPU can be a processor group consisting of multiple processors, which are coupled to each other through one or more buses. In one optional case, the CPU implements the speech recognition method or a speech recognition model construction method as described in the foregoing method embodiments by calling program instructions stored in the aforementioned on-chip memory or off-chip memory. In one optional case, the CPU and MCU jointly implement any of the speech recognition methods or speech recognition model construction methods as described in the foregoing method embodiments. In one optional case, the AP or other dedicated processor implements a speech recognition method or a speech recognition model construction method as described in the foregoing method embodiments by calling program instructions stored in the aforementioned on-chip memory or off-chip memory. The ROM or SRAM memory can also be used to store a phoneme mapping matrix.

[0098] This transmission interface can serve as the interface for the processor chip to receive and send data. It typically includes multiple interfaces; optionally, it may include an Inter-Integrated Circuit (I2C) interface, a Serial Peripheral Interface (SPI), a Universal Asynchronous Receiver-Transmitter (UART) interface, or a General-Purpose Input / Output (GPIO) interface. It should be understood that these interfaces can achieve different functions by multiplexing the same physical interface.

[0099] In one alternative, the transmission interface may also include High Definition Multimedia Interface (HDMI), V-By-One interface, Embedded Display Port (eDP), Mobile Industry Processor Interface (MIPI), or Display Port (DP), etc.

[0100] In one alternative scenario, the aforementioned components are integrated onto the same chip; in another alternative scenario, the memory can be a separate chip.

[0101] A WLAN subsystem may include, for example, radio frequency circuitry and a baseband.

[0102] The chip involved in this application embodiment is a system manufactured on the same semiconductor substrate using integrated circuit technology, also called a semiconductor chip. It can be a collection of integrated circuits formed on a substrate (usually a semiconductor material such as silicon) using integrated circuit technology, and its outer layer is typically encapsulated by semiconductor packaging materials. The integrated circuit can include various functional devices, each including logic gates, metal-oxide-semiconductor (MOS) transistors, bipolar transistors, or diodes, and may also include other components such as capacitors, resistors, or inductors. Each functional device can operate independently or under the action of necessary driving software, and can realize various functions such as communication, computation, or storage.

[0103] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0104] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0105] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

Claims

1. A speech recognition method, characterized in that, The method includes: Acquire the speech signal to be recognized; The speech recognition model determines a first candidate character based on the speech signal, and determines candidate phonemes of the speech signal based on the phonemes of the first candidate character; Based on the first candidate character and the candidate phoneme, the recognition result of the speech signal is determined; The speech recognition model determines a first candidate character based on the speech signal, and determines candidate phonemes of the speech signal based on the phonemes of the first candidate character, including: The probability of the first candidate character is determined based on the speech signal, and the probability of the phoneme of the first candidate character is determined based on the phoneme mapping matrix, wherein the phoneme mapping matrix indicates the phoneme corresponding to each candidate character. Based on the probability of the first candidate character and the probability of the phoneme of the first candidate character, the candidate phoneme and its probability are determined.

2. The method according to claim 1, characterized in that, The phoneme mapping matrix includes the probability of the phonemes corresponding to each candidate character.

3. The method according to claim 1, characterized in that, The method further includes: The phoneme mapping matrix is ​​compressed using sparse row compression.

4. The method according to any one of claims 1-3, characterized in that, The step of determining the recognition result of the speech signal based on the first candidate character and the candidate phoneme includes: The decoding model decodes the candidate phonemes and their probabilities, and outputs the second candidate character and its probability; the decoding model is obtained based on a vertical character set; Based on the first candidate character and its probability, and the second candidate character and its probability, the recognition result of the speech signal is determined.

5. A voice recognition device, characterized in that, The device includes: Acquisition unit, used to acquire the speech signal to be recognized; The recognition unit is used for the speech recognition model to determine a first candidate character based on the speech signal, and to determine candidate phonemes of the speech signal based on the phonemes of the first candidate character; A decision unit is configured to determine the recognition result of the speech signal based on the first candidate character and the candidate phoneme. Specifically, the identification unit is used for: The probability of the first candidate character is determined based on the speech signal, and the probability of the phoneme of the first candidate character is determined based on the phoneme mapping matrix, wherein the phoneme mapping matrix indicates the phoneme corresponding to each candidate character. Based on the probability of the first candidate character and the probability of the phoneme of the first candidate character, the candidate phoneme and its probability are determined.

6. The apparatus according to claim 5, characterized in that, The phoneme mapping matrix includes the probability of the phonemes corresponding to each candidate character.

7. The apparatus according to claim 5, characterized in that, The identification unit is also used for: The phoneme mapping matrix is ​​compressed using sparse row compression.

8. The apparatus according to any one of claims 5-7, characterized in that, The decision-making unit is specifically used for: The decoding model decodes the candidate phonemes and their probabilities, and outputs the second candidate character and its probability; the decoding model is obtained based on a vertical character set; Based on the first candidate character and its probability, and the second candidate character and its probability, the recognition result of the speech signal is determined.

9. A computing device, characterized in that, The computing device includes a processor and a transmission interface, the processor being configured to invoke program instructions stored in a memory such that the computing device implements the method of any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code, which, when executed by a computer or processor, enables the computer or processor to implement the method described in any one of claims 1-4.

11. A computer program product containing instructions, characterized in that, When the instructions are executed by a computer or processor, the computer or processor performs the method according to any one of claims 1-4.

Images