Speech recognition method and server

Abstract

The application relates to a speech recognition method and a server. The method comprises the following steps: obtaining a to-be-recognized speech signal; recognizing each frame of the to-be-recognized speech signal according to an acoustic model of each language, and respectively outputting corresponding language phonemes and prediction probabilities; wherein the acoustic model of each language is respectively constructed according to shared hidden layer training; sequentially traversing a sentence decoding graph and a multi-language slot decoding graph connected with each other to obtain a corresponding path; wherein the sentence decoding graph is used for decoding phonemes entering a non-slot, and the slot decoding graph is used for decoding phonemes entering a slot; when it is determined that the path passes through the multi-language slot decoding graph in the speech decoding graph, screening the path according to the prediction probabilities of the language phonemes corresponding to each language, and determining the text information corresponding to the target path as a speech recognition result. The scheme provided by the application can accurately recognize mixed multi-language speech information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, in particular to a speech recognition method and a server. BACKGROUND

[0002] With the development of intelligent vehicles, vehicle-mounted intelligent devices have speech recognition functions, so that users can directly interact with the vehicle machine through speech, which helps users to drive safely. For different users using different languages, the speech recognition system carried in the vehicle-mounted intelligent device needs to correctly distinguish different languages and accurately recognize speech.

[0003] In related technologies, for speech signals of different languages, the language needs to be classified according to a language classification model first, and then the language recognition system corresponding to the language category is used for text recognition. The current speech recognition technology still ultimately recognizes speech for a language, and cannot simultaneously recognize multiple languages, which affects the recognition efficiency. SUMMARY

[0004] To solve or partially solve the problems in related technologies, the present application provides a speech recognition method and a server, which can simultaneously and accurately recognize mixed speech information of multiple languages.

[0005] The first aspect of the present application provides a speech recognition method, which comprises: obtaining a to-be-recognized speech signal; identifying each frame of the to-be-recognized speech signal according to an acoustic model of each language, and respectively outputting corresponding language phonemes and prediction probabilities; wherein the acoustic model of each language is respectively constructed according to shared hidden layers; sequentially traversing a sentence decoding graph and a multi-language slot decoding graph connected to each other to obtain a corresponding path; wherein the sentence decoding graph is used for decoding phonemes entering a non-slot, and the slot decoding graph is used for decoding phonemes entering a slot; when it is determined that the path passes through the multi-language slot decoding graph in the speech decoding graph, screening the path according to the prediction probabilities of the language phonemes corresponding to each language and the path, and determining the text information corresponding to the target path as a speech recognition result. Such a design, on the one hand, can simplify the model architecture of the speech recognition system by using the acoustic model of the shared hidden layer, and can extend the acoustic model without limitation of language, meeting the multi-language recognition needs of users; on the other hand, by distinguishing slots and non-slots in different decoding graphs and decoding, the recognition efficiency and accuracy are improved.

[0006] In the speech recognition method, the acoustic models of the languages are respectively constructed according to a common hidden layer, including: according to the types of the languages, sample speech signals and corresponding phoneme labels are respectively acquired as training data; the acoustic models are commonly trained according to the training data corresponding to the languages, wherein different acoustic models have a shared hidden layer and independent softmax layers; loss functions of the corresponding acoustic models are respectively determined according to the output probabilities of the softmax layers, and the parameters of the shared hidden layer are iterated through back propagation to obtain the trained acoustic models. The acoustic models obtained by training the shared hidden layer and the independent softmax layers have a more simplified model architecture, and can accurately recognize phonemes of different languages; meanwhile, the newly added language can be trained based on the historically trained shared hidden layer, without affecting the recognition effect of the voice model of the historical language, and is easy to expand.

[0007] In the speech recognition method, the mutually connected sentence decoding graph and multi-language slot decoding graph are constructed in the following manner: a sentence decoding graph of a preset language is constructed, and single-language slot decoding graphs corresponding to different languages are respectively constructed according to different preset slot types; the single-language slot decoding graphs of the same slot type are respectively merged to form corresponding multi-language slot decoding graphs; and the sentence decoding graph is connected with the slot decoding graphs of the languages. The sentence decoding graph and the multi-language slot decoding graph constructed in this manner can have a more simplified network structure while decoding the preset language and the multi-languages.

[0008] In the speech recognition method, the merging of the single-language slot decoding graphs of the same slot type to form the corresponding multi-language slot decoding graphs includes: for the same slot type, nodes and connection edges connected between the nodes are respectively set in each single-language slot decoding graph, wherein the start nodes in each single-language slot decoding graph have the same number, and the remaining nodes have different numbers; and the start nodes in each single-language slot decoding graph are merged to combine each single-language slot decoding graph into a slot decoding graph corresponding to the slot type. By setting the same start nodes, each single-language slot decoding graph can be quickly combined into a slot decoding graph, improving the construction efficiency.

[0009] In the speech recognition method, before the start nodes in each single-language slot decoding graph are merged, the nodes in the single-language slot decoding graphs of different languages are respectively set with corresponding bias values. By setting the bias values, confusion of phoneme codes of different languages is avoided, ensuring that the network structure of the slot decoding graph is clear and the path is easy to identify.

[0010] In the speech recognition method, the connecting the sentence decoding graph with the slot decoding graph of the multiple languages respectively to form the speech decoding graph comprises: setting the slot mark according to the slot type; and connecting the sentence decoding graph with the slot decoding graph of the multiple languages respectively according to the slot mark to form the speech decoding graph. The slot mark is set to connect the slot decoding graph of the multiple languages with the sentence decoding graph, the network structure is simplified, and it is easy to determine whether the path enters the corresponding slot decoding graph according to the slot mark, thereby improving the output efficiency of the recognition result.

[0011] In the speech recognition method, the filtering the path according to the prediction probability of the language phoneme corresponding to each language and determining the text information corresponding to the target path as the speech recognition result comprises: obtaining the accumulated preset score of each path; weighting and counting the accumulated preset score of each path and the prediction probability of the input language phoneme to filter out the path with the maximum count result as the target path; and obtaining the corresponding text information as the speech recognition result according to the target path. The prediction probability output by the acoustic model and the accumulated preset score of each path are used to filter the target path, so that the target path can be obtained more reliably and accurately, and the accuracy of the recognition result is improved.

[0012] In the speech recognition method, the method further comprises: when it is determined that the path does not pass through the slot decoding graph, filtering the path according to the prediction probability of the language phoneme corresponding to the preset language and determining the text information corresponding to the path as the speech recognition result. When it is determined that the path does not pass through the slot decoding graph, the prediction probability output by the acoustic model of the preset language is used to filter the path, and the prediction probability output by the acoustic model of other languages does not need to be concerned, so that the recognition efficiency is improved and the reliability of the recognition result is ensured.

[0013] The second aspect of the present application provides a server, comprising:

[0014] The acquisition module is configured to acquire a speech signal to be recognized.

[0015] The acoustic processing module is configured to identify each frame of the speech signal to be recognized according to an acoustic model of each language to output corresponding language phonemes and prediction probabilities.

[0016] The decoding module is configured to sequentially traverse a sentence decoding graph and a slot decoding graph of multiple languages connected to each other by taking each language phoneme as input to obtain a corresponding path. The sentence decoding graph is used to decode a phoneme entering a non-slot, and the slot decoding graph is used to decode a phoneme entering a slot.

[0017] The output module is configured to filter the path according to the predicted probability of the language phoneme corresponding to each language and the path, and determine the text information corresponding to the target path as the speech recognition result when determining that the path passes through the slot decoding graph.

[0018] The third aspect of the present application provides a server, comprising:

[0019] a processor; and

[0020] a memory having stored executable codes, which, when executed by the processor, cause the processor to perform the method as described above.

[0021] The fourth aspect of the present application provides a computer readable storage medium having stored executable codes, which, when executed by a processor of an electronic device, cause the processor to perform the method as described above.

[0022] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0023] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which like reference characters refer to like parts throughout the figures, and wherein:

[0024] Figure 1 is a flowchart of a speech recognition method shown in the present application;

[0025] Figure 2 is a flowchart of an acoustic model construction method shown in the present application;

[0026] Figure 3 is an acoustic model construction method corresponding to Figure 2 ;

[0027] Figure 4 is a flowchart of a speech decoding graph construction method shown in the present application;

[0028] Figure 5 is a speech decoding graph construction method corresponding to Figure 4 ;

[0029] Figure 6 is a flowchart of a speech recognition method shown in the present application;

[0030] Figure 7 is a structural diagram of a server shown in the present application;

[0031] Figure 8is a structural schematic diagram of a server shown in the present application. DETAILED DESCRIPTION

[0032] Embodiments of the present application will be described in more detail with reference to the drawings. Although embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0033] The terms used in the present application are merely for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein means and includes any or all possible combinations of one or more associated listed items.

[0034] It should be understood that although the terms "first", "third", "third" and the like are used to describe various information in the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, the first information can also be referred to as the third information, and similarly, the third information can also be referred to as the first information without departing from the scope of the present application. Therefore, the features defined with "first", "third" can explicitly or implicitly include one or more features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0035] In the related art, the language recognition system can generally only recognize the speech of a single language, and a speech recognition model needs to be established for each language. When mixed languages appear in the speech, recognition cannot be performed.

[0036] To solve the above problems, the present application provides a speech recognition method which can accurately recognize mixed multi-language speech information and meet user needs.

[0037] The technical solutions of the present application will be described in detail below with reference to the drawings.

[0038] Figure 1 is a flowchart of a speech recognition method shown in the present application.

[0039] Referring to Figure 1 , a speech recognition method shown in the present application comprises:

[0040] S110, obtaining a speech signal to be recognized.

[0041] The execution subject of the voice recognition method can be a server.

[0042] In this step, if the execution subject is a server, the voice signal sent by the terminal can be received to obtain the voice data to be recognized.

[0043] In this step, the language used by the voice signal to be recognized can be one or multiple languages.

[0044] In S120, each frame of the voice signal to be recognized is recognized according to the acoustic model of each language, and the corresponding language phoneme and prediction probability are output respectively; wherein the acoustic model of each language is constructed according to the shared hidden layer training respectively.

[0045] In this step, in order to cope with the situation of multiple languages, the corresponding acoustic model is set and pre-trained according to different languages. The number of language types is not limited, and can be increased or decreased according to the actual language demand, so as to correspondingly increase or decrease the corresponding acoustic model.

[0046] Further, the voice signal to be recognized is recognized according to the pre-trained acoustic model, the deep neural network in each acoustic model has mutually shared hidden layer parameters, that is, the feature extraction and score calculation of the voice signal to be recognized are performed through the shared hidden layer, and the prediction probability of the language phoneme corresponding to each language is output. The deep learning neural network used by the acoustic model, such as TDNN (Time-Delay Neural Network) network, LSTM (Long short-term memory) network, etc., is not limited here.

[0047] In the construction process of each acoustic model of the present application, the training data respectively adopts sample voice signals of the corresponding language, and the hidden layer of the same deep neural network is used for training iteration to realize the effect of sharing the hidden layer. It can be understood that the deep neural network generally includes multiple hidden layers for deep extraction of acoustic features. The method of the present application shares the same hidden layer parameters for each language acoustic model, i.e. without setting separate hidden layer parameters for different languages, thereby simplifying the structure of each acoustic model and easily realizing the expansion requirement of different language recognition. In addition, in the process of human-computer interaction by voice of the user, the same voice expressed by the user can involve one language or multiple languages. The acoustic models of different languages can simultaneously predict the input voice signal to be recognized, thereby simultaneously outputting the phonemes involved in each language and the corresponding prediction probability of each phoneme. It can be understood that different languages have their own phonemes, so each acoustic model predicts and outputs the phonemes of the corresponding language and the corresponding prediction probability. Such a design does not need to use a language classification model to classify the language in advance according to the voice signal, but each acoustic model simultaneously predicts the phonemes of the corresponding language, thereby improving the recognition efficiency and accuracy.

[0048] S130, sequentially traversing the connected sentence decoding graph and multi-language slot decoding graph with each language phoneme as input to obtain the corresponding path; wherein the sentence decoding graph is used for decoding the phonemes entering the non-slot, and the slot decoding graph is used for decoding the phonemes entering the slot.

[0049] Wherein, the sentence can be regarded as a sentence structure combined with slots and non-slots, the slots can be at specified positions or any positions of the beginning, middle or end of the sentence, and the number and type of slots are not limited. The sentence decoding graph is a voice decoding graph using a preset language and located in the non-slot of the sentence. The multi-language slot decoding graph is a voice decoding graph using multiple languages and located in the slot of the sentence. The sentence decoding graph and the multi-language slot decoding graph are connected, so that the same input phoneme can be gradually traversed on the sentence decoding graph and the multi-language slot decoding graph in time sequence to obtain the corresponding path.

[0050] Based on the frequency of use of different languages, a certain language can be set as a preset language. That is, the preset language is also included in the multi-language, and the preset language also has a corresponding acoustic model. Alternatively, according to different regions, the language with higher frequency of use can be set as the preset language. For example, in the European region, English with wider popularity can be set as the preset language, which is only an example. By setting the words located in the non-slot to use the preset language and the words located in the slot to use multiple languages, when the voice signal appears mixed with multiple languages, the voice recognition problem of mixed languages can be regarded as the voice recognition problem of the preset language and the voice recognition problem of multiple languages.

[0051] Further, by pre-setting the slot type of the slot, the slot decoding graph can be used to decode the phonemes of the language according to the slot type. Such a design can map the phonemes of the language that do not conform to the slot type to the non-slot, and the phonemes of the language in the non-slot are decoded using the sentence decoding graph; the phonemes of the language that conform to the slot type are mapped to the slot, and the phonemes of the language in the slot are decoded using the multi-language slot decoding graph. For example, the slot type can be POI (point of interest), SONG (song name), NAME (name), etc. The corresponding slot type can be set according to the semantics and part of speech of the word, and this is only an example. Based on this, the speech decoding graph of the method consists of two parts, the sentence decoding graph and the multi-language slot decoding graph, which avoids using a multi-language network structure for all speech decoding graphs, achieves the effect of simplifying the network structure of the speech decoding graph, and improves the decoding efficiency.

[0052] Further, in related technologies, a system for speech recognition for a single language generally has a pre-constructed speech decoding graph, for example, the speech decoding graph can be a WFST graph (weighted finite state transducer). Specifically, for example, the speech decoding graph can be a total WFST graph obtained by merging four WFST graphs converted from an HMM (Hidden Markov Model) model, a language model, a pronunciation dictionary, and a context-dependent, and this is only an example. Alternatively, the structure of the sentence decoding graph and the multi-language slot decoding graph of the present application can adopt the WFST graph network structure described above.

[0053] After each acoustic model outputs the phonemes of the language of the speech signal to be recognized, each phoneme of the language is input into the network structure of the sentence decoding graph and the multi-language slot decoding graph for traversal, and a path from each phoneme of the language to a word sequence in the decoding graph is obtained, that is, all paths are obtained.

[0054] S140, when it is determined that the path passes through the slot decoding graph in the speech decoding graph, the path is filtered according to the prediction probability of the phonemes of the language corresponding to each language, and the text information corresponding to the path is determined as the speech recognition result.

[0055] When any one of the paths in the speech decoding graph passes through the multi-language slot decoding graph, it indicates that the corresponding phonemes of the language match the slot type and are decoded by the slot decoding graph. It can be understood that when all paths do not pass through the slot decoding graph, it indicates that the path does not enter the slot decoding graph and only passes through the sentence decoding graph, and the phonemes of the language are decoded by the sentence decoding graph.

[0056] When it is determined that at least one path passes through the slot decoding graph of the phonetic decoding graph, the target path can be obtained by screening the paths in S130 in combination with the preset rule and the prediction probability of each language phoneme obtained in S120. For example, the cumulative preset weight of each path in the phonetic decoding graph is obtained, the cumulative preset weight of each path is weighted and summed with the prediction probability of the input language phoneme corresponding to each path, and the counts of each weighted sum are compared to determine the path with the maximum count as the target path. The text information corresponding to the target path is the text information corresponding to the speech signal to be recognized, thereby obtaining the speech recognition result of the speech signal to be recognized.

[0057] As can be seen from the example, the speech recognition method of the present application can use each acoustic model with a common hidden layer trained and constructed in advance to simultaneously perform speech recognition on a single preset language or a mixed language speech signal, without introducing an additional language classification model, thereby improving data processing efficiency. In addition, according to the pre-constructed sentence decoding graph and the multi-language slot decoding graph, the slot decoding graph decodes the language phonemes entering the slot, and the sentence decoding graph decodes the language phonemes not in the slot. When it is determined that at least one path passes through the slot decoding graph, the prediction probability output by the acoustic model and each path can be combined and screened at the same time to more comprehensively and accurately screen a more optimal target path, and then the text information in the target path is taken as the recognition result. Such a design, on the one hand, uses an acoustic model with a shared hidden layer to simplify the model architecture of the speech recognition system, and can extend the acoustic model without limitation of language, thereby meeting the multi-language recognition needs of users. On the other hand, by distinguishing the decoding graphs of the slot and the non-slot, the recognition efficiency and accuracy are improved.

[0058] Figure 2 is a flowchart of the acoustic model construction method shown in the present application, Figure 3 is Figure 2 the acoustic model construction method of

[0059] Referring to Figure 2 and Figure 3 , the acoustic model construction method shown in the present application includes:

[0060] S210, according to the types of each language, respectively obtaining sample speech signals and corresponding real phoneme labels as training data.

[0061] It can be understood that the construction method does not limit the types and quantities of languages. Accordingly, according to the determined language, the sample speech signals and the corresponding true phonemes of each language are selected as labels, and the corresponding training data of each language is obtained. In different languages, the amount of data contained in the respective training data is set independently.

[0062] Further, sample speech signals corresponding to each language and different from the training data can also be used as verification data corresponding to each acoustic model, so as to subsequently verify the trained acoustic model of each language to evaluate the prediction accuracy of the acoustic model of each language.

[0063] S220, training each acoustic model according to the training data corresponding to each language, wherein different acoustic models have shared hidden layers and independent softmax layers.

[0064] In this step, each type of language has a corresponding acoustic model, and each acoustic model has the same hidden layer of the preset deep neural network, i.e. without setting independent hidden layers for each acoustic model, so that the acoustic models have shared hidden layers.

[0065] The preset deep neural network used by the acoustic model can be a currently known neural network or a self-developed neural network, such as a TDNN network, an LSTM network, a CNN (Convolutional Neural Networks) network, etc., which are only used as examples and are not limited.

[0066] Further, the deep neural network of each acoustic model includes an input layer, multiple shared hidden layers, and an output layer. Optionally, each acoustic model has a respective input layer or a shared input layer for inputting the training data of each language. Each acoustic model has shared hidden layers, and each shared hidden layer performs acoustic feature extraction and score calculation on the input sample speech signal layer by layer. Each acoustic model can have a respective output layer for outputting the score of each phoneme corresponding to the mapped language of each sample speech signal. The number of shared hidden layers is not limited, each shared hidden layer has a respective corresponding parameter, such as a weight parameter and a bias parameter, and the initial parameter value can be randomly set. Optionally, each acoustic model shares all hidden layers.

[0067] Further, the acoustic models of each language respectively have a corresponding Softmax layer, and each Softmax layer is used to connect the output layer; the corresponding Softmax layer is used to convert the scores of each phoneme mapped by the sample speech signal into a probability value. Specifically, taking three languages as an example, the three acoustic models have a shared hidden layer, and each has an independent Softmax layer, and the Softmax layer of each language is connected with the shared hidden layer. For example, the languages include German, English, and Norwegian, etc., wherein a certain sample speech signal of one language passes through the input layer, the multi-layer shared hidden layer, and the output layer, and the output layer outputs the predicted scores of each phoneme of the sample speech signal belonging to the language, and the scores of each phoneme enter the Softmax layer of the corresponding language for normalization processing, and then each score is converted into a corresponding probability. For a single language, the Softmax layer takes the probability of the real phoneme of the corresponding sample speech signal as the output probability.

[0068] S230, respectively determine the loss function of the corresponding acoustic model according to the output probability of each softmax layer, and iteratively update the parameters of the shared hidden layer through back propagation to obtain the corresponding trained acoustic model.

[0069] For the acoustic model of each language, the loss value (loss) of the output probability of the acoustic model based on the current sample speech signal can be calculated by a preset loss function, for example, the loss value can be the negative value of the logarithmic conversion of the output probability, so as to determine the loss degree of each acoustic model in the current round of training.

[0070] For ease of understanding, for example, when the languages include English, German, and French, the training data of the three languages is obtained respectively. The three languages have respective acoustic models, and each acoustic model has a shared hidden layer, and optionally, a shared input layer. In training, the training data of all languages is input into the shared hidden layer through the input layer for layer-by-layer acoustic feature extraction and score calculation. Finally, the scores of each sample speech signal in the English training data are output from the output layer of the English acoustic model, and each score is input into the corresponding English Softmax layer for normalization processing to obtain the output probability corresponding to the real phoneme, and then the output probability of each sample speech signal is converted into a corresponding loss value; similarly, the output probability corresponding to the real phoneme output from the German Softmax layer of each German training data can be obtained, and the loss value corresponding to each output probability can be obtained; and the output probability corresponding to the real phoneme output from the French Softmax layer of each French training data can be obtained, and the loss value corresponding to each output probability can be obtained.

[0071] With the goal of reducing the loss value of each acoustic model, through multiple rounds of training, each round of training respectively synchronously returns the loss value corresponding to each acoustic model to the shared hidden layer according to the back propagation algorithm, so that the parameters of each shared hidden layer are respectively iteratively adjusted according to the loss value of the previous round. For example, the parameters of each layer of shared hidden layer are iteratively adjusted according to the gradient descent method. When the loss value of the prediction result of each acoustic model becomes smaller and smaller until it tends to be constant, that is, the loss value converges, it indicates that the acoustic model of each language is trained. That is, the trained acoustic model of each language has a shared hidden layer, and the parameters of each layer of shared hidden layer are iteratively obtained based on the loss value of each acoustic model, so that the trained hidden layer can be synchronized to be applicable to phoneme prediction of each language.

[0072] Further, the trained corresponding acoustic model is evaluated according to the verification data of each language. According to the input sample voice signal, the corresponding phonemes and probabilities are outputted when they are acceptable, which indicates that the acoustic model based on the shared hidden layer has good training results.

[0073] It can be understood that if a new language is added based on the already trained acoustic model of each language, the training data of the new language and the historical language training data can be iteratively used to update the parameters of the shared hidden layer based on the original parameters of the shared hidden layer. Of course, the historical language training data can be randomly selected, and it is not necessary to use all the historical training data for the training of the new language. The training can be efficiently completed, the acoustic model of the new language can be accurately predicted based on the updated shared hidden layer, and the acoustic model of the historical language can still be accurately predicted based on the updated shared hidden layer.

[0074] From the example, it can be seen that the number of languages is not limited, and the acoustic model in the present application is trained based on the shared hidden layer, so that the parameters of the shared hidden layer after iterative adjustment can be synchronized to be applicable to different languages, and the structure of all acoustic models can be simplified through the shared hidden layer. At the same time, the newly added language can be trained based on the historical trained shared hidden layer, which will not affect the recognition effect of the voice model of the historical language, and is easy to expand. In practical application, different languages can be recognized synchronously, without the need for classification through a language classification model, and has a wide application range and higher recognition efficiency.

[0075] Figure 4 is a flowchart of the construction method of the speech decoding graph shown in the present application, Figure 5 is a schematic diagram of a single language slot decoding graph before and after merging, which is used to specifically introduce the construction process of the speech decoding graph used in the speech recognition method of the present application.

[0076] Referring to Figure 4 and Figure 5The application shows a method for constructing a speech decoding graph, which comprises:

[0077] S310, constructing a sentence decoding graph of a preset language, and constructing a single-language slot decoding graph corresponding to each language according to each preset slot type.

[0078] In this step, a language is selected as the preset language in advance. When the preset language is different, the corresponding sentence decoding graph is different according to the language characteristics of the different preset languages.

[0079] In addition, according to the number of slot types, a slot decoding graph corresponding to each language for each slot type is constructed, i.e. a single-language slot decoding graph. For example, when the slot types include POI, NAME, SONG, etc. 3 types, if the number of languages is 20, the single-language slot decoding graph constructed is 3*20, i.e. a total of 60 single-language slot decoding graphs. That is, each single-language slot decoding graph represents the decoding graph when each language corresponds to the current slot type. It can be understood that, compared with covering all words of each language to form a decoding graph of the language, the method of the application only needs to form a decoding graph for the words associated with the slot type, greatly reducing the network structure of the single-language slot decoding graph, reducing the path during decoding, and at the same time meeting the user's speech habits, improving the speech recognition efficiency.

[0080] Preferably, each single-language slot decoding graph and the sentence decoding graph are WFST graphs.

[0081] S320, merging the single-language slot decoding graphs of the same slot type to form a slot decoding graph corresponding to multiple languages.

[0082] Optionally, for the same slot type, each single-language slot decoding graph is provided with nodes and connection edges connected between the nodes, wherein the start nodes in each single-language slot decoding graph have the same number, and the remaining nodes have different numbers; the start nodes in each single-language slot decoding graph are merged, and each single-language slot decoding graph is merged into a slot decoding graph corresponding to the slot type.

[0083] For ease of understanding, as shown in FIG. a and FIG. b in the drawings, Figure 5 for example, FIG. a is a partial schematic view of a single-language slot decoding graph of English, FIG. b is a partial schematic view of a single-language slot decoding graph of German, and FIG. c is a multi-language slot decoding graph after merging the single-language slot decoding graphs of English and German.

[0084] In order to distinguish the paths in the multilingual slot decoding graph, different numbers are set in each single language slot decoding graph except the start node, so as to distinguish the connection edges through which the paths pass. In the graph, the circle represents the node, and the number in the circle is the number corresponding to the current node. The line between different nodes or the same node is the connection edge. Further, the numbers of the start nodes in different single language slot decoding graphs can be 0, and there is only one start node in each single language slot decoding graph. The dashed circle represents the end node, and the end node can be the same as the start node. The value on each connection edge is represented in the data structure of [input value: output value / score]. For example, 1:0 / 3 in Figure a represents that the score obtained when the input value is 1 and the output value is 0 is 3.

[0085] The representative meaning of each value of the input value and the output value is determined according to the lower WFST in the speech decoding graph. For example, in the four WFST graphs respectively converted from the HMM (Hidden Markov Model), the language model, the pronunciation dictionary, and the context, the numbers in the input value and the output value can be the codes of the pronunciation attribute, the phoneme, the word or the word sequence, and different value labels represent different contents. For example, the input value is 2 and the output value is 1, which means that the input code is 2 phoneme and the predicted output code is 1 phoneme. For each language, different codes can be used to represent the corresponding pronunciation attribute, phoneme, word or word sequence. Preferably, in order to deal with the situation that silence or noise may occur in each frame of speech signal, the same code representing silence or noise can be set in all single language slot decoding graphs. For example, 10 codes such as 1-10 are reserved for different frequencies of silence or noise. Further, the score on the connection edge can represent the confidence degree of the corresponding output value from the input value. The score on each connection edge can be set in advance according to different path directions.

[0086] Since the start nodes in different single language slot decoding graphs have the same number, for example, 0, the start nodes can be merged nodes, and different single language slot decoding graphs can be merged to obtain one slot decoding graph in Figure c according to the two single language slot decoding graphs in Figures a and b.

[0087] It can be understood that the number of merged multilingual slot decoding graphs is the same as the number of slot types, for example, the above-mentioned slot types have 3, and 20 single-language slot decoding graphs corresponding to POI are merged into 1 corresponding POI slot decoding graph, 20 single-language slot decoding graphs corresponding to NAME are merged into 1 corresponding NAME slot decoding graph, and 20 single-language slot decoding graphs corresponding to SONG are merged into 1 corresponding SONG slot decoding graph, and so on. According to the increase or decrease of the slot type, the corresponding multilingual slot decoding graph is obtained by merging.

[0088] Preferably, before merging the single-language slot decoding graphs, a corresponding offset value can also be set for the nodes in the single-language slot decoding graphs of different languages respectively. That is to say, each language has a corresponding offset value. For example, the offset value of English is 0, the offset value of German is 100000, the offset value of French is 200000, and so on. It needs to be understood that, in general, in a single-language speech decoding graph, each language has its own phonemes and corresponding codes, and the number of codes corresponds to the actual number of phonemes, for example, 0-100000. When the phonemes of different languages are independent of each other, the same set of codes is used, for example, all 0-100000. In this application, since the number of languages is not limited to one, in order to distinguish the codes of the phonemes of different languages, a corresponding offset value is set for each code based on the original code, so that the codes of the phonemes of different languages with the same original code are distinguished. Therefore, before merging the single-language slot decoding graphs, the offset value of each code can be superimposed, so that the same codes of different languages have significant differences after superimposing the offset value, thereby avoiding confusion of the codes on the path in the merged multilingual slot decoding graph. Further, for the empty nodes in each single-language slot decoding graph, that is, the nodes whose output value of the connection edge is 0, these empty nodes have no actual meaning and do not need to superimpose the offset value when merging the decoding graphs.

[0089] For example, as shown in FIG. b of Figure 5 , the offset value of German represented by FIG. b is 100000, and the code 1123 on one connection edge, assuming that the English represented by FIG. a also has a phoneme with a code of 1123, in order to distinguish, before merging the decoding graphs, 1123 in FIG. b is superimposed with an offset of 100000 to obtain the corresponding value 101123 in FIG. c, so as to form a significant gap with the English 1123. For another example, as shown in FIG. a, since FIG. a represents a single-language slot decoding graph of English, the preset offset value of English is 0, so the code value in the merged graph c remains unchanged.

[0090] S330, respectively connecting the sentence decoding graph with the multilingual slot decoding graph.

[0091] In order to obtain the speech decoding graph, in this step, the obtained slot decoding graph of each multi-lingual is connected with the sentence decoding graph respectively. Specifically, the corresponding slot mark can be set according to the slot type; according to the slot mark, the sentence decoding graph is connected with the corresponding multi-lingual slot decoding graph to form the speech decoding graph.

[0092] For example, for three slot types POI, SONG, NAME, different slot marks are set in the speech decoding graph, such as #nonterm:poi, #nonterm:song, #nonterm:name, the sentence decoding graph is connected with the multi-lingual slot decoding graph of POI through the slot mark #nonterm:poi, the sentence decoding graph is connected with the multi-lingual slot decoding graph of SONG through the slot mark #nonterm:song, and so on, so as to connect the sentence decoding graph with the multi-lingual slot decoding graph corresponding to each slot type to form a speech decoding graph that can be used to recognize a mixed-lingual speech signal.

[0093] As can be seen from the example, the speech decoding graph construction method of the present application can construct a corresponding speech decoding graph according to the preset language sentence of the slot filling structure, and by first merging each monolingual slot decoding graph of the same slot type into a corresponding multi-lingual slot decoding graph, and finally merging each slot decoding graph with the sentence decoding graph, a more compact speech decoding graph can be constructed to cope with the decoding recognition of a preset language or a mixed language speech signal, thereby improving the decoding efficiency.

[0094] Figure 6 is a flowchart of the speech recognition method shown in the present application, which is used to recognize a speech signal with a single language (i.e. a preset language) and a multi-lingual according to the model constructed by the above-mentioned acoustic model construction method and the speech decoding graph constructed by the above-mentioned speech decoding graph construction method.

[0095] Referring to Figure 6 , a speech recognition method shown in the present application comprises:

[0096] S410, obtaining a to-be-recognized speech signal.

[0097] This step is the same as S110, which is not repeated here.

[0098] S420, recognizing each frame of the to-be-recognized speech signal according to the acoustic model of each language, wherein the shared hidden layer of each acoustic model respectively outputs the corresponding language phoneme and the prediction probability of each acoustic model.

[0099] It can be understood that the same frame of speech signal is extracted and output by the hidden layer common to each acoustic model, i.e., the shared hidden layer, and is output from the Softmax layer corresponding to the language, respectively, and the output belongs to each phoneme in each language and the corresponding prediction probability.

[0100] S430, input each language phoneme into the inter-connected sentence decoding graph and the multi-language slot decoding graph, and obtain all paths and the cumulative preset score corresponding to each path.

[0101] It can be understood that the language phoneme is a phoneme in each language, and each language phoneme is input as an input, and each path in the decoding graph can be obtained. It can be understood that each path at least passes through the sentence decoding graph. When the sentence decoding graph and the multi-language slot decoding graph are WFST graphs, each connection edge on each path has a corresponding preset score, and the cumulative preset score corresponding to the current path can be obtained by accumulating the preset score on each connection edge.

[0102] S440, when the path passes through the multi-language slot decoding graph, the target path corresponding to the text information is determined as the speech recognition result according to the prediction probability of each language phoneme and each path.

[0103] If the path passes through the multi-language slot decoding graph, it means that the word in the speech signal matches the slot type. The language of the speech signal corresponding to the slot can be a preset language or a certain other language containing the preset language. In order to improve the accuracy of the recognition result, the prediction probability of each language phoneme and the cumulative preset score of each path are screened when decoding the speech.

[0104] Specifically, the cumulative preset score of each path in the sentence decoding graph and the multi-language slot decoding graph is obtained; the cumulative preset score of each path is weighted and counted with the prediction probability of the corresponding input language phoneme, and the path corresponding to the maximum counting result is selected as the target path; and the corresponding text information is obtained as the speech recognition result according to the target path. That is, for each path, the score on the corresponding connection edge can be accumulated to obtain the cumulative preset score. The input corresponding to each path, i.e., the prediction probability of the language phoneme, is weighted and summed with the cumulative preset score of the path, and the weighted sum count of all paths and the prediction probability of the respective input is obtained. The counts after the weighted sum are compared, and the path with the maximum count is selected as the target path, and the word sequence output by the target path is the speech recognition result of the speech to be recognized. Alternatively, in addition to the above-mentioned weighted sum, other calculation methods can also be used, which are not limited herein.

[0105] S450, when determining that the path does not pass through the multi-lingual slot decoding graph, filtering the paths according to the prediction probability corresponding to the language phoneme of the preset language and determining the text information corresponding to the target path as the speech recognition result.

[0106] That is, according to all the paths passing through the speech decoding graph, none of the paths passes through the multi-lingual slot decoding graph, indicating that the speech signal does not match the slot in the preset language sentence pattern, and only matches the non-slot, that is, only the preset language is used in the speech signal. Therefore, only the language phoneme and probability output by the Softmax layer of the acoustic model of the preset language are considered, and the language phonemes and probabilities output by the Softmax layer of other languages do not need to be considered.

[0107] In this step, it can be regarded as single language speech decoding. For example, the preset language is English, and each language phoneme output by the Softmax layer of English is taken as input to obtain the corresponding accumulated preset scores in each path in the sentence decoding graph. After weighting and summing the prediction probability of each input language phoneme and the accumulated preset scores of the corresponding path, respectively, the counts after weighting and summing are compared, and the path with the largest count is selected as the target path. The word sequence output by the target path is the speech recognition result of the speech to be recognized. Alternatively, in addition to the above-mentioned weighted summing, other calculation methods can also be used, which are not limited herein.

[0108] As can be seen from the example, the speech recognition method of the present application can determine the method of selecting the recognition result according to whether the path enters the slot through the pre-constructed sentence decoding graph and multi-lingual slot decoding graph, so as to reliably cope with the situation of multiple languages and ensure the accuracy of the recognition result.

[0109] Corresponding to the above-mentioned application function implementation method embodiment, the present application also provides a server and a corresponding embodiment.

[0110] Figure 7 is a structural schematic diagram of the server shown in the present application.

[0111] Referring to Figure 7 The server shown in the present application includes an acquisition module 710, an acoustic processing module 720, a decoding module 730 and an output module 740. Wherein:

[0112] The acquisition module is used to acquire the speech signal to be recognized.

[0113] The acoustic processing module 720 is used to identify each frame of the speech signal to be recognized according to the acoustic model of each language, and output the corresponding language phoneme and prediction probability respectively; wherein the acoustic model of each language is constructed according to the shared hidden layer training respectively.

[0114] The decoding module 730 is configured to sequentially traverse the inter-connected sentence decoding graph and the multi-lingual slot decoding graph with language-specific phonemes as input to obtain corresponding paths; wherein the sentence decoding graph is used to decode phonemes entering non-slots, and the slot decoding graph is used to decode phonemes entering slots.

[0115] The output module 740 is configured to, when determining that the path passes through the multi-lingual slot decoding graph, filter the paths according to the prediction probabilities of the language-specific phonemes corresponding to each language and the prediction probabilities of the paths in the acoustic model, and determine the text information corresponding to the target path as the speech recognition result.

[0116] Further, the acoustic processing module 720 loads each acoustic model composed of a shared hidden layer and an independent softmax layer, and each acoustic model outputs the corresponding prediction probabilities of multi-lingual phonemes and phonemes of each language. The acoustic model is constructed according to the construction method of the acoustic model, and will not be described here.

[0117] The sentence decoding graph and the multi-lingual slot decoding graph loaded in the decoding module 730 can be WFST graphs. The sentence decoding graph and the multi-lingual slot decoding graph are constructed according to the construction method of the acoustic model, and will not be described here.

[0118] Further, when determining that the path passes through the multi-lingual slot decoding graph, the output module 740 is configured to obtain the cumulative preset scores of each path obtained by the decoding module 730; weight and count the cumulative preset scores of each path and the prediction probabilities of the corresponding input language-specific phonemes, respectively, to filter out the path with the largest count result as the target path; and obtain the corresponding text information as the speech recognition result according to the target path.

[0119] Further, the output module 740 can also be configured to, when determining that the path passes through the multi-lingual slot decoding graph, filter the paths according to the prediction probabilities of the language-specific phonemes corresponding to each language and the prediction probabilities of the paths, and determine the text information corresponding to the target path as the speech recognition result.

[0120] The server of the present application can simplify the model architecture of the speech recognition system based on the acoustic model using the shared hidden layer, and can extend the acoustic model without being limited by the language, meeting the multi-lingual recognition needs of users; on the other hand, by decoding according to the decoding graph that distinguishes slots and non-slots, the recognition efficiency is improved.

[0121] As to the server in the above-mentioned embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0122] Figure 8 is a structural schematic diagram of the server shown in the present application.

[0123] Referring to Figure 8 The server 1000 includes a memory 1010 and a processor 1020.

[0124] The processor 1020 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can be any conventional processor.

[0125] The memory 1010 can include various types of storage units, such as a system memory, a read-only memory (ROM), and a permanent storage device. Among them, the ROM can store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device can be a read-write storage device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a mass storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (such as a floppy disk, an optical drive). The system memory can be a read-write storage device or a volatile read-write storage device, such as a dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during runtime. In addition, the memory 1010 can include a combination of any computer readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 1010 can include a read and / or write removable storage device, such as a compact disc (CD), a read-only digital versatile disc (such as DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a min SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer readable storage medium does not include a carrier wave and a transient electronic signal transmitted through a wireless or wired transmission.

[0126] The memory 1010 stores executable code, which, when processed by the processor 1020, can cause the processor 1020 to perform part or all of the above-mentioned methods.

[0127] Furthermore, the method according to the present application can also be implemented as a computer program or a computer program product, which comprises computer program code instructions for performing some or all of the steps of the above-mentioned method according to the present application.

[0128] Alternatively, the present application can also be implemented as a computer readable storage medium (or a non-transitory machine readable storage medium or a machine readable storage medium) having stored thereon executable codes (or computer programs or computer instruction codes) which, when executed by a processor of an electronic device (or a server, etc.), cause the processor to perform some or all of the steps of the above-mentioned method according to the present application.

[0129] The above has described the embodiments of the present application, the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, practical application, or improvement to the technology in the market, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims

1. A speech recognition method, characterized in that, include: Acquire the speech signal to be recognized; The speech signal to be identified in each frame is identified according to the acoustic model of each language, and the corresponding language phonemes and predicted probabilities are output respectively; wherein, the acoustic model of each language is constructed based on a shared hidden layer training. Using the phonemes of each language as input, the system sequentially traverses the interconnected sentence decoding graph and the slot decoding graph of the multilingual languages ​​to obtain the corresponding paths. The sentence decoding graph is used to decode phonemes that enter non-slots and belong to a preset language, while the slot decoding graph is used to decode phonemes that enter slots and belong to any language in the multilingual languages. The sentence structure is considered a combination of slots and non-slots, and the slots are located at a specified or arbitrary position at the beginning, middle, or end of the sentence. When it is determined that the path passes through the slot decoding map of the multilingual languages, the path is filtered according to the predicted probability of the phonemes of each language and the path itself, and the text information corresponding to the path is determined as the speech recognition result.

2. The method according to claim 1, characterized in that, The acoustic models for each of the aforementioned languages ​​are constructed based on training using a common hidden layer, including: Based on the language, sample speech signals and corresponding phoneme labels are obtained as training data. Each acoustic model is trained together based on training data corresponding to each language. Different acoustic models share a hidden layer and have their own independent softmax layer. The loss function of the corresponding acoustic model is determined based on the output probability of each softmax layer, and the parameters of the shared hidden layer are iterated through backpropagation to obtain the corresponding trained acoustic model.

3. The method according to claim 1, characterized in that, The interconnected sentence decoding graphs and multilingual slot decoding graphs are constructed in the following manner: Construct sentence structure decoding diagrams for preset languages, and construct single-language slot decoding diagrams for each language according to each preset slot type; The monolingual slot decoding maps of the same slot type are merged to form the corresponding multilingual slot decoding maps; The sentence structure decoding diagram is connected to the slot decoding diagram of the multilingual languages ​​respectively.

4. The method according to claim 3, characterized in that, The step of merging the monolingual slot decoding maps of the same slot type to form corresponding multilingual slot decoding maps includes: For the same slot type, nodes and connecting edges are set for each monolingual slot decoding graph, wherein the starting nodes in each monolingual slot decoding graph have the same number, and the remaining nodes have different numbers. The starting nodes in each of the monolingual slot decoding diagrams are merged to form a slot decoding diagram corresponding to the slot type.

5. The method according to claim 4, characterized in that, Before merging the starting nodes in each of the monolingual slot decoding graphs, the method further includes: Set corresponding bias values ​​for the nodes in the monolingual slot decoding graphs for different languages.

6. The method according to claim 3, characterized in that, The step of connecting the sentence structure decoding map to the slot decoding map of the multilingual languages ​​includes: Set the corresponding slot flag according to the slot type; Based on the slot markers, the sentence decoding diagrams are connected to the corresponding multilingual slot decoding diagrams.

7. The method according to claim 1, characterized in that, Based on the predicted probabilities of the phonemes corresponding to each language and the path, the text information corresponding to the path is determined as the speech recognition result, including: Obtain the cumulative preset score for each path in the sentence structure decoding map and the multilingual slot decoding map; The cumulative preset score of each path is weighted and counted with the predicted probability of the corresponding input language phoneme, and the path with the largest count result is selected as the target path. Based on the target path, the corresponding text information is obtained as the speech recognition result.

8. The method according to claim 1, characterized in that, The method further includes: When it is determined that the path does not pass through the slot decoding map, the path is filtered according to the predicted probability of the language phonemes of the preset language and the path itself, and the text information corresponding to the target path is determined as the speech recognition result.

9. A server, characterized in that, include: The acquisition module is used to acquire the speech signal to be recognized; An acoustic processing module is used to identify the speech signal to be identified in each frame according to the acoustic model of each language, and output the corresponding language phonemes and prediction probabilities respectively; wherein, the acoustic model of each language is constructed by training based on a shared hidden layer. The decoding module is used to sequentially traverse the interconnected sentence decoding graph and the multilingual slot decoding graph, taking the phonemes of each of the aforementioned languages ​​as input, to obtain the corresponding paths; wherein, the sentence decoding graph is used to decode phonemes that enter non-slots and belong to a preset language, and the slot decoding graph is used to decode phonemes that enter slots and belong to any language in the multilingual system; the sentence is considered as a sentence structure combining slots and non-slots, and the slots are located at a specified position or any position at the beginning, middle, or end of the sentence; The output module is used to filter the target path as the speech recognition result by using the predicted probability of the phonemes of each language and the path when it is determined that the path passes through the slot decoding map of the multilingual languages.

10. A server, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-8.

11. A computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-8.

Images