Speech recognition method, device, electronic device, storage medium, and computer program

Abstract

The present disclosure provides a speech recognition method, device, equipment and medium, which relates to the field of artificial intelligence, particularly to the technical fields of speech recognition, natural language processing and deep learning. A specific implementation of the method includes: processing the speech data to be recognized and a first text segment obtained by the recognition using an acoustic model to obtain an acoustic probability of each of a plurality of candidate text segments; processing the first text segment using a first language sub-model to obtain an initial language probability of each of the plurality of candidate text segments; processing the first text segment using a constraint sub-model to obtain an extensible relationship of each of the plurality of candidate text segments to the first text segment; adjusting the initial language probability of the candidate text segment according to the extensible relationship, and obtaining a first language probability of each of the plurality of candidate text segments; and identifying a target text segment in the plurality of candidate text segments according to the first language probability and the acoustic probability.

Description

[Technical Field]

[0001] This application claims priority from a Chinese patent application bearing application number 202211064891.8, filed on September 1, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the field of artificial intelligence, particularly to technical fields such as speech recognition, natural language processing, and deep learning, and in particular to speech recognition methods, devices, electronic devices, and storage media. and computer programs Regarding. [Background technology]

[0003] With the development of computer and network technologies, deep learning technology has been widely used in many fields. For example, an acoustic model built based on deep learning technology can be used to recognize speech and convert collected speech into text. Summary of the Invention

[0004] The present disclosure relates to a speech recognition method, an apparatus, electronic device , memory medium and computer programs The purpose is to provide the following.

[0005] According to one aspect of the present disclosure, there is provided a speech recognition method, comprising: processing the speech data to be recognized and the first text segment obtained by the recognition using the acoustic model to obtain acoustic probabilities for each of a plurality of candidate text segments; processing the first text segment using a first language sub-model in the language model to obtain initial language probabilities for each of a plurality of candidate text segments; processing the first text segment using a constraint sub-model in the language model to obtain an extensible relationship to the first text segment in each of a plurality of candidate text segments; adjusting initial language probabilities of the candidate text segments based on the extensible relationship to obtain a first language probability for each of the plurality of candidate text segments; identifying a target text segment among a plurality of candidate text segments based on first language probabilities and acoustic probabilities to obtain a text sequence for the speech data to be recognized, wherein the constrained sub-model is obtained by training based on text in a predetermined text set.

[0006] According to another aspect of the present disclosure, there is provided a speech recognition device, comprising: an acoustic probability obtaining module for processing the speech data to be recognized and the first text segment obtained by the recognition using the acoustic model to obtain an acoustic probability for each of a plurality of candidate text segments; an initial probability obtaining module that processes the first text segment using a first language sub-model in the language model to obtain initial language probabilities for each of a plurality of candidate text segments; an extended relation obtaining module that processes the first text segment using the constraint sub-model in the language model to obtain an extendible relation of each of a plurality of candidate text segments to the first text segment; a probability adjustment module that adjusts initial language probabilities of the candidate text segments based on the extensible relationship to obtain a first language probability for each of the plurality of candidate text segments; a text identification module that identifies a target text segment among a plurality of candidate text segments based on the first language probability and the acoustic probability to obtain a text sequence for the speech data to be recognized; Here, the constrained submodel is obtained by training the speech recognizer on the basis of text in a predetermined text set.

[0007] According to another aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a speech recognition method according to the present disclosure.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium having computer instructions stored thereon is provided, the computer instructions causing a computer to perform a speech recognition method according to the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program In The computer program M is , a computer program product that, when executed by a processor, implements the speech recognition method according to the present disclosure.

[0010] It should be understood that the contents described in this section are not intended to identify key features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily apparent from the following specification. [Brief explanation of the drawings]

[0011] The drawings are for a better understanding of the invention and are not intended to limit the disclosure. [Figure 1] FIG. 1 is a schematic diagram of an application scene of a speech recognition method and device according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a schematic flowchart of a speech recognition method according to an embodiment of the present disclosure. [Figure 3] FIG. 3 is a principle schematic diagram of obtaining initial language probabilities of multiple candidate text segments according to the first embodiment of the present disclosure. [Figure 4] FIG. 4 is a principle schematic diagram of obtaining first language probabilities of multiple candidate text segments according to the second embodiment of the present disclosure. [Figure 5] FIG. 5 is a structural schematic diagram of a language model according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is a principle schematic diagram of identifying a target text segment according to the first embodiment of the present disclosure. [Figure 7] FIG. 7 is a principle schematic diagram of identifying a target text segment according to the second embodiment of the present disclosure. [Figure 8] FIG. 8 is a principle schematic diagram of identifying a target text segment according to the third embodiment of the present disclosure. [Figure 9] FIG. 9 is a schematic diagram illustrating the principle of generating negative samples for training a constraint sub-model according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a structural block diagram of a speech recognition device according to an embodiment of the present disclosure. [Figure 11] FIG. 11 is a block diagram of an electronic device for implementing the speech recognition method of the embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012]

[0023] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings, including various details of the embodiments of the present disclosure, which are merely illustrative for ease of understanding. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, the following description will omit descriptions of known functions and structures.

[0013] Typically, speech recognition tasks can be completed using acoustic modeling technology. For example, building an end-to-end attention model can improve speech recognition accuracy. However, in actual services, recognition tasks are performed solely by relying on the acoustic model obtained through modeling, making it difficult to meet the high-precision speech recognition needs of specific services. This is because the training data for acoustic models is generally limited and cannot cover a wide range of business fields. Furthermore, business demands in each business field usually change with current topics, and therefore acoustic models usually need to be repeatedly updated to improve recognition accuracy. However, due to the high cost of updating acoustic models and the long update cycle, they usually cannot keep up with the changing speed of accuracy requirements.

[0014] According to this, the speech recognition task can be completed by combining the language model and the acoustic model, and in this way, the language model's advantages of large amount of training data and fast update speed can be utilized to compensate for the shortage of the acoustic model and meet the service demand for high accuracy of speech recognition.

[0015] Here, the language model can be, for example, a neural network language model (NNLM). The NNLM is essentially a sequence model that inputs a text sequence including a text segment obtained by prediction in the previous cycle and outputs a probability distribution for multiple predetermined text segments obtained in the current cycle. In this embodiment, the predetermined text segment with the highest probability value based on the probability distribution can be the text segment obtained by prediction in the current cycle. Here, the acoustic model can be an attention-based acoustic model. Here, each text segment can be text of any granularity, such as one word, text having one syllable, or one phrase.

[0016] According to an embodiment of the present disclosure, a decoding algorithm using a language model and an attention-based acoustic model can fuse a probability distribution output from a single acoustic model and a probability distribution output from a single NNLM, and employ a beam search method to obtain a candidate path selected by a single decoding process based on the fusion result. For example, assuming that there are N predetermined text segments and three beams are used in the beam search, the first decoding will screen three segments with the highest probability values ​​from the N predetermined text segments as candidate text segments, and each subsequent decoding will screen three paths with the highest total probability values ​​from 3*N paths as candidate paths, and if all of the screened candidate paths are text end identifiers, <eos>or until the lengths of all the text segments in the screened candidate paths reach a length threshold, the path may be selected as a candidate path. Here, a path may be represented by a segment sequence obtained from the first decoding to the current decoding, in which the segments are arranged in the order of creation. The total probability value of the path may be the product of the probability values ​​of each segment in the segment sequence, or the sum of the logarithms of the probability values ​​of each segment in the segment sequence.

[0017] Although the method combining a language model and an acoustic model can improve recognition accuracy to a certain extent, this method guides the expansion of the decoding path based on the probability distribution output from the language model. For closed-set recognition tasks, it cannot guarantee that the final recognized text is a text in the text set set for the closed-set recognition task, which will affect the performance of downstream tasks (such as searching based on the recognized text and providing voice response). In other words, this method still has problems such as low recognition accuracy and poor completion of the recognition task.

[0018] Based on this, the present disclosure provides a speech recognition method and apparatus for improving speech recognition accuracy so that the recognition result matches the recognition task. Hereinafter, first, an application scenario of the method and apparatus according to the present disclosure will be described with reference to FIG.

[0019] FIG. 1 is a schematic diagram of an application scene of a speech recognition method and device according to an embodiment of the present disclosure.

[0020] As shown in FIG. 1, the application scenario 100 of this embodiment may include an electronic device 110, which may be any electronic device with processing capabilities, including, but not limited to, a smartphone, a tablet computer, a laptop portable computer, a desktop computer, a smart watch, or a smart speaker.

[0021] The electronic device 110 may, for example, process the acquired voice data 120, for example, by performing speech recognition on the voice data 120 and converting the voice data 120 into text 130. For example, the voice data 120 may be data obtained by processing collected voice. The collected voice may be user voice collected using an audio collector such as a microphone.

[0022] In one embodiment, an audio collector may be installed in the electronic device 110, and a client application with a voice recognition function, such as an input method, a browser, a smart speaker APP, or an in-vehicle APP (for example only), may be installed in the electronic device 110, and the electronic device 110 may convert voice data into input words through voice recognition, so as to perform information queries, smart speaker remote control, or vehicle remote control, etc.

[0023] In one embodiment, the electronic device 110 may complete the speech recognition task using the end-to-end model 140. Here, the end-to-end model 140 may include, for example, the language model and acoustic model described above, and the end-to-end model 140 may acquire the text 130 using a beam search method. Alternatively, the end-to-end model 140 may be the end-to-end streaming attention model described above. Alternatively, the electronic device 110 may complete the speech recognition task using a speech recognition method described below, and the present disclosure is not limited thereto.

[0024] 1, application scene 100 may further include server 150. Server 150 may be, for example, a background management server that supports execution of client applications on electronic device 110. Electronic device 110 may be communicatively coupled to server 150 via a network, which may include a wired or wireless communication link.

[0025] For example, server 150 may train a language model based on a large number of text samples and train an acoustic model based on speech-text pairs. Server 150 may configure the trained language model and acoustic model into end-to-end model 140 and fine-tune the end-to-end model 140 with reference to a specific scene. For example, server 150 may transmit the fine-tuned end-to-end model 140 to electronic device 110 in response to a request from electronic device 110, so that electronic device 110 can use the end-to-end model 140 to complete a speech recognition task.

[0026] In one embodiment, the electronic device 110 can transmit the acquired voice data 120 to the server 150, and the server 150 can perform voice recognition on the voice data 120 based on the end-to-end model 140 to obtain the text 130.

[0027] The speech recognition method according to the present disclosure may be executed by electronic device 110 or server 150. Accordingly, the speech recognition device according to the present disclosure may be provided in electronic device 110 or server 150.

[0028] It should be understood that the number and types of electronic devices 110 and servers 150 in Figure 1 are merely examples. Any number and types of electronic devices 110 and servers 150 may be included as needed.

[0029] The speech recognition method according to the present disclosure will be described in detail below with reference to FIGS.

[0030] FIG. 2 is a flowchart of a speech recognition method according to an embodiment of the present disclosure.

[0031] As shown in FIG. 2, the speech recognition method 200 of this embodiment may include operations S210 to S250.

[0032] In operation S210, the speech data to be recognized and the first text segment obtained by recognition are processed using the acoustic model to obtain acoustic probabilities for each of a plurality of candidate text segments.

[0033] According to an embodiment of the present disclosure, the acoustic model may employ a model consisting of a Gaussian Mixture Model (GMM) and a Hidden Markov Model (HMM), or may employ a model consisting of a Deep Neural Network (DNN) and an HMM. It will be understood that the acoustic model includes, for example, an encoder and a decoder. The input of the encoder is speech data to be recognized and extracted acoustic features. The input of the decoder includes the acoustic features and the embedding features of the first text segment obtained by recognition. The output of the acoustic model is a probability distribution of multiple candidate text segments, which includes the acoustic probability of each of the multiple candidate text segments.

[0034] In the initial stage of speech recognition, the first text segment obtained by recognition is the text start word. <sos>and in a subsequent stage, the first text segment obtained by recognition may be a text starting word <sos>and the text segments obtained by recognition.

[0035] Here, the plurality of candidate text segments may be, for example, a plurality of words in a word library, and the words included in the word library can be set according to actual needs, and the present disclosure is not limited thereto.

[0036] In operation S220, the first text segment is processed using a first language sub-model in the language model to obtain initial language probabilities for each of a plurality of candidate text segments.

[0037] In operation S230, the first text segment is processed using a constraint sub-model in the language model to obtain an extensible relationship of each of a plurality of candidate text segments to the first text segment.

[0038] In operation S240, the initial language probabilities of the candidate text segments are adjusted based on the extensible relationship to obtain a first language probability for each of the plurality of candidate text segments.

[0039] According to an embodiment of the present disclosure, the language model may adopt the above-mentioned NNLM or N-gram model, and the embodiment inputs a first text segment into the language model, and the language model outputs a probability distribution of multiple candidate text segments, the probability distribution including a first language probability for each of the multiple candidate text segments.

[0040] According to an embodiment of the present disclosure, the language model may include, for example, a first language submodel and a constraint submodel. The first language submodel and the constraint submodel may be installed in parallel, and the first language submodel may employ the above-mentioned NNLM. The structure of the constraint submodel is similar to that of the NNLM. The inputs of the first language submodel and the constraint submodel may both be the embedding features of the first text segment. The network structures of the two submodels may be similar, but the main difference is that the first language submodel processes the first text segment to obtain a probability distribution, and the second language submodel processes the first text segment to obtain a vector representing an extensible relationship. Here, the probability distribution obtained by the first language submodel includes a linguistic probability for each of multiple candidate text segments, and the linguistic probability may be used as an initial linguistic probability. The vector representing the extensible relationship includes multiple elements, each of which represents the extensible relationship of a candidate text segment to the first text segment. Here, the extensible relationship may represent the candidate text segment as the segment immediately following the first text segment.

[0041] In one embodiment, the value of each element in the plurality of elements is 0 or 1, where 0 indicates no extensible relationship and 1 indicates an extensible relationship.

[0042] After obtaining the extensible relationship of each of the plurality of candidate text segments to the first text segment based on the output of the constraint sub-model, the initial probability of the candidate text segment can be adjusted based on the extensible relationship. For example, the initial language probability of each candidate text segment can be obtained by multiplying the element value indicating the extensible relationship of each candidate text segment to the first text segment by the element value indicating the extensible relationship of each candidate text segment to the first text segment. Alternatively, the first language probability of each candidate text segment can be obtained by taking the logarithm of the element value indicating the extensible relationship of each candidate text segment to the first text segment and taking the logarithm of the initial language probability of each candidate text segment, and then adding the two logarithms obtained.

[0043] In one embodiment, the constraint sub-model may be obtained based on text training in a predetermined text set, where the predetermined text set may be a text set set for a closed set recognition task, and the closed set recognition task may be set according to actual needs.

[0044] In operation S250, a target text segment among the plurality of candidate text segments is identified based on the first language probability and the acoustic probability to obtain a text sequence for the speech data to be recognized.

[0045] According to embodiments of the present disclosure, for each candidate text segment, the first language probability and the acoustic probability may be added or multiplied, and the resulting value may be a probability value for the candidate text segment. The embodiments may then select the text segment with the highest probability value as the target text segment.

[0046] After obtaining the target text segment, add the target text segment to the first text segment obtained by recognition, and select the text segment with the highest probability value as the text end identification feature. <eos>Steps S210-operation S250 can be continued until the sum of the text segment with the highest probability value and the text segment in the first text segment reaches a predetermined number.

[0047] In one embodiment, a beam search method can be used to identify the last segment of a predetermined number (e.g., M) of paths with a large total probability value as the target text segment. Then, each target text segment is added to the first text segment to obtain M adjusted text segments. Then, each adjusted text segment is taken as one first text segment, and the process returns to operation S210-operation S240 to obtain a total of M*N candidate paths. Then, M paths with the highest total probability value are selected from the M*N candidate paths. By analogy, it is possible to determine whether all of the screened candidate paths have the text end identification feature. <eos>or until the lengths of the text segments in the screened candidate paths all reach a length threshold. Finally, the text sequence of the speech data to be recognized is constructed from the text segments on the candidate path with the highest probability sum.

[0048] In an embodiment of the present disclosure, a constraint sub-model is set in a language model to predict the extensible relationship of a candidate text segment to a first text segment, and the predicted initial language probability is adjusted based on the extensible relationship, so that the extensible relationship and the initial language probability can be combined to guide the extension of a decoding path. In this way, the constraint sub-model is a neural network model that learns the extensible relationship between each segment of multiple candidate text segments based on a text set set in a closed set recognition task, and then, under the guidance of the extensible relationship, the recognized text can be made to be a certain text in the text set set in the closed set recognition task, thereby improving recognition accuracy, improving the completion effect of the recognition task, and benefiting the implementation of downstream tasks.

[0049] Hereinafter, the implementation of the above steps S220 to S240 will be further expanded and limited with reference to FIGS.

[0050] FIG. 3 is a principle schematic diagram of obtaining first language probabilities of multiple candidate text segments according to the first embodiment of the present disclosure.

[0051] According to an embodiment of the present disclosure, when a language model is used to obtain a language probability, for example, a category mark may be added to the input of the language model. In this way, the language model can guide different paths for different types of text. The language model of the present disclosure can predict multiple different types of text, which is advantageous for improving the robustness of the speech recognition method of the present disclosure.

[0052] 3, in this embodiment 300, when determining the initial language probabilities for each of the candidate text segments, a first text segment 301 can be processed first to obtain text embedding features 302 for the first text segment 301. For example, the first text segment 301 can be processed using the word2vec method or the Global Vectors for Word Representation (GloVe) method.

[0053] At any timing when obtaining the text embedding feature 302, the embodiment can further identify a category 303 to which the first text segment belongs and a first discriminative feature 304 of the category 303. As can be understood, the category 303 to which the first text segment belongs can be determined, for example, in response to a user operation. Alternatively, in an early stage of speech recognition, multiple predetermined categories can all be determined as the category 303 to which the first text segment belongs, and one probability distribution can be obtained for each predetermined category. As the path expands, the predetermined category corresponding to the selected path can be determined as the category to which the first text segment obtained by recognition belongs. The embodiment can assign a discriminative feature to each of the multiple predetermined categories, and the embodiment can obtain the first discriminative feature of the category by encoding the discriminative feature of the category.

[0054] After obtaining the text embedding features 302 and the first discriminative features 304, the embodiment may first fuse the text embedding features 302 and the first discriminative features 304. The features obtained by the fusion are then input into the first language sub-model 320, and after processing through the first language sub-model 320, a language probability distribution 305 may be obtained. The language probability distribution 305 includes initial language probabilities for a plurality of predetermined text segments.

[0055] For example, the text embedding feature 302 and the first identifying feature 304 can be joined together to achieve fusion. Alternatively, the text embedding feature 302 and the first identifying feature 304 can be set to have the same dimension, and the embodiment can achieve fusion by adding the text embedding feature 302 and the first identifying feature 304 using the adder 310. The above fusion method is merely an example and is useful for understanding the present disclosure, and the present disclosure is not limited thereto.

[0056] Exemplarily, the first language sub-model 320 may employ an NNLM model. For example, the first language sub-model 320 may include an input layer, a hidden layer, and an output layer connected in series. Here, the input layer may be used to convert text into embedding features. It should be understood that the input layer may include a function for processing the first text segment described above to obtain text embedding features, a function for obtaining first discriminative features based on vertical type, and a function for fusing the text embedding features and the first discriminative features. The hidden layer may be a fully connected layer, or may have a network structure consisting of a sequence network and a fully connected layer to facilitate learning of contextual information between multiple data in the input sequence. Here, the sequence network may include a network based on an attention mechanism (e.g., a Transformer) or a long short-term memory network (LSTM), etc., but the present disclosure is not limited thereto. The output layer may include a logistic regression network such as softmax.

[0057] FIG. 4 is a principle schematic diagram of obtaining first language probabilities of multiple candidate text segments according to the second embodiment of the present disclosure.

[0058] According to an embodiment of the present disclosure, a language model can have one general-purpose language model branch, which is obtained by text training using multiple categories. Considering the lack of bias in the categories branched by the general-purpose language model, since the language model for a category generally has too many parameters, this embodiment combines the two, sharing the parameters of the general-purpose language model branch with the language model for the category, and adding some additional parameters to the language model for the category to perform individual reinforcement learning for the category. That is, the language model has two branches: one general-purpose language model branch and one language model branch for the category. In this way, the recognition rate of multi-draw types using the language model is optimized, while ensuring a small model volume, thereby reducing the computational power required when running the model and improving the robustness of the method of this embodiment.

[0059] 4, in this embodiment 400, the language model may include a first language submodel 410, a second language submodel 420 arranged in parallel with the first language submodel 410, and a constraint submodel 430. Here, the first language submodel 410 and the constraint submodel 430 constitute a language model branch for a category.

[0060] To obtain the first language probability, the embodiment may input the text embedding features 401 into the second language sub-model 420 and obtain a first implicit representation of the hidden layer output of the second language sub-model 420.

[0061] In this embodiment, the text embedding feature 401 may be fused with the first discriminative feature 402 of the category to which it belongs, and then input to the first language sub-model 410, and the first implicit representation may be fused with the second implicit representation output from the hidden layer of the first language sub-model 410. The fused feature is then input to the output layer of the first language sub-model 410, and the output layer outputs a language probability distribution, thereby obtaining the initial language probability for each of the multiple candidate text segments.

[0062] In this embodiment, the text embedding features 401 may be fused with the first indicator features 402 before being input to a constraint sub-model 430, which may then output a vector representing the extensible relationship. The vector and the initial language probabilities may be input to a fusion layer 440, which may then adjust the initial language probabilities based on the vector representing the extensible relationship, thereby outputting the first language probabilities 403 for each of the multiple candidate text segments.

[0063] Here, the hidden layer in the first language submodel may be a first feature extraction network, and the output layer may be a first prediction network. The input of the first prediction network includes the second implicit representation and the features after the first implicit representation is fused (e.g., fused using an adder), and the output of the first prediction network is a probability distribution. The vector representing the scalable relationship may be adjusted by adjusting the logarithm of the probability value in the probability distribution. In this embodiment, by determining the linguistic probability based on the logarithm of the probability value, the multiplication relationship between numbers can be converted into an addition relationship between the logarithms of the numbers, ensuring calculation accuracy. This is because electronic devices typically have relatively low calculation accuracy for multiplication of floating-point numbers and high calculation accuracy for addition.

[0064] In one specific embodiment, the first language sub-model may include an LSTM layer, an adder, a fully connected layer, and a logistic regression layer (softmax). Here, the adder may be provided between the fully connected layer and the logistic regression layer. Correspondingly, the LSTM and the fully connected layer constitute a first feature extraction network, and the softmax layer constitutes a first prediction network. In one specific embodiment, an adder is provided not only between the fully connected layer and the logistic regression layer, but also between the LSTM layer and the fully connected layer. Correspondingly, the LSTM layer, the adder provided between the LSTM layer and the fully connected layer, and the fully connected layer constitute a first feature extraction network 411, and the adder provided between the fully connected layer and the logistic regression layer and the logistic regression layer constitute a first prediction network 412. Here, the adder between the LSTM layer and the fully connected layer is used to fuse the features of the first implicit representation and the LSTM output, and the adder between the fully connected layer and the logistic regression layer is used to fuse the first implicit representation and the second implicit representation. In this way, sufficient fusion between the first implicit representation and the features in the first language submodel is achieved, the sharing between the network parameters in the first language submodel and the network parameters in the second language submodel is strengthened, the accuracy of the obtained first language probability is improved, and the speech recognition accuracy is improved.

[0065] In one specific embodiment, the second language sub-model 420 may include an LSTM layer, a fully connected layer, and a softmax layer. The LSTM layer and the fully connected layer constitute a second feature extraction network 421 of the second language sub-model, and the softmax layer constitutes a second prediction network 422 of the second language sub-model. In this embodiment, the text embedding features 401 of the first text segment are input to the second feature extraction network 421 to obtain a second implicit representation, and then the second implicit representation is input to the second prediction network 422, and another probability distribution is output from the second prediction network 422 to obtain a second language probability 404 for each of the multiple candidate text segments. Finally, the embodiment may identify a target text segment based on the first language probability 403, the second language probability 404, and the acoustic probability. Specifically, the first language probability 403 and the second language probability 404 may be added to the acoustic probability, respectively. If the number of predetermined text segments is set to N, a total of 2*N probability values ​​are obtained after addition. Then, by selecting M largest probability values ​​from the 2*N summed probability values, the candidate path obtained by the current decoding is obtained. In this manner, the method of the embodiment of the present disclosure can be applied not only to scenes of multiple categories but also to general-purpose speech recognition scenes, thereby improving the robustness of the method of the embodiment.

[0066] FIG. 5 is a structural schematic diagram of a language model according to an embodiment of the present disclosure.

[0067] According to an embodiment of the present disclosure, a language model can be configured with a third language sub-model in parallel with a first language sub-model for learning the relationship between speech data and text from different sources. In this embodiment, the language probabilities obtained by branching from the third language sub-model and the language probabilities obtained by branching from the language model for a category can be screened as parallel options. In this way, the language model of this embodiment can be applied to different categories in different scenes, eliminating the need for separate training for different categories, thereby improving model robustness and reducing model training costs.

[0068] 5, in this embodiment 500, the language model may include a first language submodel 510, a second language submodel 520, a constraint submodel 530, and a third language submodel 540. Here, the first language submodel 510, the second language submodel 520, and the constraint submodel 530 are similar to the corresponding models in the above-mentioned FIG. 4, and therefore will not be described again.

[0069] In this embodiment 500, the third language sub-model 540 is similar to the first language sub-model 510, except that the input of the third language sub-model is the fused features of the second discriminative features 503 and the text embedding features 501, which represent the source of the speech data to be recognized.

[0070] Accordingly, when performing speech recognition, this embodiment can further identify a second indicator feature 503 that indicates the source of the speech data to be recognized. For example, if a user identifies poor speech recognition results, training data can be provided. The method of this embodiment can assign a distinguishing feature to the user and train a third language sub-model based on the training data provided by the user. In actual speech recognition, the second indicator feature can be obtained by identifying a user based on the source of the speech to be recognized and encoding the distinguishing feature assigned to the identified user. It should be understood that the user may be various client applications with speech recognition capabilities. The second distinguishing feature may be obtained by encoding the name of the client application, etc., and this disclosure is not limited thereto.

[0071] After obtaining the second discriminative features 503, the embodiment 500 can use a third language sub-model 540 to process the fused features of the text embedding features 501 and the second discriminative features 503. Based on a principle similar to that of the first language sub-model obtaining the initial language probabilities, the third language sub-model 540 can output a probability distribution. By taking the logarithm of the probability values ​​in the probability distribution, a third language probability 506 can be obtained for each of the multiple candidate text segments.

[0072] As shown in FIG. 5 , similar to the first language sub-model described above, in this embodiment 500, the third language sub-model 540 may include a third feature extraction network and a third prediction network. In this embodiment, the features obtained by fusing the text-embedded features 501 and the second sign features 503 may be input to the third feature extraction network 541 to obtain a third implicit representation. Then, the features obtained by fusing the first implicit representation and the third implicit representation are input to the third prediction network 542, which outputs a probability distribution. By taking the logarithm of the probability values ​​in the probability distribution, a third language probability 506 for each of the multiple candidate text segments may be obtained.

[0073] After obtaining the third language probability 506, the embodiment can identify a target text segment based on the third language probability 506, the first language probability 504, and the acoustic probability, the principle of which is similar to the principle of identifying a target text segment based on the first language probability, the second language probability, and the acoustic probability described above, and therefore will not be described here.

[0074] In one embodiment, the second language sub-model 520 obtains the second language probability 505, and the embodiment 500 can identify the target text segment based on the first language probability 504, the second language probability 505, the third language probability 506, and the acoustic probability. The principle is similar to the principle of identifying the target text segment based on the first language probability, the second language probability, and the acoustic probability described above, and therefore will not be described here.

[0075] If the language model is a sequence model and recognizes the speech to be recognized, the initial input of the first language submodel in the language model is a text start identifier (TII). <sos>It can be seen that the second language submodel includes P features obtained by adding the embedding features of the first language submodel and the discriminative features of the P predetermined categories. The initial input of the second language submodel is a text start identifier (TBI). <sos>The initial input for the third language submodel is the text start identifier. <sos>The feature is a sum of the embedding feature of the first feature and the second discriminative feature representing the source of the speech to be recognized. After processing the language model, (P+2)*N probability values ​​are obtained, which correspond to (P+2)*N expansion paths. This embodiment can select M paths with the highest total probability values ​​from the (P+2)*N expansion paths. Thus, in the second decoding, the first text segment obtained by recognition includes M text segments, and the text start identifier <sos>and the text segments corresponding to the M paths with the highest total probability values. Next, the M text segments are input into the second language sub-model, respectively, to obtain M*N expanded paths. The M text segments are then fused with the discriminative features of the categories corresponding to the M paths with the highest total probability values, and then input into the first language sub-model, to obtain M*N expanded paths. The M text segments are fused with the second discriminative features, respectively, and then input into the third language sub-model, to obtain M*N expanded paths, for a total of 3M*N expanded paths. Then, the M paths with the highest total probability values ​​are selected from the 3M*N expanded paths, and it can be inferred that all of the M paths obtained by screening are text end discriminative features. <eos>Decoding is performed multiple times until the length of the text segments in the M screened paths reaches the length threshold. Finally, the text sequence corresponding to the path with the highest probability sum is the recognized text sequence of the speech data to be recognized. In the i-th decoding, the number of text segments included in the screened paths is (i+1), and the text segments have a text start identifier (TBI). <sos>It can be understood that this includes

[0076] The implementation of the above operation S250 will be further expanded and limited below.

[0077] According to an embodiment of the present disclosure, for a closed set recognition task, a predetermined text list can be set based on a text set set for the closed set recognition task. When identifying a target text segment, the target text segment is selected from a plurality of candidate text segments based on the predetermined text list. In this way, the text comprising the recognized text sequence can belong to the text set set for the closed set recognition task, and the method of this embodiment can forcibly recognize the ability to set a certain text into a closed set. If the closed set recognition task is a speech recognition task for a smart speaker, the method of this embodiment can ensure that the song titles, artist names, etc. included in the recognized text sequence are existing song titles and artist names, which is advantageous for playing music that meets the user's actual needs based on the recognition results.

[0078] In this embodiment, the plurality of candidate text segments may include, for example, a plurality of first candidate segments indicating candidate words. The candidate words may be set according to actual needs, and the present disclosure is not limited thereto. In this embodiment, when identifying a target text segment, a predetermined text list may be first queried based on the first text segment, and a first designated segment from the plurality of first candidate segments may be identified based on the query result. For example, a predetermined text list may be searched, and a text including the first text segment in the predetermined text list may be identified as the first text. For example, if the set text set includes the text "Please broadcast singer A's song a" and the first text segment is "Please release...", the text "Please broadcast singer A's song a" may be identified as the first text. Then, in this embodiment, the word "send" following the first text segment in the first text may be identified as the first designated segment. That is, the text formed by combining the first designated segment and the first text segment belongs to the predetermined text list.

[0079] After obtaining the first designated segment, the embodiment may identify a target text segment among the plurality of first candidate segments based on the first language probability and acoustic probability of the first designated segment. For example, the embodiment may add the logarithm of the first language probability of the first designated segment to the logarithm of the acoustic probability. The resulting value is used as the probability value of the first designated segment for the first text segment. If there is only one first text segment, the embodiment may determine M first designated segments with the highest probability value for the first text segment as the target text segment. If there are multiple first text segments, the embodiment may first select a second text segment from the plurality of first text segments, the text of which combined with the first designated segment belongs to a predetermined text list, and multiply the probability value of the second text segment by the probability value of the first designated segment for the second text segment to obtain a probability value for the text obtained by stitching the first designated segment and the second text segment. Finally, the embodiment may determine the first designated segment among the M texts with the highest probability value as the target text segment.

[0080] In one embodiment, for example, a discrimination weight can be set for each text in a predetermined text list, and the discrimination weight can be determined based on the discrimination difficulty. For example, the recognition weight is positively correlated with the recognition difficulty. When identifying a target text segment, the recognition weight can be combined with the candidate text segment for screening, so that the speech recognition method can easily recognize text with high recognition difficulty and enhance the speech recognition method's recognition ability for high-difficulty text. For example, the discrimination weight can be set and modified according to actual needs, and the present disclosure is not limited thereto.

[0081] FIG. 6 is a principle schematic diagram of identifying a target text segment according to the first embodiment of the present disclosure.

[0082] 6 , in identifying a target text segment, the embodiment 600 may first query a predetermined text list 602 based on a first text segment 601, and identify a text in the predetermined text list 602 that includes the first text segment 601 as a first text 603. In the embodiment, a text segment that belongs to a plurality of first candidate segments and is located in the first text 603 after the first text segment 601 may be a first specified segment 604.

[0083] The embodiment may then identify text resulting from the splicing of the first text segment 601 and the first designated segment 604 as spliced ​​text 605, and a portion of the first text 603 including the spliced ​​text as first target text 606. Finally, the embodiment may identify a target text segment based on the recognition weight of the first target text 606, the first language probability of the first designated segment 604, and the acoustic probability of the first designated segment 604. For example, the recognition weight of the first target text 606, the logarithm of the first language probability of the first designated segment 604, and the logarithm of the acoustic probability of the first designated segment 604 may be added together to obtain a probability value of the first designated segment 604 relative to the first text segment 601, and then select a target text segment from the identified first designated segment 604 based on the probability value.

[0084] According to an embodiment of the present disclosure, in a predetermined text list, for example, a predetermined text is represented in a template format, and entity-class text segments in the predetermined text are represented as slots. In the predetermined text list, entities included in the entity class corresponding to the slots can be listed, which is advantageous for path management for refining speech recognition and for improving speech recognition accuracy. Here, the entity-class text segments may include text segments indicating, for example, song titles, artist names, and point of interest names, and different types of entities correspond to one slot. For example, a slot corresponding to an entity in the song title category is [song], a slot corresponding to an entity in the musical piece name category is [singer], and a slot corresponding to an entity in the point of interest name category is [POI].

[0085] Correspondingly, this embodiment can predict and obtain a text sequence using a macrograph decoding method, where the large figure corresponds to a text template and the small figure corresponds to a slot. If the text segment after the first text segment is an entity represented by a slot, this embodiment can predict the entity represented in the slot based on the discriminative features of the slot, thereby enabling the language model to predict different slots and helping to improve the accuracy of the predicted target text segment. By considering the discriminative features of the slot, the language model can learn the mapping relationship between different slots and the predicted text segment.

[0086] The principle of identifying the target text segment in this embodiment will be described in detail below with reference to FIG.

[0087] FIG. 7 is a principle schematic diagram of identifying a target text segment according to the second embodiment of the present disclosure.

[0088] 7, in this embodiment 700, the plurality of candidate text segments includes a plurality of first candidate segments indicating candidate words and a plurality of second candidate segments indicating candidate slots. Here, the candidate slots may be set according to actual needs, for example, the candidate slots may be set based on the categories of entities in actual scenes, and each category of entity corresponds to one candidate slot. The slots of the entities corresponding to each category in the candidate slots may be understood as incoming slots, and an outgoing slot may be further set in the prediction process to indicate that prediction for the entity is complete.

[0089] In this embodiment 700, after obtaining the first language probability using the language model, for example, a similar method of identifying the first specified segment based on the predetermined text list may be adopted to first identify a target slot 703 in the incoming slot 701 that belongs to the predetermined text list 702 based on the predetermined text list 702. This embodiment can filter incoming slots that cannot recognize closed-set text. Specifically, as described above, the text in the predetermined text list is composed of words and slots, and the slot corresponding position is the position where the entity is located in the predetermined text. In this embodiment, the slots that constitute the text in the predetermined text list can be compared with the incoming slot 701 to obtain the target slot 703.

[0090] The embodiment then uses a language model 710 to identify the third distinguishing feature 704 of the target slot 703 and the beginning identifier of the text. <sos>The features obtained based on 705 can be processed to obtain fourth language probabilities for a plurality of first candidate segments. The fourth language probabilities can represent the probability that each candidate word belongs to a segment in the target slot 703. This part is a decoding process in which the embedding feature of the start identifier of the text is used to replace the text embedding feature of the first text segment, and the third discriminative feature 704 of the target slot 703 is used to replace the first discriminative feature of the category to which the first text segment belongs. Specifically, this embodiment can first identify the third discriminative feature 704 of the target slot 703, and the third discriminative feature 704 can be encoded by the discriminative feature assigned to the target slot 703. At the same time, the start identifier <sos>705 to obtain the start identifier coding feature. Then, the third identifying feature 704 is added to the start identifier code feature, and the third identifying feature 704 of the target slot 703 and the start identifier of the text are combined. <sos>The features obtained based on 705 are obtained, and the features can be input to the first language sub-model and the constraint sub-model in the language model 710. Using a principle similar to that for obtaining the first language probability above, a fourth language probability 706 for the target slot of the first candidate segment is obtained.

[0091] After obtaining the fourth language probability 706, the embodiment may identify target text segments in the first candidate segments based on the fourth language probability 706, the first language probability, and the acoustic probability. For example, assuming that there are Q target slots, for each target slot, the embodiment may identify probabilities of multiple first candidate segments as text segments in the target slot based on the fourth language probability obtained based on the third discriminant feature of the target slot and the first language probability indicating the second candidate segment for the target slot. For example, the fourth language probability of each first candidate segment may be multiplied by the first language probability indicating the second candidate segment for the target slot to determine the probability of each first candidate segment as a text segment in the target slot. If there are N' multiple first candidate segments, N' probabilities are obtained for each target slot, resulting in a total of Q*N' probabilities for the Q target slots. The embodiment may arrange the Q*N' probabilities and the first language probabilities of the N' first candidate segments into a probability set, the probability set including a total of (Q+1)*N' probabilities.

[0092] In the embodiment 700, for example, the logarithm values ​​of the (Q+1)*N' probabilities can be added to the logarithm values ​​corresponding to the acoustic probabilities of the first candidate segment to obtain (Q+1)*N' expanded probabilities. In the embodiment, M paths can be selected from the (Q+1)*N' paths corresponding to the (Q+1)*N' expanded probabilities based on the (Q+1)*N' expanded probabilities, and the text segment corresponding to the last position of the M paths can be determined as the target text segment.

[0093] According to an embodiment of the present disclosure, a target text segment can be identified for an outgoing slot in a manner similar to that for an incoming slot. The difference is that for an outgoing slot, instead of the text embedding feature of the first text segment, the input language model 710 features a discriminative feature of the outgoing slot, specifically, the fourth discriminative feature of the slot corresponding to the last text segment in the first text segment. The first discriminative feature should be a discriminative feature of the category to which the first text segment belongs. This embodiment can obtain a second fusion feature by fusing the fourth indicator feature and the first indicator feature. The second fusion feature can be used as an input for a language model to obtain a fifth language probability for the outgoing slot of the plurality of first candidate segments through language model processing. Finally, this embodiment can identify a target text segment in the plurality of first candidate segments based on the fifth language probability, the first language probability, and the acoustic probability.

[0094] For example, this embodiment may employ the above-described method to obtain a total of Q*N' probabilities for Q target slots. This embodiment may multiply the first language probability of the second text segment indicating the output slot by the fifth language probability for each first candidate segment for the output slot to obtain a total of N' probabilities for N' first candidate segments as the probability of each first candidate segment being the first text segment after the slot out. This embodiment may construct a probability set using the obtained Q*N' probabilities, N' probabilities of the N' first candidate segments being the first text segment after the slot, and N' first language probabilities for the N' first candidate segments, where the probability set includes a total of (Q+2)*N' probabilities.

[0095] Then, the embodiment may add the logarithmic values ​​of the (Q+2)*N' probabilities to the logarithmic values ​​corresponding to the acoustic probabilities of the first candidate segment, respectively, to obtain (Q+2)*N' expanded probabilities. The embodiment may select M paths from the (Q+2)*N' paths corresponding to the (Q+2)*N' expanded probabilities based on the (Q+2)*N' expanded probabilities, and determine the text segment corresponding to the last position of the M paths as the target text segment.

[0096] According to an embodiment of the present disclosure, when identifying the target slot 703, for example, a slot belonging to a predetermined text list in the incoming slot can be designated as the initial slot. Next, the first language probability of the second candidate segment representing the initial slot is compared with the first language probabilities of the multiple first candidate segments, and the initial slot represented by the second candidate segment with the relatively higher probability value is designated as the target slot. For example, this embodiment first identifies a predetermined number of probabilities with the highest values ​​among the multiple first candidate segments, then compares the first language probability of the second candidate segment representing the initial slot with the minimum probability among the predetermined number of probabilities, and identifies a certain initial slot bit as the target slot if the first language probability of the second candidate segment representing the initial slot is higher than the minimum probability or lower than the minimum probability and the absolute value of the difference between the first language probability and the minimum probability is equal to or less than a first predetermined threshold. Alternatively, this embodiment can compare the first language probability of the second candidate segment representing the initial slot with the maximum probability among the multiple first candidate segments, and identify the initial slot bit as the target slot bit if the absolute value of the difference between the first language probability and the minimum probability is less than a second predetermined threshold. The above method of identifying a target slot by the difference is merely an example for ease of understanding the present disclosure, and the present disclosure is not limited thereto.

[0097] The embodiments of the present disclosure can further screen the incoming slots by identifying the target slot based on the difference in first language probability with multiple first candidate segments, and by removing slots with low expanded probabilities, can ensure prediction accuracy while reducing the amount of calculation and obtain the computational efficiency of the target text segment through decoding.

[0098] FIG. 8 is a principle schematic diagram of identifying a target text segment according to the third embodiment of the present disclosure.

[0099] According to an embodiment of the present disclosure, when decoding by jumping to a sub-figure, for example, the discrimination weights assigned to the texts in a predetermined text list may be combined to obtain the target text segment by screening, so that the speech recognition method can easily recognize the texts with high recognition difficulty, and the recognition ability of the speech recognition method for the texts with high recognition difficulty can be improved.

[0100] For example, after obtaining the fourth language probability, or at any time, a predetermined text list is queried based on the first text segment to obtain a second target text and a second designated segment from among the plurality of first candidate segments. Specifically, the first text segment and second candidate segments indicating slots corresponding to the first candidate segments can be stitched together to obtain a plurality of stitched texts. Next, a predetermined text list is queried based on the spliced ​​text, and a predetermined text including any one of the plurality of spliced ​​texts is identified as the second target text, and a first candidate segment corresponding to an indicated slot included in any one of the texts is identified as the second designated segment. For convenience of explanation, this embodiment may also use the second candidate segment indicating the slot corresponding to the second designated segment as the target candidate segment.

[0101] Then, the embodiment can determine the initial probability of the target candidate segment based on the recognition weight of the second target text and the first language probability of the target candidate segment. For example, the recognition weight of the second target text can be multiplied by the first language probability of the target candidate segment, and the product can be used as the initial probability. Alternatively, the logarithm of the recognition weight of the second target text can be added to the logarithm of the first language probability of the target candidate segment to obtain the initial probability, but the present disclosure is not limited thereto.

[0102] After obtaining the initial probability, the embodiment may determine the probability that the second designated segment is the first text segment in the target slot based on the initial probability and the fourth language probability of the second designated segment, for example, by adding the initial probability to the logarithm of the fourth language probability of the second designated segment to obtain the probability that the second designated segment is the first text segment in the target slot. The probability may be replaced with a corresponding probability among the above-mentioned Q*N' probabilities.

[0103] Hereinafter, with reference to FIG. 8, an example will be taken to describe in detail the principle of decoding the target text segment in the embodiment of the present disclosure.

[0104] As shown in Figure 8, in this embodiment 800, when a text sequence is obtained by decoding using a beam search method, if the number of beams is set to M, the number of first text segments in each cycle of the decoding process is M except for the first cycle. The number of candidate words is N', and the candidate slot bits include Q' incoming slots and one outgoing slot. For a text segment 801 in the M first text segments, this embodiment can obtain N' acoustic probabilities 802 using an acoustic model 810. Using a language model 820, it can obtain N' language probabilities corresponding to the N' candidate words, incoming probabilities corresponding to the Q' incoming slots, and outgoing probabilities corresponding to the outgoing slot, totaling (N' + Q' + 1) language probabilities 803.

[0105] In addition, this embodiment may query a predetermined text list 830 based on a text segment 801 to obtain query acquisition information 804, which may include the first target text and its recognition weight w1 and the second target text and its recognition weight w2. This embodiment may obtain an expandable word 805, the target slot 806, and the throw slot 807 by screening the text segment corresponding to the predicted linguistic probability based on the query-obtained information 804. It is understood that the expandable field may be the first specified segment. If the throw probability of the throw slot 807 is much smaller than the probability of the target slot and the expandable word, the throw slot may be deleted. Here, the expansion probability of the expandable word 805 may be expressed as the sum of the logarithm of the acoustic probability of the expandable word, the logarithm of the linguistic probability of the expandable word, and the discrimination weight w1 of the first target text corresponding to the expandable word. The expandable initial probability of the target slot 806 can be expressed as the sum of the logarithm of the entry probability of the target slot 806 and the discriminative weight w2 of the second target text corresponding to the target slot. The expandable initial probability of the exit slot is expressed as the logarithm of the exit probability.

[0106] In this embodiment, the expandable word 805 can be taken as a candidate text segment, the candidate text segment and the text segment 801 can be stitched together, and the stitched text can be added to the first candidate pool 808 for the text segment 801 .

[0107] For the target slot, this embodiment employs a similar method to that described above to input the embedding feature of the text start identifier and the discriminative feature of the target slot into language model 820, and then jump to the small block to perform a decoding operation to obtain the fourth language probability. For the output slot, this embodiment employs a similar method to that described above to input the discriminative feature of the category to which the first text segment belongs and the discriminative feature of the slot corresponding to the text segment at the last position in the first text segment into language model 830, and then jump to the large block to perform a decoding operation to obtain the fifth language probability. Then, this embodiment can search a predetermined text list to constrain the fourth language probability and the fifth language probability based on the text in the list, and screen and obtain text segments belonging to the text in the predetermined text list, and combine the text segments with text segment 801 to add them to first candidate pool 808.

[0108] Based on a similar principle, M candidate pools can be obtained for each text segment in the M first text segments. In this embodiment, M candidate text segments with the largest sum of probabilities can be selected from the M candidate pools to be the M first text segments in the next cycle. Any of the selected M candidate text segments has a text end identification feature. <eos>or the number of text segments in the M candidate text segments all reaches a predetermined number.

[0109] As can be seen from the above, in an embodiment of the present disclosure, two calculations using a language model are typically required in one cycle. To improve calculation efficiency, in the embodiment, when a first target feature is processed using the language model a predetermined number of times, the language probability obtained by processing the first target feature using the language model can be stored in a cache for subsequent retrieval. Accordingly, when it is determined that a target feature (e.g., a second target feature) needs to be processed using the language model, the cache is first searched to determine whether the language probability for the second target feature is stored in the cache. If so, the language probability is directly read from the cache to complete the processing of the second target feature using the language model, eliminating the need for complex calculations using the language model.

[0110] It should be understood that the first and second target features may include any one of the following features: text embedding features of the first text segment, features obtained by combining text embedding features and discriminative features of a category, features obtained by combining text embedding features and discriminative features of a data source, and features obtained by combining text embedding features and discriminative features of a slot. That is, the first and second target features may be any features of the hidden layer in the input language model, and the present disclosure is not limited thereto.

[0111] In one embodiment, the operation of identifying the target text segment may be performed using a high performance processor such as a graphics processor GPU, so that the calculations for the M first text segments or any parallel calculations in the process of identifying the target text segment can be performed in parallel by the GPU or the like, to further improve decoding efficiency and improve speech recognition efficiency.

[0112] According to an embodiment of the present disclosure, a text segment table can be maintained for a candidate slot, and a text segment belonging to the candidate slot can be added to the text segment table. In this embodiment, after recognizing and obtaining a text sequence, for example, a slot text segment belonging to a candidate slot in the text sequence can be compared with a text segment in the text segment table for the candidate slot. Specifically, in response to a slot text segment belonging to a candidate slot being included in the text sequence, the text segment table for the candidate slot can be queried based on the slot text segment. If the slot text segment does not belong to the text segment table for the candidate slot, the slot text segment can be compared with each text segment in the text segment table for the slot, and the text segment with the greatest similarity to the slot text segment in the text segment table can be determined as a candidate segment. Then, the slot text segment in the text sequence can be replaced with the candidate segment, and the replaced text segment is used as the recognition result for the speech data to be recognized.

[0113] This method can ensure that the text segment in the candidate slot in the text sequence is the text segment in the text segment table, and ensure that the text segment in the generated recognition result is a reasonable segment. For example, if the slot text segment is "moment fruit", the query can replace "moment fruit" with "apple", so that the generated recognition result is reasonable and improves the accuracy of the recognition result.

[0114] Referring to FIG. 9 below, the sample creation adopted in training the constraint sub-model in the language model can be expanded and limited so that the constraint sub-model can learn scalable relationships between multiple candidate text segments in the closed set recognition task, thereby improving the task completion effect and benefiting the implementation of downstream tasks.

[0115] FIG. 9 is a schematic diagram illustrating the principle of generating negative samples for training a constraint sub-model according to an embodiment of the present disclosure.

[0116] According to an embodiment of the present disclosure, the samples for training the constraint sub-model may include, for example, positive samples and negative samples. Here, the positive samples may include text in a predetermined text set, and the negative samples may be any text other than the text in the predetermined text set. In this manner, the text creation paths that do not belong to the predetermined text set can be trimmed during the decoding process based on the vector representing the extensible relationship created by the constraint sub-model.

[0117] In one embodiment, the predetermined text segment can be adjusted based on a second text segment among the plurality of candidate text segments that does not match the text segment at the target position in the predetermined text, and the adjusted text can be a negative sample. Here, the target position can be any position in the predetermined text. In this manner, the negative sample is generated, and the only difference between the negative sample and the positive sample is the text segment at the target position, which can improve the learning ability of the constraint submodel.

[0118] For example, as shown in Figure 9, the embodiment 900 can randomly extract one predetermined text from a predetermined text set 910 as a positive sample 911. The embodiment can remove a predetermined number of text segments that are ejected to the last position in the extracted predetermined text, and the resulting text can also be a positive sample.

[0119] After extracting and obtaining the predetermined text, the negative sample 930 can be obtained by replacing the text segment at the target position in the predetermined text with the second text segment 920 described above.

[0120] In one embodiment, the target position may be, for example, the last position of the given text, so that the negative and positive samples have the same prefix, and the decoding process can effectively prune the text-creation paths that do not belong to the given text set text in the last cycle.

[0121] In one embodiment, the target position can be any position, and the embodiment can replace the text segment at the target position in the extracted predetermined text with the second text segment 920, and then remove the text segment located at the target position in the predetermined text to obtain a negative sample.

[0122] This embodiment can remove the text segment after the target position to obtain negative samples, so that all negative samples have the same prefix as the positive samples. By selecting the target position at an arbitrary position, the constraint sub-model can learn an extensible relationship between any two text segments in a given text, which helps improve the clipping accuracy and effectiveness of the decoding path.

[0123] In one embodiment, when adjusting a given text using a second text segment, for example, a segment to be replaced in the second text segment can be first identified based on the confusion relationship between the second text segment and the text segment at a target position in the given text. The segment to be replaced can then be replaced with the text segment at the target position in the given text, and the replaced text can be designated as a negative sample. This method allows the generated negative sample to be considered as text that is easily confused with the given text (i.e., a positive sample), which is beneficial to improving the discrimination ability of the constraint submodel. Furthermore, the selection of the segment to be replaced in this embodiment can effectively reduce the number of negative samples and the pairing of negative samples, which helps improve the training efficiency of the constraint submodel.

[0124] Here, the confusion relationship may be expressed, for example, by text similarity or syllable similarity between text segments, and the higher the similarity, the easier it is to confuse them.

[0125] In one embodiment, when generating negative samples, for example, the second text segment may be used to replace a text segment at a target position in a given text, and the resulting text segment may be used as a candidate sample. Then, the pre-trained first language sub-model is used to process each candidate sample, and a first language sub-model is obtained to generate a language probability for each candidate sample. The language probability may be sequentially generated by multiplying multiple language probabilities of multiple text segments in each candidate sample. The embodiment then screens the candidate samples based on the sixth language probability, and selects candidate samples with a sixth language probability higher than a probability threshold as negative samples. Alternatively, select several candidate samples with a high sixth language probability as negative samples. This method can control the size of negative samples and ensure that the first language sub-model can decode the generation path of the negative samples to obtain selectable paths for the text sequence, thereby enabling pairwise training of the constrained sub-model and improving the training efficiency of the constrained sub-model and the accuracy of the trained constrained sub-model.

[0126] In one embodiment, the sixth language probability and the confusion relationship can be combined to control the size of negative samples, thereby improving the training efficiency and training effect of the constraint sub-model.

[0127] Based on the speech recognition method of the present disclosure, the present disclosure further provides a speech recognition device, which will be described in detail below with reference to FIG.

[0128] FIG. 10 is a structural block diagram of a speech recognition device according to an embodiment of the present disclosure.

[0129] As shown in FIG. 10, the speech recognition device 1000 of this embodiment may include an acoustic probability acquisition module 1010, an initial probability acquisition module 1020, an extended relation acquisition module 1030, a probability adjustment module 1040, and a text identification module 1050.

[0130] The acoustic probability obtaining module 1010 processes the speech data to be recognized and the first text segment obtained by the recognition using the acoustic model to obtain the acoustic probability of each of the plurality of candidate text segments. In one embodiment, the acoustic probability obtaining module 1010 may be configured to perform the operation S210 described above, and will not be described here.

[0131] The initial probability acquisition module 1020 processes the first text segment using a first language sub-model in the language model to obtain initial language probabilities for each of the multiple candidate text segments. The extended relationship acquisition module 1030 processes the first text segment using a constraint sub-model in the language model to obtain an extensible relationship for the first text segment for each of the multiple candidate text segments. The probability adjustment module 1040 adjusts the initial language probabilities for the candidate text segments based on the extensible relationships to obtain first language probabilities for each of the multiple candidate text segments, where the constraint sub-model is obtained based on text training in a predetermined text set. In one embodiment, the initial probability acquisition module 1020, the extended relationship acquisition module 1030, and the probability adjustment module 1040 can respectively perform operations S220 to S240 described above, and descriptions thereof will be omitted here.

[0132] The text identification module 1050 identifies a target text segment among the plurality of candidate text segments based on the first language probability and the acoustic probability to obtain a text sequence for the speech data to be recognized. In one embodiment, the text identification module 1050 may be configured to perform the operation S250 described above, and will not be described here.

[0133] According to an embodiment of the present disclosure, the initial probability obtaining module 1020 may include: an embedding processing submodule for performing an embedding process on the first text segment to obtain text embedding features; a feature identification submodule for identifying first discriminative features of a category to which the first text segment belongs; and a first probability identification submodule for processing the features after fusing the text embedding features and the first discriminative features using a first language sub-model to obtain initial language probabilities for each of a plurality of candidate text segments.

[0134] According to an embodiment of the present disclosure, the language model further includes a second language sub-model arranged in parallel with the first language sub-model. The apparatus further includes an implicit expression acquisition module that inputs text embedding features to the second language sub-model and acquires a first implicit expression of the first text segment. The first language sub-model includes a first feature extraction network and a first prediction network. The first probability identification sub-module may include: an implicit expression acquisition means that inputs features obtained by fusing the text embedding features and the first discriminative features to the first feature extraction network to acquire a second implicit expression; and a first probability acquisition means that inputs features obtained by fusing the first implicit expression and the second implicit expression to the first prediction network to acquire an initial language probability for each of a plurality of candidate text segments. Here, the second language sub-model is obtained by training using sample texts of a plurality of predetermined categories.

[0135] According to an embodiment of the present disclosure, the second language sub-model includes a second feature extraction network and a second prediction network. The implicit representation acquisition module inputs text embedding features into the second feature extraction network to obtain a second implicit representation. The apparatus 1000 may further include a first probability acquisition module for inputting the second implicit representation into the second prediction network to obtain a second language probability for each of a plurality of candidate text segments. The text identification module 1050 is further configured to identify a target text segment based on the second language probability, the first language probability, and the acoustic probability.

[0136] According to an embodiment of the present disclosure, the language model further includes a third language sub-model arranged in parallel with the first language sub-model. The apparatus 1000 may further include a sign feature identification module for identifying second sign features representing a source of speech data to be recognized, and a second probability acquisition module for processing the text-embedding features and the second sign features after fusing them using the third language sub-model to obtain third language probabilities for each of a plurality of candidate text segments. The text identification module 1050 is further configured to identify a target text segment based on the third language probabilities, the first language probabilities, and the acoustic probabilities.

[0137] According to an embodiment of the present disclosure, the third language sub-model includes a third feature extraction network and a third prediction network, and the second probability acquisition module may include: an implicit representation acquisition sub-module for inputting features obtained by fusing the text embedding features and the second discriminative features into the third feature extraction network to obtain a third implicit representation; and a first probability acquisition sub-module for inputting features obtained by fusing the first implicit representation and the third implicit representation into the third prediction network to obtain a third language probability for each of a plurality of candidate text segments.

[0138] According to an embodiment of the present disclosure, when the first text segment is a starting identifier of a text, the category to which the first text segment belongs includes a plurality of predetermined categories, and the first probability identification submodule may include: a feature fusion means for fusing the text embedding feature and the discriminative feature of each predetermined category to obtain a first fused feature, and a second probability obtaining means for processing the first fused feature using a first language submodel to obtain an initial language probability for each of the plurality of candidate text segments.

[0139] According to an embodiment of the present disclosure, the plurality of candidate text segments includes a plurality of first candidate segments representing candidate words. The text identification module 1050 may include: a designated segment identification submodule configured to query a predetermined text list based on the first text segment and identify a first designated segment in the plurality of first candidate segments; a first segment identification submodule configured to identify that a text formed by combining the first text segment and the first designated segment belongs to the predetermined text list; and a first segment identification submodule configured to identify a target text segment in the plurality of first candidate segments based on a first linguistic probability and an acoustic probability of the first designated segment.

[0140] According to an embodiment of the present disclosure, the predetermined text list includes a plurality of texts and a discrimination weight for each text in the plurality of texts, where the discrimination weight indicates the recognition difficulty of the text. The first segment identification submodule includes: a first identification means for identifying a first target text to which a text formed by combining the first text segment and the first specified segment in the predetermined text list belongs; and a second identification means for identifying a target text segment among the plurality of candidate text segments based on the discrimination weight of the first target text, the first linguistic probability and the acoustic probability of the first specified segment.

[0141] According to an embodiment of the present disclosure, the plurality of candidate text segments further includes a plurality of second candidate segments indicating candidate slots, where the candidate slots include the incoming slot. The text identification module 1050 may include: a slot identification submodule for identifying a target slot belonging to a predetermined text list in the incoming slot; a second probability identification submodule for processing the features obtained based on the third identifying feature of the target slot and the beginning identifier of the text using a language model to obtain a fourth language probability for the target slot in each of the plurality of first candidate segments; and a second segment identification submodule for identifying a target text segment in the plurality of first candidate segments based on the fourth language probability, the first language probability, and the acoustic probability.

[0142] According to an embodiment of the present disclosure, the candidate slots further include an outgoing slot. The text identification module 1050 may further include: a fusion submodule for fusing the first discriminative feature of the category to which the first text segment belongs and the fourth discriminative feature of the slot corresponding to the text segment at the last position in the first text segment to obtain a second fused feature; a second probability identification submodule for processing the second fused feature using a language model to obtain a fifth language probability for each outgoing slot of the plurality of first candidate segments; and a third segment identification submodule for identifying a target text segment in the plurality of first candidate segments based on the fifth language probability, the fourth language probability, the first language probability, and the acoustic probability.

[0143] According to an embodiment of the present disclosure, the slot identification submodule may include: an initial slot identification means configured to identify a slot belonging to a predetermined text list in the incoming slot and obtain an initial slot; and a target slot identification means configured to identify a target slot in the initial slot based on a difference between a first language probability of a second candidate segment indicating the initial slot and the first language probabilities of the plurality of first candidate segments, wherein the first language probability of the second candidate segment indicating the target slot is greater than the first language probability of the second candidate segment indicating a slot other than the target slot in the initial slot.

[0144] According to an embodiment of the present disclosure, the second segment identification submodule may include: a third identification means configured to query a predetermined text list based on the first text segment to obtain a second target text and a second designated segment in a plurality of first candidate segments; a first text segment indicating that a text stitched with a target candidate segment corresponding to a target slot corresponding to the second designated segment belongs to the second target text; a probability identification means configured to obtain an initial probability of the target candidate segment based on the discrimination weight of the second target text and a first linguistic probability of the target candidate segment; and a segment identification means configured to identify a target text segment in the second designated segment based on the initial probability and a fourth linguistic probability of the second designated segment.

[0145] According to an embodiment of the present disclosure, the above-mentioned apparatus 1000 may further include: a table query module for querying a text segment table for the candidate slot based on the slot text segment in response to the text sequence including a slot text segment belonging to the candidate slot; a candidate segment identification module for identifying a text segment having the greatest similarity to the slot text segment in the text segment table as a candidate segment in response to the slot text segment not belonging to the text segment table; and a recognition result acquisition module for replacing the slot text segment in the text sequence with the candidate segment to obtain a recognition result for the speech data to be recognized.

[0146] According to an embodiment of the present disclosure, the device 1000 may further include: a probability storage module for storing a linguistic probability obtained by processing the language model for the first target feature in a buffer in response to the number of times the first target feature has been processed using the language model reaching a predetermined number; a buffer query module for querying the buffer based on the second target feature in response to the need to process a second target feature using the language model; and a probability reading module for reading the linguistic probability for the second target feature from the buffer in response to the linguistic probability for the second target feature being stored in the buffer, and completing processing for the second target feature using the language model, wherein the first target feature and the second target feature include any one of a text embedding feature of the first text paragraph, a feature obtained by fusing a text embedding feature and a category discriminative feature, a feature obtained by fusing a text embedding feature and a data source discriminative feature, and a feature obtained by fusing a text embedding feature and a slot discriminative feature.

[0147] According to an embodiment of the present disclosure, the operation of identifying a target text segment among a plurality of candidate text segments based on the first language probability and the acoustic probability is performed by a graphics processor provided in the electronic device.

[0148] According to an embodiment of the present disclosure, the samples for training the constraint sub-model include positive samples and negative samples, and the positive samples include text in a predetermined text set. The apparatus further includes a negative sample obtaining module for adjusting the predetermined text to obtain a negative sample based on a second text segment among a plurality of candidate text segments that does not match the text segment at the target position in the predetermined text.

[0149] According to an embodiment of the present disclosure, the negative sample obtaining module includes: a fourth segment identifying sub-module for identifying a segment to be replaced in the second text segment based on a confusion relationship between the second text segment and the text segment at the target position in the specified text; and a first replacement sub-module for replacing the text segment at the target position in the specified text with the segment to be replaced to obtain a negative sample.

[0150] According to an embodiment of the present disclosure, the negative sample acquisition module includes: a second replacement submodule for replacing a text segment at a target position in a predetermined text with a second text segment to obtain candidate samples; a second probability acquisition submodule for processing each sample in the candidate samples using a first language submodel to obtain a sixth language probability for each sample; and a sample screening submodule for screening the candidate samples based on the sixth language probability to obtain negative samples.

[0151] According to an embodiment of the present disclosure, the negative sample acquisition module may include a third replacement sub-module for replacing a text segment at a target position in a given text with a second text segment to obtain an initial text, and a segment removal sub-module for removing a text segment after the target position in the initial text to obtain a negative sample.

[0152] In addition, in the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, application, etc. of such user personal information shall all comply with the provisions of relevant laws, take necessary security measures, and not violate public order and morals. In the technical solution disclosed herein, the user's approval or permission shall be obtained before obtaining or collecting the user's personal information.

[0153] According to an embodiment of the present disclosure, the present disclosure provides an electronic device, a readable storage medium, and a computer program. M More to offer.

[0154] 11 is a schematic block diagram illustrating an example of an electronic device 1100 capable of implementing the speech recognition method of the present disclosure. The electronic device may represent various types of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various types of mobile devices, such as personal digital assistants, mobile phones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are merely exemplary and do not limit the implementation of the present disclosure as described and / or claimed herein.

[0155] 11, electronic device 1100 may include a computing means 1101 that performs various appropriate operations and processes based on a computer program stored in a read-only memory (ROM) 1102 or loaded from a storage means 1108 into a random access memory (RAM) 1103. RAM 1103 may further store various programs and data necessary for the operation of electronic device 1100. The computing means 1101, ROM 1102, and RAM 1103 are interconnected by a bus 1104. An input / output interface 1105 is also connected to bus 1104.

[0156] The components of the electronic device 1100 are connected to an I / O interface 1105, which includes input means 1106 such as a keyboard, a mouse, etc., output means 1107 such as various types of displays, speakers, etc., storage means 1108 such as a magnetic disk, an optical disk, etc., and communication means 1109 such as a network card, a modem, a wireless communication transceiver, etc. The communication means 1109 enables the electronic device 1100 to exchange information / data with other devices via a computer network such as the Internet or various types of telecommunications networks.

[0157] The computing means 1101 may be a general-purpose and / or dedicated processing module having various processing and computing capabilities. Examples of the computing means 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, computing means for executing various machine learning model algorithms, a digital signal processor (DSP), any suitable processor, controller, microcontroller, etc. The computing means 1101 performs the methods and processes described above, such as the speech recognition method. For example, in one embodiment, the speech recognition method is implemented as a computer software program that is temporarily contained in a machine-readable medium, such as the storage means 1108. In one embodiment, some or all of the computer program is loaded and / or installed into the electronic device 1100 via the ROM 1102 and / or the communication means 1109. When the computer program is loaded into the RAM 1103 and executed by the computing means 1101, it may perform one or more steps of the speech recognition method described above. Alternatively, in other embodiments, the computing means 1101 is configured to perform the speech recognition method in any other suitable manner (eg, firmware).

[0158] Various embodiments of the systems and techniques described herein may be realized in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be embodied in one or more computer programs that may be executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor, and that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0159] Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, so that when the program code is executed by the processor or controller, the functions and operations specified in the flowcharts and / or block diagrams are performed. The program code may be executed entirely on a device, partially on a device, partially on a device as a separate software package, and partially on a remote device, or entirely on a remote device or server.

[0160] In the context of the present disclosure, a machine-readable medium may be a tangible medium, and may contain or store a program for use in or in connection with an instruction execution system, device, or electronic device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or electronic device, or any suitable combination of the above. More specific examples of machine-readable storage media include an electrical connection of one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0161] To provide for user interaction, a computer may implement the systems and techniques described herein and include a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which a user may provide input to the computer. Other types of devices may also provide for user interaction; for example, the feedback provided to the user may be any form of sensing feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and may receive input from the user in any form (including voice input, audio input, or tactile input).

[0162] The systems and techniques described herein may be implemented in a computing system including background components (e.g., a data server), or middleware components (e.g., an application server), or front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or any combination of such background, middleware, or front-end components. The components of the system may be connected to each other by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include, by way of example, a local area network (LAN), a wide area network (WAN), and the Internet.

[0163] The computer system may include a client and a server. The client and server are generally separate and typically interact via a communication network. The relationship between the client and the server is established by a computer program running on the corresponding computer and having a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in a cloud computing service system and solves the drawbacks of traditional physical hosts and VPS services (abbreviated as "Virtual Private Server" or "VPS"), such as high management difficulty and poor service scalability. The server may be a server in a distributed system or a server connected to a blockchain.

[0164] It should be understood that various types of flows shown above may be used, and operations may be rearranged, added, or deleted. For example, the operations described in this disclosure may be performed in parallel, sequentially, or in a different order, as long as the desired results of the invention of this disclosure are achieved, and this specification is not limited thereto.

[0165] The above specific embodiments do not limit the scope of protection of the present disclosure. Those skilled in the art should understand that various modifications, combinations, subcombinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present disclosure should be included within the scope of protection of the present disclosure.< / eos> < / sos> < / sos> < / sos> < / sos> < / eos> < / sos> < / sos> < / sos> < / sos> < / eos> < / eos> < / sos> < / sos> < / eos>

Claims

1. 1. A speech recognition method, comprising: processing the speech data to be recognized and the first text segment obtained by the recognition using the acoustic model to obtain acoustic probabilities for each of a plurality of candidate text segments; processing the first text segment using a first language sub-model in a language model to obtain initial language probabilities for each of a plurality of the candidate text segments; processing the first text segment using a constraint sub-model in the language model to obtain an extensible relationship to the first text segment in each of a plurality of the candidate text segments; adjusting initial language probabilities of the candidate text segments based on the extensible relationship to obtain first language probabilities for each of a plurality of the candidate text segments; identifying a target text segment among the plurality of candidate text segments based on the first language probability and the acoustic probability to obtain a text sequence for the speech data to be recognized; wherein the constraint sub-model is trained based on text in a predetermined text set. Speech recognition methods.

2. Processing the first text segment with a first language sub-model in a language model to obtain initial language probabilities for each of a plurality of the candidate text segments includes: performing an embedding process on the first text segment to obtain text embedding features; identifying a first distinguishing feature of a category to which the first text segment belongs; and processing the text embedding features and the first discriminative features after fusing them using the first language sub-model to obtain initial language probabilities for each of the plurality of candidate text segments. The method of claim 1.

3. the language model further includes a second language sub-model arranged in parallel with the first language sub-model; The method comprises: inputting the text embedding features into the second language sub-model to obtain a first implicit representation of the first text segment; the first language sub-model includes a first feature extraction network and a first prediction network; processing the text embedding features and the first discriminative features fused with the first language sub-model to obtain initial language probabilities for each of the plurality of candidate text segments; inputting the fused text embedding feature and the first discriminative feature into the first feature extraction network to obtain a second implicit representation; inputting features obtained by fusing the first implicit representation and the second implicit representation into the first prediction network to obtain initial language probabilities for each of the plurality of candidate text segments; Here, the second language sub-model is obtained by training using sample texts of a plurality of predetermined categories. The method of claim 2.

4. the second language sub-model includes a second feature extraction network and a second prediction network; inputting the text embedding features into the second language sub-model to obtain a first implicit representation of the first text segment includes inputting the text embedding features into the second feature extraction network to obtain the second implicit representation; The method comprises: inputting the second implicit representation into the second prediction network to obtain second language probabilities for each of a plurality of the candidate text segments; and identifying the target text segment based on the second language probability, the first language probability, and the acoustic probability. The method of claim 3.

5. the language model further includes a third language sub-model arranged in parallel with the first language sub-model; The method comprises: identifying a second distinguishing feature indicative of the source of the audio data to be identified; processing the text embedding features and the second discriminative features fused together using the third language sub-model to obtain third language probabilities for each of the plurality of candidate text segments; identifying the target text segment based on the third language probability, the first language probability, and the acoustic probability. The method of claim 3.

6. the third language sub-model includes a third feature extraction network and a third prediction network; processing the fused text embedding features and the second discriminative features using the third language sub-model to obtain third language probabilities for each of the plurality of candidate text segments; inputting the fusion features of the text embedding features and the second discriminative features into the third feature extraction network to obtain a third implicit representation; and inputting features obtained by fusing the first implicit representation and the third implicit representation into the third prediction network to obtain a third language probability for each of the plurality of candidate text segments. The method of claim 5.

7. If the first text segment is a start identifier of a text, the category to which the first text segment belongs includes a plurality of predetermined categories; processing the fused text embedding features and the first discriminative features using the first language sub-model to obtain initial language probabilities for each of the plurality of candidate text segments; For each predetermined category, fusing the text embedding feature and the discriminative feature of each predetermined category to obtain a first fused feature; and processing the first fused features with the first language sub-model to obtain initial language probabilities for each of a plurality of the candidate text segments. The method of claim 2.

8. the plurality of candidate text segments include a plurality of first candidate segments representing candidate words; Identifying a target text segment among the plurality of candidate text segments based on the first language probability and the acoustic probability to obtain a text sequence for the speech data to be recognized includes: querying a predetermined text list based on the first text segment, identifying a first designated segment among the plurality of first candidate segments, and a text formed by combining the first text segment and the first designated segment belongs to the predetermined text list; and identifying a target text segment among the plurality of first candidate segments based on the first language probability and the acoustic probability of the first specified segment. The method of claim 1.

9. the predetermined text list includes a plurality of texts and a recognition weight for each text in the plurality of texts; the recognition weight indicates the difficulty of recognizing the text; Identifying a target text segment from the plurality of first candidate segments based on a first language probability and the acoustic probability of the first specified segment includes: Identifying a first target text to which the text obtained by combining the first text segment and the first specified segment in the predetermined text list belongs; identifying a target text segment among the plurality of candidate text segments based on a recognition weight of the first target text, a first language probability of the first specified segment, and the acoustic probability. The method of claim 8.

10. the plurality of candidate text segments further include a plurality of second candidate segments indicating candidate slots; the candidate slots include incoming slots; Identifying a target text segment among the plurality of candidate text segments based on the first language probability and the acoustic probability to obtain a text sequence for the speech data to be recognized includes: identifying a target slot in the incoming slot that belongs to the predetermined text list; using the language model to process features obtained based on a third distinguishing feature of the target slot and a beginning of text identifier to obtain a fourth language probability for the target slot for each of a plurality of the first candidate segments; and identifying a target text segment among the plurality of first candidate segments based on the fourth language probability, the first language probability, and the acoustic probability. The method of claim 8.

11. the candidate slots further include an outgoing slot; Identifying a target text segment among the plurality of candidate text segments based on the first language probability and the acoustic probability to obtain a text sequence for the speech data to be recognized includes: Fusing a first discriminative feature of the category to which the first text segment belongs and a fourth discriminative feature of a slot corresponding to the text segment at the end of the first text segment to obtain a second fused feature; processing the second fused features with the language model to obtain a fifth language probability for the outgoing slot for each of a plurality of the first candidate segments; and identifying a target text segment among the plurality of first candidate segments based on the fifth language probability, the fourth language probability, the first language probability, and the acoustic probability. The method of claim 10.

12. Identifying a target slot belonging to the predetermined text list in the incoming slot includes: Identifying slots belonging to the predetermined text list in the incoming slots and obtaining an initial slot; identifying a target slot in the initial slot based on a difference between a first language probability of a second candidate segment representing the initial slot and a first language probability of a plurality of the first candidate segments; Here, the first language probability of the second candidate segment indicating the target slot is greater than the first language probability of the second candidate segment indicating a slot other than the target slot in the initial slot. The method of claim 10.

13. Identifying a target text segment among the plurality of first candidate segments based on the fourth language probability, the first language probability, and the acoustic probability includes: Querying the predetermined text list based on the first text segment to obtain a second target text and a second designated segment in the plurality of first candidate segments, and determining that a text obtained by combining the first text segment and a target candidate segment indicating a target slot corresponding to the second designated segment belongs to the second target text; obtaining an initial probability of the target candidate segment based on a recognition weight of the second target text and a first language probability of the target candidate segment; and identifying a target text segment in the second specified segment based on the initial probability and a fourth language probability for the second specified segment. The method of claim 10.

14. responsive to the text sequence including a slot text segment belonging to the candidate slot, querying a text segment table for the candidate slot based on the slot text segment; In response to the slot text segment not belonging to the text segment table, identifying a text segment in the text segment table that has a greatest similarity to the slot text segment as a candidate segment; and replacing the slot text segment in the text sequence with the candidate segment to obtain a recognition result for the speech data to be recognized. The method of claim 10.

15. storing language probabilities obtained by processing the first target feature using the language model in a cache in response to the number of times the first target feature has been processed using the language model reaching a predetermined number; responsive to a need to process a second target feature using the language model, querying the cache based on the second target feature; and in response to the cache storing the language probability for the second target feature, reading the language probability for the second target feature from the cache and completing processing of the second target feature using the language model; Here, the first target feature and the second target feature include any one of a text embedding feature of the first text segment, a feature obtained by combining the text embedding feature and a category discriminant feature, a feature obtained by combining the text embedding feature and a data source discriminant feature, and a feature obtained by combining the text embedding feature and a slot discriminant feature. The method according to any one of claims 10 to 13.

16. Identifying a target text segment from the plurality of candidate text segments based on the first language probability and the acoustic probability is performed by a graphics processor in an electronic device. The method according to any one of claims 10 to 13.

17. the samples for training the constraint sub-model include positive samples and negative samples; the positive samples include text in the predetermined text set; The negative sample is obtained by adjusting the given text based on a second text segment among the plurality of candidate text segments that does not match the text segment at the target position in the given text to obtain the negative sample. The method of claim 1.

18. adjusting the predetermined text based on a second text segment among the plurality of candidate text segments that does not match the text segment at the target position in the predetermined text to obtain the negative sample; identifying a segment to replace in the second text segment based on a confusion relationship between the second text segment and a text segment at the target position in the given text; and replacing the text segment at the target position in the given text with the segment to be replaced to obtain the negative sample.

18. The method of claim 17.

19. adjusting the predetermined text based on a second text segment among the plurality of candidate text segments that does not match the text segment at the target position in the predetermined text to obtain the negative sample; replacing a text segment at the target position in the given text with the second text segment to obtain a candidate sample; processing each sample in the candidate samples with the first language sub-model to obtain a sixth language probability for each sample; and screening the candidate samples based on the sixth language probability to obtain the negative samples.

18. The method of claim 17.

20. adjusting the predetermined text based on a second text segment among the plurality of candidate text segments that does not match the text segment at the target position in the predetermined text to obtain the negative sample; replacing the text segment at the target position in the given text with the second text segment to obtain an initial text; removing a text segment after the target position in the initial text to obtain the negative sample.

18. The method of claim 17.

21. A speech recognition device, an acoustic probability obtaining module for processing the speech data to be recognized and the first text segment obtained by the recognition using the acoustic model to obtain an acoustic probability for each of a plurality of candidate text segments; an initial probability obtaining module that processes the first text segment using a first language sub-model in a language model to obtain initial language probabilities for each of the plurality of candidate text segments; an extended relationship acquisition module that processes the first text segment using a constraint sub-model in the language model to acquire an extensible relationship to the first text segment for each of a plurality of the candidate text segments; a probability adjustment module that adjusts initial language probabilities of the candidate text segments based on the extensible relationship to obtain first language probabilities for each of a plurality of the candidate text segments; a text identification module that identifies a target text segment among the plurality of candidate text segments based on the first language probability and the acoustic probability to obtain a text sequence for the speech data to be recognized; wherein the constraint sub-model is obtained by training based on text in a predetermined text set. Voice recognition device.

22. at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor such that the at least one processor can perform the method of any one of claims 1 to 14. electronic equipment.

23. A non-transitory computer-readable storage medium having computer instructions stored thereon, comprising: The computer instructions cause the computer to perform the method of any one of claims 1 to 14. storage medium.

24. A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 14. Computer program.

Images