Speech recognition method and device, electronic equipment and storage medium

Abstract

The invention provides a voice recognition method and device, electronic equipment and a storage medium, and relates to the technical field of voice recognition, and the method comprises the steps: obtaining at least one sample voice and a standard voice text corresponding to the sample voice; performing voice feature extraction on the sample voice to obtain a sample voice feature; inputting the sample voice features into the trained acoustic model to obtain acoustic modeling features output by the trained acoustic model; the acoustic modeling features and the standard voice text are input into a large language model, prediction probability distribution and prediction voice text output by the large language model are obtained, and the prediction probability distribution is used for indicating probability distribution of prediction words corresponding to the sample voice; and training a large language model based on the prediction probability distribution, the prediction voice text and the standard voice text, so that the trained large language model and the trained acoustic model have a voice recognition function. The method can improve the speech recognition accuracy.

Description

Technical Field

[0001] The present disclosure relates to the field of speech recognition technology, and in particular to a speech recognition method, device, electronic device and storage medium. Background Art

[0002] Speech recognition refers to converting a speech signal into corresponding text information. It is widely used in scenarios such as smart customer service, smart robots, smart phones, and smart homes. Currently, mainstream speech recognition systems usually only use one model to model speech recognition tasks, that is, input speech and output text. However, end-to-end speech recognition systems have the problem of weak semantic content and ignore semantic information, resulting in low speech recognition accuracy. Summary of the invention

[0003] The present disclosure provides a speech recognition method, device, electronic device and storage medium, which can improve the accuracy of speech recognition. The technical solution is as follows:

[0004] According to one aspect of the present disclosure, a speech recognition method is provided, the method comprising:

[0005] Acquire at least one sample speech and a standard speech text corresponding to the sample speech;

[0006] Extracting speech features from the sample speech to obtain sample speech features;

[0007] Inputting the sample speech features into the trained acoustic model to obtain acoustic modeling features output by the trained acoustic model;

[0008] Inputting the acoustic modeling features and the standard speech text into a large language model to obtain a predicted probability distribution and a predicted speech text output by the large language model, wherein the predicted probability distribution is used to indicate a probability distribution of predicted words corresponding to the sample speech;

[0009] Based on the predicted probability distribution, the predicted speech text and the standard speech text, the large language model is trained so that the trained large language model and the trained acoustic model have speech recognition functions.

[0010] According to another aspect of the present disclosure, a speech recognition method is provided, the method comprising:

[0011] Extract speech features of the target speech to be recognized to obtain target speech features;

[0012] Inputting the target speech feature into the trained acoustic model to obtain the target modeling feature output by the trained acoustic model;

[0013] Inputting the target modeling feature and the start label into the trained large language model, obtaining a target probability distribution and a first speech text output by the trained large language model, wherein the target probability distribution is used to indicate a probability distribution of predicted words corresponding to the target speech;

[0014] Decoding the target probability distribution to obtain a second speech text;

[0015] Based on the first voice text and / or the second voice text, a target voice text corresponding to the target voice is generated.

[0016] According to another aspect of the present disclosure, a speech recognition device is provided, the device comprising:

[0017] An acquisition module, used to acquire at least one sample speech and a standard speech text corresponding to the sample speech;

[0018] A feature extraction module, used to extract speech features from the sample speech to obtain sample speech features;

[0019] An acoustic processing module, used for inputting the sample speech features into the trained acoustic model to obtain acoustic modeling features output by the trained acoustic model;

[0020] A speech recognition module, used for inputting the acoustic modeling features and the standard speech text into a large language model to obtain a predicted probability distribution and a predicted speech text output by the large language model, wherein the predicted probability distribution is used to indicate a probability distribution of predicted words corresponding to the sample speech;

[0021] A training module is used to train the large language model based on the predicted probability distribution, the predicted speech text and the standard speech text, so that the trained large language model and the trained acoustic model have speech recognition function.

[0022] According to another aspect of the present disclosure, a speech recognition device is provided, the device comprising:

[0023] A feature extraction module is used to extract speech features of the target speech to be recognized and obtain target speech features;

[0024] An acoustic processing module, used for inputting the target speech feature into the trained acoustic model to obtain the target modeling feature output by the trained acoustic model;

[0025] A speech recognition module, used to input the target modeling features and the start tag into the trained large language model, and obtain a target probability distribution and a first speech text output by the trained large language model, wherein the target probability distribution is used to indicate a probability distribution of predicted words corresponding to the target speech;

[0026] A decoding module, used for decoding the target probability distribution to obtain a second speech text;

[0027] A generation module is used to generate a target voice text corresponding to the target voice based on the first voice text and / or the second voice text.

[0028] According to one aspect of the present disclosure, an electronic device is provided, including: a processor and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the speech recognition method as described above.

[0029] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the speech recognition method as described above.

[0030] According to another aspect of the present disclosure, a computer program product is provided, the computer program product comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned speech recognition method.

[0031] The beneficial effects brought by the technical solution provided by the embodiments of the present disclosure include at least:

[0032] By building a new speech recognition model (acoustic model and large language model), and inputting the acoustic modeling features and standard speech text output by the acoustic model into the large language model, the acoustic modeling results (prediction probability distribution) and text prediction results (prediction speech text) are output, and then the large language model is trained based on the prediction results and standard speech text. This allows the new speech recognition model to have both acoustic and language characteristics, that is, it ensures the acoustic information and uses the language information of the large language model, thereby improving the speech recognition accuracy of the trained speech recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0034] Figure 1 A schematic diagram showing an example system in which various methods described herein may be implemented according to an exemplary embodiment of the present disclosure;

[0035] Figure 2 A flow chart of a speech recognition method according to an exemplary embodiment of the present disclosure is shown;

[0036] Figure 3 is a schematic diagram of a training process of a speech recognition model provided by an exemplary embodiment of the present disclosure;

[0037] Figure 4 A flowchart of another speech recognition method according to an exemplary embodiment of the present disclosure is shown;

[0038] Figure 5 is a schematic diagram of an architecture of an acoustic model provided by an exemplary embodiment of the present disclosure;

[0039] Figure 6 It is a schematic diagram of the principle of a large language model in text autoregressive decoding provided by an exemplary embodiment of the present disclosure;

[0040] Figure 7 is a schematic diagram of a training process of another speech recognition model provided by an exemplary embodiment of the present disclosure;

[0041] Figure 8 A flowchart of another speech recognition method according to an exemplary embodiment of the present disclosure is shown;

[0042] Fig. 9 is a schematic diagram of a speech recognition process provided by an exemplary embodiment of the present disclosure;

[0043] Fig.10 is a structural schematic diagram of a speech recognition device provided by an embodiment of the present disclosure;

[0044] Fig.11 is a structural schematic diagram of another speech recognition device provided by an embodiment of the present disclosure;

[0045] Fig.12 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0046] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0047] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0048] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more". The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are for illustrative purposes only, and are not used to limit the scope of these messages or information.

[0049] The solution of the present invention is described below with reference to the accompanying drawings, and the technical solution provided by the embodiment of the present invention is explained in detail through specific embodiments and their application scenarios.

[0050] Figure 1 Schematic diagram of an example system in which various methods described herein may be implemented according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the system includes a first device 110 and a second device 120 .

[0051] The first device 110 is a training device for training a speech recognition model. After the speech recognition model is trained, the first device 110 may send the trained speech recognition model to the second device 120 so as to deploy the speech recognition model in the second device 120. The second device 120 is a device that uses the speech recognition model for speech recognition.

[0052] In the model training stage: the sample speech and its corresponding standard speech text are used as training data 111 to train the speech recognition model 112, and the speech recognition model 112 includes two parts: an acoustic model and a large language model.

[0053] In the model application stage: the target speech 121 to be recognized can be input into the trained speech recognition model 122 to obtain the target speech text 123 outputted by it.

[0054] Optionally, the first device 110 and the second device 120 may be computer devices with machine learning capabilities, for example, the computer device may be a terminal or a server.

[0055] Optionally, the first device 110 and the second device 120 may be the same computer device, or the first device 110 and the second device 120 may be different computer devices. Furthermore, when the first device 110 and the second device 120 are different devices, the first device 110 and the second device 120 may be devices of the same type, such as the first device 110 and the second device 120 may both be servers; or the first device 110 and the second device 120 may also be devices of different types. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal may be a smart phone, a vehicle-mounted terminal, a smart TV, a wearable device, a tablet computer, a laptop, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and the present disclosure is not limited thereto.

[0056] Please refer to Figure 2 , which shows a flow chart of a speech recognition method according to an exemplary embodiment of the present disclosure. Figure 1 The first device 110 shown in the figure is used as an example for explanation. Figure 2 As shown, the method includes:

[0057] Step 201: Obtain at least one sample speech and a standard speech text corresponding to the sample speech.

[0058] In order to enable the constructed model (i.e., speech recognition model) to have speech-to-text speech recognition function, before training the model, it is necessary to first collect a large number of training data pairs, and each training data pair includes a speech and its corresponding real speech text, and the real speech text is the learning target of the model. In a possible implementation, during the training of the speech recognition model, it is first necessary to obtain a number of sample speech and the standard speech text corresponding to each sample speech.

[0059] Step 202: extract speech features from the sample speech to obtain sample speech features.

[0060] Before inputting the sample speech into the speech recognition model, the sample speech needs to be preprocessed first. The preprocessing process can also be called speech feature extraction, so as to extract a continuous section of sample speech as speech features, that is, to obtain the sample speech features corresponding to each section of sample speech. Among them, the algorithms used for speech feature extraction may include: MFCC (Mel Frequency Cepstral Coefficient), LPC (Linear Prediction Coefficients), LSF (Line Spectral Frequencies), DWT (Discrete Wavelet Transform), PLP (Perceptual Linear Predictive), LPCC (Linear Predictive Cepstral Coefficient), etc. This embodiment does not limit the specific method adopted for speech feature extraction.

[0061] Exemplarily, if the sample speech is a speech signal of duration t1, the sample speech signal is first divided into T frames according to the preset duration, and each frame uses a speech feature extraction algorithm to extract an N-dimensional speech vector, and the speech features are usually Fbank and MFCC features, so as to convert the sample speech of duration t1 into a T*N matrix. For example, if t1 is 1s, T is 100, and N is 80, then the 1-second speech is converted into a 100*80 matrix, and the sample speech features are obtained. It should be noted that the values ​​of T and N are not fixed and are set by the model requirements.

[0062] Step 203: input the sample speech features into the trained acoustic model to obtain the acoustic modeling features output by the trained acoustic model.

[0063] In order to improve the semantic recognition ability of the speech recognition model and take into account the acoustic information, the embodiment of the present disclosure adds a large language model to the traditional semantic recognition model (acoustic model), so that the updated speech recognition model has both acoustic and language characteristics, that is, it ensures the acoustic information and uses the language information of the large language model. In other words, the speech recognition model (or speech recognition system) in the embodiment of the present disclosure includes two parts: acoustic model + large language model, and the role of the two models in the speech recognition task is specially set to achieve a better speech recognition effect.

[0064] Since the role of the acoustic model in the speech recognition task has not changed, it is still based on the input speech features to perform acoustic high-dimensional feature modeling and predict the probability distribution of words corresponding to the speech frame. Therefore, the acoustic model is still trained in the traditional way. After the acoustic model training is completed, the trained acoustic model is used to participate in the training process of the speech recognition model.

[0065] Specifically, the training process of the acoustic model may include the following steps A to C.

[0066] Step A: input the sample speech into the acoustic model to obtain the sample modeling features output by the acoustic model.

[0067] Step B: decode the sample modeling features to obtain the predicted speech text.

[0068] Step C, training the acoustic model based on the difference between the predicted speech text and the standard speech text.

[0069] Among them, the training samples of the acoustic model can also use several sample speech and its corresponding standard speech text. In each round of training, the speech feature of the sample speech is first extracted to obtain the sample speech feature, and the sample speech feature is input into the acoustic model to obtain the sample modeling feature output by the acoustic model, which represents the probability distribution of the predicted word corresponding to each sample speech feature; the sample modeling feature is then decoded to obtain the predicted speech text; then based on the difference (or loss) between the predicted speech text and the standard speech text, the parameters of the acoustic model are updated; after multiple rounds of training, until the loss between the predicted speech text and the standard speech text is less than the preset value, it is determined that the acoustic model training is completed.

[0070] Optionally, the sample speech and standard speech text used to train the acoustic model can be reused to train the speech recognition model in the embodiment of the present disclosure; or, different training samples can be used in the two training stages.

[0071] In one possible implementation, after the acoustic model training is completed, the sample speech features are input into the trained acoustic model to perform acoustic high-dimensional feature modeling and probability distribution prediction of words corresponding to the speech frame, and the acoustic modeling features output by the trained acoustic model can be obtained.

[0072] Since the acoustic model generally includes a multi-layer network, in which the intermediate layer network outputs high-dimensional features and the last layer is used to output probability distribution; however, the output of each layer contains acoustic information. Therefore, the acoustic modeling features output by the acoustic model used in the embodiments of the present disclosure can be high-dimensional features output by the intermediate layer network, or can be the probability distribution output by the last layer; it can also be the concatenation of high-dimensional features and probability distribution output by the intermediate layer network; or, when there are multiple layers of the intermediate layer network, the acoustic modeling features can also be the concatenation of at least two high-dimensional features output by at least two intermediate layer networks, or can be the weighted processing of at least two high-dimensional features output by at least two intermediate layer networks.

[0073] Step 204, input the acoustic modeling features and the standard speech text into the large language model to obtain the predicted probability distribution and the predicted speech text output by the large language model, where the predicted probability distribution is used to indicate the probability distribution of the sample speech corresponding to the predicted words.

[0074] Since the output of the acoustic model can provide acoustic information, and the large language model itself has strong semantic understanding ability, in order to take into account both acoustic information and language information in the final speech recognition stage. In a possible implementation, both the acoustic modeling features and the standard speech text are input into the large language model, and the large language model learns the acoustic information from the acoustic modeling features and the contextual semantic information from the standard speech text, and finally outputs the predicted probability distribution and the predicted speech text.

[0075] The predicted probability distribution is used to indicate the probability distribution of the predicted words corresponding to the sample speech. The difference from the probability distribution output by the last layer in the acoustic model is that the acoustic model is the predicted probability distribution of a single word, while the large language model takes into account the semantic information and outputs the probability distribution of words.

[0076] Optionally, if there is a difference between the input feature dimension of the large language model and the output feature dimension of the acoustic model, a mapping layer needs to be added between the acoustic model and the large language model. The mapping layer is used to change the feature dimension of the acoustic modeling feature so that the acoustic modeling feature can adapt to the model input requirements of the large language model. After obtaining the acoustic modeling features output by the trained acoustic model, the acoustic modeling features need to be input into the mapping layer for dimension mapping to obtain the acoustic modeling features after dimension mapping. The processed acoustic modeling features and the standard speech text are then input into the large language model for text prediction.

[0077] Step 205 , based on the predicted probability distribution, the predicted speech text and the standard speech text, the large language model is trained so that the trained large language model and the trained acoustic model have speech recognition function.

[0078] Since the acoustic model in the speech recognition model provided by the embodiment of the present disclosure is pre-trained, during the speech recognition model training process, it is only necessary to train the large language model based on the difference between the predicted results and the standard results, without updating the parameters of the acoustic model. In a possible implementation, the parameters of the large language model are updated based on the difference between the predicted probability distribution, the predicted speech text (prediction result) and the standard speech text (standard result). After multiple rounds of training, the trained acoustic model and the large language model have accurate speech recognition capabilities.

[0079] Please refer to Figure 3 , which is a schematic diagram of a training process of a speech recognition model provided by an exemplary embodiment of the present disclosure. Figure 3 As shown, firstly, speech features are extracted from the sample speech 301 to obtain the sample speech features 302, and the sample speech features 302 are input into the trained acoustic model 303 for acoustic feature modeling to obtain the acoustic modeling features 304; then the acoustic modeling features 304 and the standard speech text 305 are input into the large language model 306 to obtain the predicted probability distribution 307 and the predicted speech text 308 output by the large language model 306; and then the large language model 306 is trained based on the predicted probability distribution 307, the predicted speech text 308 and the standard speech text 305.

[0080] In summary, the disclosed embodiment provides a speech recognition method: by constructing a new speech recognition model (acoustic model and large language model), and inputting the acoustic modeling features and standard speech text output by the acoustic model into the large language model, outputting the acoustic modeling results (prediction probability distribution) and text prediction results (prediction speech text), and then training the large language model based on the prediction results and the standard speech text. This enables the new speech recognition model to have both acoustic and language characteristics, that is, it ensures the acoustic information and uses the language information of the large language model, thereby improving the speech recognition accuracy of the trained speech recognition model.

[0081] Considering that the input of the large language model is two types of information, and the large language model plays different roles in processing these two types of information, in each round of training, the acoustic modeling features and standard speech text are not input into the large language model at the same time, but there is a sequence of input.

[0082] Please refer to Figure 4 , which shows a flowchart of another speech recognition method according to an exemplary embodiment of the present disclosure. Figure 1 The first device 110 shown in the figure is used as an example for explanation. Figure 4 As shown, the method includes:

[0083] Step 401: Obtain at least one sample speech and a standard speech text corresponding to the sample speech.

[0084] Step 402: extract speech features from the sample speech to obtain sample speech features.

[0085] The implementation of step 401 and step 402 may refer to step 201 and step 202, and will not be described in detail in this embodiment.

[0086] Step 403: Input the sample speech features into the trained acoustic model to obtain at least two sample feature vectors outputted by at least two layers of modeling networks.

[0087] Among them, the trained acoustic model has at least two layers of modeling networks; after the sample speech features are input into the trained acoustic model, at least two sample feature vectors output by at least two layers of modeling networks can be obtained; that is, each layer of the modeling network outputs a sample feature vector.

[0088] Please refer to Figure 5 , which is a schematic diagram of the architecture of an acoustic model provided by an exemplary embodiment of the present disclosure. Figure 5 As shown, taking the acoustic model including three layers of modeling networks as an example, that is, the acoustic model includes a first layer network 501, a second layer network 502 and a third layer network 503; the sample speech feature 504 is input into the first layer network 501 to obtain a first feature vector 505 output by the first layer network 501, the first feature vector 505 is input into the second layer network 502 to obtain a second feature vector 506 output by the second layer network 502; the second feature vector 506 is input into the third layer network 503 to obtain a third feature vector 507 output by the third layer network 503 (and the third feature vector 507 represents the probability distribution of the word corresponding to the sample speech).

[0089] Step 404: perform concatenation or weighting processing on at least two sample feature vectors to obtain acoustic modeling features.

[0090] When the trained acoustic model includes at least two layers of modeling networks, the acoustic modeling features of the input large language model exist in the following situations: (1) taking the sample feature vector output by any intermediate layer modeling network as the acoustic modeling feature; (2) when the acoustic model includes multiple intermediate layer modeling networks (that is, the acoustic model includes at least three layers of modeling networks), taking the sample feature vectors output by at least two intermediate layer modeling networks for concatenation or weighting to obtain the acoustic modeling feature; (3) taking the sample feature vector output by the last layer modeling network as the acoustic modeling feature; (4) taking the sample feature vector output by at least one intermediate layer modeling network and the sample feature vector output by the last layer modeling network for concatenation or weighting to obtain the acoustic modeling feature.

[0091] In a possible implementation, when the trained acoustic model outputs at least two sample feature vectors, the at least two sample feature vectors are concatenated or weighted to obtain acoustic modeling features. The weight of each sample feature vector during weighted processing can be determined by the order of the modeling network, that is, the earlier the modeling network is in the acoustic model, the lower the weight of the sample feature vector output by it.

[0092] like Figure 5 As shown, the first eigenvector 505 can be used as an acoustic modeling feature; the second eigenvector 506 can also be used as an acoustic modeling feature; or the third eigenvector 507 can also be used as an acoustic modeling feature; or the first eigenvector 505 and the second eigenvector 506 can also be concatenated or weighted to obtain an acoustic modeling feature; or the first eigenvector 505, the second eigenvector 506 and the third eigenvector 507 can also be concatenated or weighted to obtain an acoustic modeling feature. When weighted, the weight of the first eigenvector 505 is lower than the weight of the second eigenvector 506, and the weight of the second eigenvector 506 is lower than the weight of the third eigenvector 507.

[0093] Step 405: Input the acoustic modeling features into the large language model to obtain the predicted probability distribution output by the large language model.

[0094] Among them, the large language model can be used as a part of the acoustic model in the speech recognition model to predict the probability distribution of the acoustic modeling features and output a predicted probability distribution similar to the final output of the acoustic model; it can also play the role of a language model to perform text autoregressive decoding to output predicted speech text. In order to allow the large language model to still refer to the acoustic information when performing text autoregressive decoding, the acoustic modeling features are preferentially input into the large language model for word probability prediction to obtain the output predicted probability distribution; at the same time, because the acoustic modeling features are preferentially input, the large language model itself also obtains acoustic information for subsequent text autoregressive decoding.

[0095] Step 406: input the standard speech text into the large language model to obtain the predicted speech text output by the large language model.

[0096] After the large language model obtains acoustic information through acoustic modeling features, the standard speech text can continue to be input into the large language model. The large language model learns semantic information from the standard speech text and combines it with the acoustic information to perform speech text prediction to obtain the predicted speech text output by the large language model.

[0097] Among them, the process of predicting speech text from the standard speech text output by the large language model is the process of text autoregressive decoding; and the process of text autoregressive decoding is: predicting the next word based on the previous word to fully learn the contextual semantic information; that is, each round of prediction gets a predicted word, and after multiple rounds of prediction, the final predicted speech text is obtained.

[0098] The corresponding step 406 may also be replaced by step 406A:

[0099] Step 406A, input the standard speech text into the large language model, obtain m predicted words and m predicted probabilities output by the large language model, and the m predicted words are used to be combined into the predicted speech text in sequence, where m is determined by the number of words in the standard speech text, and m is a positive integer.

[0100] Corresponding to inputting the standard voice text into the large language model, the predicted words predicted by the large language model based on each standard word in the standard voice text can be obtained. When the number of words in the standard voice text is m, m predicted words output by the large language model can be obtained; at the same time, when the large language model outputs m predicted words, it also outputs the prediction probability when predicting each predicted word based on the standard voice text; that is to say, after inputting the standard voice text into the large language model, m predicted words and m prediction probabilities output by the large language model can be obtained, and the m predicted words can be combined into a predicted voice text in the prediction order.

[0101] Specifically, similar to obtaining a predicted word in each round, in the text autoregressive decoding process, the complete standard speech text is not directly input into the large language model, but also needs to be input in multiple rounds. The corresponding step 406A may also include step 406A1 and step 406A2:

[0102] Step 406A1, for each standard word in the standard speech text, input the i-th standard word in the standard speech text into the large language model, and obtain the i+1-th predicted word and the i-th predicted probability predicted by the large language model based on the i-th standard word, the i+1-th predicted word is the predicted word corresponding to the i+1-th standard word in the standard speech text, and i is an integer.

[0103] Among them, the process of text autoregressive decoding by the large language model is: predicting the next word based on the previous word to fully learn the contextual semantic information. In order to enable the large language model to accurately learn the contextual semantic information in the standard voice text during the training process, during the training process, each standard word contained in the standard voice text needs to be used as the input of the large language model, and the large language model predicts the next standard word based on the previous standard word to obtain the predicted word, and then composes the predicted voice text according to the predicted word. In other words, the process of inputting the standard voice text into the large language model is: inputting the i-th standard word (i.e., the i-th standard word) in the standard voice text into the large language model, obtaining the i+1-th predicted word predicted by the large language model based on the i-th standard word and the i-th predicted probability corresponding to the i-th predicted word, and the i+1-th predicted word is the predicted word corresponding to the i+1-th standard word in the standard voice text.

[0104] Exemplarily, the value of i is an integer. When i is 0, the i-th standard word is the start label, that is, the large language model predicts the first predicted word (the first predicted word is the predicted word of the first standard word in the standard speech text) based on the start label and the acquired acoustic information (the information acquired by the large language model in the process of obtaining the predicted probability distribution based on the acoustic modeling features); when i is 1, the first standard word (that is, the first standard word) in the standard speech text is input into the large language model to obtain the second predicted word and the second predicted probability of the second standard word output by the large language model; and so on, until the last standard word in the standard speech text is input into the large language model to obtain the predicted end label and the predicted probability output by the large language model, then this round of speech recognition ends.

[0105] Please refer to Figure 6 , which is a schematic diagram of the principle of a large language model in text autoregressive decoding provided by an exemplary embodiment of the present disclosure. Figure 6 As shown, taking the standard speech text "The weather is very good today" as an example, the process of obtaining the predicted speech text by autoregressive decoding of the large language model is as follows: the start tag is input into the large language model 601, and the large language model 601 combines the start tag and acoustic information to predict the first predicted word as "today"; the first standard word "today" in the standard speech text is input into the large language model, and the second predicted word is output as "day"; the second standard word "day" in the standard speech text is input into the large language model 601, and the third predicted word "day" is output; the third standard word "day" in the standard speech text is input into the large language model 601, and the fourth predicted word is output as "air"; the fourth standard word "air" in the standard speech text is input into the large language model, and the fifth standard word "very" is output; the fifth standard word "very" in the standard speech text is input into the large language model 601, and the sixth standard word "good" is output; the sixth standard word "good" is input into the large language model 601, and the seventh standard word "end tag" is output. In other words, the input is "<sos>The weather is very good today", the output is "The weather is very good today <eos> ”， <sos>For the start tag, <eos>This is the end tag. It should be noted that Figure 6 Taking the case where the predicted speech text output by the large language model is the same as the standard speech text as an example, in the initial stage of training the large language model, the predicted word predicted by the large language model based on the previous standard word is not the same as the next standard word; for example, when the second standard word "天 (tiān)" in the standard speech text is input into the large language model, the third predicted word output by the large language model may be "空 (kōng)"; rather than "天 (tiān)"; by training the large language model, the output of the large language model is made consistent with the standard text.

[0106] Step 406A2, obtain m predicted words and m predicted probabilities obtained through m rounds of prediction. The m predicted words are used to be combined in order into a predicted speech text. m is determined by the number of words in the standard speech text, and m is a positive integer.

[0107] Then, after multiple rounds of text autoregressive decoding (or after text autoregressive prediction), m predicted words and m predicted probabilities obtained through m rounds of prediction can be obtained. These m predicted words can be combined in the prediction order into a predicted speech text, and the value of m is determined by the number of words in the standard speech text, and m is a positive integer; for example, m = the number of words in the standard speech text + 1. If the standard speech text has 6 words, then m is 6 + 1 = 7.

[0108] Such as Figure 6 shown, taking the standard speech text "今天天气很好 (jīn tiān tiān qì hěn hǎo)" as an example, then m is 7. After 7 rounds of autoregressive decoding, the predicted speech text output "今天天气很好 (jīn tiān tiān qì hěn hǎo)" can be obtained <eos>”.

[0109] Step 407: training a large language model based on the predicted probability distribution, the predicted speech text, and the standard speech text.

[0110] Optionally, when constructing the text autoregressive decoding loss, considering that the training goal of the large language model is to accurately predict each predicted word, the prediction probability of each predicted word needs to be introduced when training the large language model. The corresponding step 407 can also be replaced by step 407A.

[0111] Step 407A, training a large language model based on the predicted probability distribution, m predicted probabilities, m standard probabilities and standard speech text.

[0112] The standard probability is the probability of predicting the (i+1)th standard word based on the (i)th standard word.

[0113] After obtaining the predicted probability distribution and m predicted probabilities output by the large language model, in order to model the text autoregressive decoding loss of the large language model, it is also necessary to obtain m standard probabilities, which are the probability of predicting the i+1th standard word based on the i-th standard word; and then based on the prediction results (prediction probability distribution and m predicted probabilities) and the standard learning objectives (standard speech text and m standard probabilities), update the parameters of the large language model.

[0114] Since the large language model plays two roles in the speech recognition task, one is acoustic modeling and the other is text autoregressive decoding, when determining the loss function of the large language model, the loss function should also include two parts. Correspondingly, in an exemplary example, step 407A may include steps 407A1 to 407A3.

[0115] Step 407A1, based on the predicted probability distribution and the standard speech text, determine the acoustic modeling loss of the large language model.

[0116] When the large language model performs acoustic modeling, it directly outputs the recognition result - predicted probability distribution based on the input acoustic modeling features. For the acoustic modeling part, the acoustic modeling loss of the large language model can be determined directly based on the sequence corresponding to the predicted probability distribution and the standard speech text. For example, the acoustic modeling loss can use CTCLoss.

[0117] Step 407A 2: Determine the text autoregression loss of the large language model based on the m predicted probabilities and the m standard probabilities.

[0118] When the large language model performs text autoregressive decoding, it needs to predict the next word based on the previous word, that is, it pays more attention to the predicted probability between the two words before and after. For the text part, the text autoregressive loss of the large language model is determined by comparing the difference between the predicted probability and the standard probability of each word. Exemplarily, the text autoregressive loss can use the cross entropy loss.

[0119] Step 407A3: train a large language model based on acoustic modeling loss and text autoregression loss.

[0120] After obtaining the acoustic modeling loss and text autoregression loss, the parameters of the large language model can be updated based on the sum of the two; until after multiple rounds of training, the sum of the acoustic modeling loss and the text autoregression loss can be less than the preset value, it is determined that the training of the large language model is completed. The trained large language model and acoustic model can be combined into a speech recognition model to perform speech recognition tasks, that is, input speech and obtain the predicted speech text corresponding to the speech.

[0121] Please refer to Figure 7 , which is a schematic diagram of the training process of another speech recognition model provided by an exemplary embodiment of the present disclosure. First, speech feature extraction is performed on the sample speech 701 to obtain the sample speech feature 702, and the sample speech feature 702 is input into the trained acoustic model 703 for acoustic feature modeling to obtain the acoustic modeling feature 704; then the acoustic modeling feature 704 and the standard speech text 705 are input into the large language model 706 to obtain the m predicted probabilities 708 and the predicted probability distribution 707 corresponding to the m predicted words output by the large language model 706; when determining the loss function, the acoustic modeling loss 709 is determined based on the predicted probability distribution 707 and the standard speech text 705, and the text autoregression loss 711 is determined based on the m predicted probabilities 708 and the m standard probabilities 710, and then the large language model 706 is trained based on the text autoregression loss 711 and the acoustic modeling loss 709.

[0122] In this embodiment, by specially setting the large language model's processing method for the input acoustic modeling features and standard speech text, and setting the losses corresponding to the acoustic part and the text part respectively, the large language model not only serves as a part of the acoustic model, but also exerts the semantic recognition ability of the large language model, and has both acoustic and language characteristics; when the trained acoustic model and the large language model are subsequently applied to perform speech task recognition, the large language model can directly output the recognition result like the acoustic model, and can also decode the recognition result autoregressively like the language model, and has both acoustic and language characteristics, which greatly improves the recognition performance of the speech recognition model.

[0123] The above embodiments mainly describe the training process of the speech recognition model. After the speech recognition model training is completed (mainly after the large language model training is completed), the trained speech recognition model can be deployed to perform speech recognition tasks.

[0124] Please refer to Figure 8 , which shows a flowchart of another speech recognition method according to an exemplary embodiment of the present disclosure. Figure 1 The second device 120 shown in the figure is used as an example for explanation. Figure 8 As shown, the method includes:

[0125] Step 801, extracting speech features of the target speech to be recognized to obtain target speech features.

[0126] Similar to the model training stage, after obtaining the target speech to be recognized in the model application stage, the preset speech feature extraction algorithm is first used to extract the speech features of the target speech to obtain the target speech features. It should be noted that the speech feature extraction algorithm used in the model training stage should be the same as the speech feature extraction algorithm used in the model application stage.

[0127] Step 802, input the target speech features into the trained acoustic model to obtain the target modeling features output by the trained acoustic model.

[0128] After the target speech features are acquired, the target speech features are input into the trained acoustic model for acoustic feature modeling to obtain the target modeling features output by the trained acoustic model.

[0129] Optionally, similar to the training stage, when the trained acoustic model includes at least two layers of modeling networks, the target modeling features exist in the following situations: (1) taking the feature vector output by any intermediate layer modeling network as the target modeling feature; (2) when the acoustic model includes multiple intermediate layer modeling networks (that is, the acoustic model includes at least three layers of modeling networks), taking the feature vectors output by at least two intermediate layer modeling networks for concatenation or weighting to obtain the target modeling feature; (3) taking the feature vector output by the last layer modeling network as the target modeling feature; (4) taking the feature vector output by at least one intermediate layer modeling network and the feature vector output by the last layer modeling network for concatenation or weighting to obtain the target modeling feature.

[0130] Optionally, step 802 may include step 802A and step 802B.

[0131] Step 802A, input the target speech feature into the trained acoustic model to obtain at least two target feature vectors output by at least two layers of modeling networks.

[0132] Among them, the trained acoustic model has at least two layers of modeling networks; after the target speech features are input into the trained acoustic model, at least two target feature vectors output by at least two layers of modeling networks can be obtained; that is, each layer of the modeling network outputs a target feature vector.

[0133] Step 802B, concatenate or weight at least two target feature vectors to obtain target modeling features.

[0134] In a possible implementation, when the trained acoustic model outputs at least two target feature vectors, the at least two target feature vectors are concatenated or weighted to obtain acoustic modeling features. The weight of each target feature vector during weighted processing can be determined by the order of the modeling network, that is, the earlier the modeling network is in the acoustic model, the lower the weight of the target feature vector it outputs.

[0135] Step 803, input the target modeling features and the start label into the trained large language model, and obtain the target probability distribution and the first speech text output by the trained large language model, where the target probability distribution is used to indicate the probability distribution of the predicted words corresponding to the target speech.

[0136] Unlike the model training phase, the large language model inputs standard speech text in the model training phase in order to allow the large language model to better learn semantic information; in the model application phase, there is no standard speech text itself, and the input text at this time is the start tag and the next predicted word predicted by the large language model each time. In a possible implementation, the target modeling features and the start tag are input into the trained large language model, and the target probability distribution and the first speech text output by the trained large language model can be obtained. The target probability distribution is used to indicate the probability distribution of the predicted words corresponding to the target speech.

[0137] Similar to the training process of the large language model, when the large language model processes the input information, it also needs to go through a two-stage processing process. Correspondingly, in an exemplary example, step 803 may also include steps 803A to 803C.

[0138] Step 803A, input the target modeling features into the trained large language model to obtain the target probability distribution output by the trained large language model.

[0139] Similar to the training process of the large language model, in the acoustic part, the target modeling features are first input into the trained large language model for word prediction to obtain the target probability distribution output by the trained large language model. The target probability distribution can be used to decode into speech text.

[0140] Step 803B, input the jth target word into the trained large language model, and obtain the j+1th target word predicted by the trained large language model based on the jth target word, where j is an integer, and when j=0, the jth target word is the start tag.

[0141] The difference from the training process of the large language model is that the input of each round of the large language model training process is the standard words in the standard speech text, while in the model application stage, the input of each round is the words predicted in the previous round. In other words, the process of text autoregressive decoding of the trained large language model is: input the jth target word into the trained large language model, and obtain the j+1th target word predicted by the trained large language model based on the jth target word; and when j is 0, the jth target word is the start label.

[0142] Exemplarily, taking the prediction target as "The weather is very good today", the first round inputs the start label and outputs the predicted first target word "today"; the second round, inputs the first target word "today" and outputs the second target word "day"; the third round, inputs the second target word "day" and outputs the third target word "day"; the fourth round, inputs the third target word "day" and outputs the fourth target word "air"; the fifth round, inputs the fourth target word "day" and outputs the fifth target word "very"; the sixth round, inputs the fifth target word "day" and outputs the sixth target word "good"; the seventh round, inputs the sixth target word "day" and outputs the seventh target word "end label".

[0143] Step 803C, combining the n target characters obtained through n rounds of prediction into a first voice text in order, where n is determined by the number of characters in the first voice text and is a positive integer.

[0144] After multiple rounds of text autoregressive decoding (or text autoregressive prediction), n target words obtained by n rounds of prediction can be obtained. The n target words can be combined into the first speech text according to the prediction order, and n is a positive integer.

[0145] Step 804: decode the target probability distribution to obtain a second speech text.

[0146] After the acoustic part of the large language model outputs the target probability distribution, the target probability distribution can be decoded into corresponding characters, and the second speech text can be obtained after combination.

[0147] Optionally, during decoding, considering that the last layer of the acoustic model will also output the corresponding probability distribution, in order to improve the accuracy of decoding, the probability distribution output by the last layer of the acoustic model and the target probability distribution output by the large language model can be combined to obtain a weighted probability distribution, and then decoding is performed based on the weighted probability distribution to obtain the second speech text.

[0148] Step 805: Generate a target voice text corresponding to the target voice based on the first voice text and / or the second voice text.

[0149] As can be seen from the above embodiments, in the model application stage, the large language model can output two text recognition results, namely the first voice text and the second voice text; both of these text recognition results can be used as the final recognition results, and the target voice text corresponding to the target voice can be generated based on the first voice text and / or the second voice text. Specifically, the first voice text can be determined as the target voice text; or, the second voice text can be determined as the target voice text; or, the first voice text and the second voice text can be combined to comprehensively generate the target voice text.

[0150] Please refer to Fig. 9 , which is a schematic diagram of a speech recognition process provided by an exemplary embodiment of the present disclosure. Fig. 9 As shown, firstly, speech features of the target speech 901 to be recognized are extracted to obtain target speech features 902; and the target speech features 902 are input into the trained acoustic model 903 to obtain the target modeling features 904 outputted therefrom; and then the target modeling features 904 and the start tag 905 are input into the trained large language model 907 to obtain the target probability distribution 907 and the first speech text 908 outputted therefrom, and then the target probability distribution 907 is decoded to obtain the second speech text 909.

[0151] In this embodiment, by applying the constructed new speech recognition model (acoustic model + large language model) in the speech recognition task, the large language model is regarded as a part of the acoustic model, and the large language model plays the role of a language model. The two sides work together to improve the speech recognition performance; in addition, the large language model can output two speech recognition results, so that the final generated speech text can be comprehensively considered, avoiding the accidental errors existing in a single prediction, and improving the accuracy of speech recognition.

[0152] Please refer to Fig.10 , which is a structural diagram of a speech recognition device provided by an embodiment of the present disclosure. The device is applied to Figure 1 The first device 110 is shown. For example, Fig.10 As shown, the device 1000 includes:

[0153] An acquisition module 1001 is used to acquire at least one sample speech and a standard speech text corresponding to the sample speech;

[0154] The feature extraction module 1002 is used to extract the speech features of the sample speech to obtain the sample speech features;

[0155] The acoustic processing module 1003 is used to input the sample speech features into the trained acoustic model to obtain the acoustic modeling features output by the trained acoustic model;

[0156] The speech recognition module 1004 is used to input the acoustic modeling features and the standard speech text into a large language model to obtain a predicted probability distribution and a predicted speech text output by the large language model, wherein the predicted probability distribution is used to indicate a probability distribution of predicted words corresponding to the sample speech;

[0157] The training module 1005 is used to train the large language model based on the predicted probability distribution, the predicted speech text and the standard speech text, so that the trained large language model and the trained acoustic model have speech recognition function.

[0158] Optionally, the speech recognition module 1004 is further used for:

[0159] Inputting the acoustic modeling features into the large language model to obtain the predicted probability distribution output by the large language model;

[0160] Inputting the standard speech text into the large language model to obtain the predicted speech text output by the large language model;

[0161] Optionally, the speech recognition module 1004 is further used for:

[0162] Inputting the standard speech text into the large language model, obtaining m predicted words and m predicted probabilities output by the large language model, wherein the m predicted words are used to be combined in sequence into the predicted speech text, where m is determined by the number of words in the standard speech text and is a positive integer;

[0163] The training module 1005 is further used for:

[0164] The large language model is trained based on the predicted probability distribution, the m predicted probabilities, the m standard probabilities and the standard speech text.

[0165] Optionally, the speech recognition module 1004 is further used for:

[0166] For each standard word in the standard speech text, input the i-th standard word in the standard speech text into the large language model to obtain the i+1-th predicted word and the i-th predicted probability predicted by the large language model based on the i-th standard word, wherein the i+1-th predicted word is the predicted word corresponding to the i+1-th standard word in the standard speech text, and i is an integer;

[0167] Obtaining m predicted words and m predicted probabilities obtained through m rounds of prediction, wherein the m predicted words are used to be combined in sequence into the predicted speech text, where m is determined by the number of words in the standard speech text, and m is a positive integer;

[0168] Optionally, the training module 1005 is further used to:

[0169] Determining the acoustic modeling loss of the large language model based on the predicted probability distribution and the standard speech text;

[0170] Determining a text autoregression loss of the large language model based on the m prediction probabilities and the m standard probabilities;

[0171] The large language model is trained based on the acoustic modeling loss and the text autoregression loss.

[0172] Optionally, the trained acoustic model includes at least two layers of modeling networks;

[0173] The acoustic processing module is also used for:

[0174] Inputting the sample speech features into the trained acoustic model to obtain at least two sample feature vectors respectively output by the at least two layers of modeling networks;

[0175] The at least two sample feature vectors are concatenated or weighted to obtain the acoustic modeling feature.

[0176] Please refer to Fig.11 , which is a structural diagram of another speech recognition device provided by an embodiment of the present disclosure. The device is applied to Figure 1 The second device 120 shown. For example, Fig.11 As shown, the device 1100 includes:

[0177] The feature extraction module 1101 is used to extract the speech features of the target speech to be recognized and obtain the target speech features;

[0178] The acoustic processing module 1102 is used to input the target speech feature into the trained acoustic model to obtain the target modeling feature output by the trained acoustic model;

[0179] The speech recognition module 1103 is used to input the target modeling features and the start tag into the trained large language model to obtain the target probability distribution and the first speech text output by the trained large language model, wherein the target probability distribution is used to indicate the probability distribution of the predicted words corresponding to the target speech;

[0180] A decoding module 1104, configured to decode the target probability distribution to obtain a second speech text;

[0181] The generating module 1105 is used to generate a target voice text corresponding to the target voice based on the first voice text and / or the second voice text.

[0182] Optionally, the speech recognition module 1103 is further used for:

[0183] Inputting the target modeling features into the trained large language model to obtain the target probability distribution output by the trained large language model;

[0184] Inputting the jth target word into the trained large language model to obtain the j+1th target word predicted by the trained large language model based on the jth target word, where j is an integer and when j=0, the jth target word is the start tag;

[0185] The n target words obtained through n rounds of prediction are combined in order to form the first speech text, where n is determined by the number of words in the first speech text and n is a positive integer.

[0186] Optionally, the trained acoustic model includes at least two layers of modeling networks;

[0187] The acoustic processing module 1102 is further used for:

[0188] Inputting the target speech feature into the trained acoustic model to obtain at least two target feature vectors respectively output by the at least two layers of modeling networks;

[0189] The at least two target feature vectors are concatenated or weighted to obtain the target modeling feature.

[0190] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0191] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0192] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0193] refer to Fig.12 , a block diagram of an electronic device 1200 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0194] like Fig.12 As shown, the electronic device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0195] Multiple components in the electronic device 1200 are connected to the I / O interface 1205, including: an input unit 1206, an output unit 1207, a storage unit 1208, and a communication unit 1209. The input unit 1206 can be any type of device that can input information to the electronic device 1200, and the input unit 1206 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 1207 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1208 can include but is not limited to a disk, an optical disk. The communication unit 1209 allows the electronic device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0196] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above. For example, in some embodiments, Figure 2 , Figure 4 , Figure 8 The method shown may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1200 via the ROM 1202 and / or the communication unit 1209. In some embodiments, the computing unit 1201 may be configured to execute the computer program by any other suitable means (e.g., by means of firmware). Figure 2 , Figure 4 , Figure 8 The method shown.

[0197] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0198] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0199] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0200] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0201] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0202] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.< / eos> < / eos> < / sos> < / eos> < / sos>

Claims

1. A speech recognition method, characterized in that: The method comprises: Acquire at least one sample speech and a standard speech text corresponding to the sample speech; Extracting speech features from the sample speech to obtain sample speech features; Inputting the sample speech features into the trained acoustic model to obtain acoustic modeling features output by the trained acoustic model; Inputting the acoustic modeling features and the standard speech text into a large language model to obtain a predicted probability distribution and a predicted speech text output by the large language model, wherein the predicted probability distribution is used to indicate a probability distribution of predicted words corresponding to the sample speech; Based on the predicted probability distribution, the predicted speech text and the standard speech text, the large language model is trained so that the trained large language model and the trained acoustic model have speech recognition functions.

2. The method according to claim 1, characterized in that The step of inputting the acoustic modeling features and the standard speech text into a large language model to obtain a predicted probability distribution and a predicted speech text output by the large language model includes: Inputting the acoustic modeling features into the large language model to obtain the predicted probability distribution output by the large language model; The standard speech text is input into the large language model to obtain the predicted speech text output by the large language model.

3. The method according to claim 2, characterized in that The step of inputting the standard speech text into the large language model to obtain the predicted speech text output by the large language model includes: Inputting the standard speech text into the large language model, obtaining m predicted words and m predicted probabilities output by the large language model, wherein the m predicted words are used to be combined in sequence into the predicted speech text, where m is determined by the number of words in the standard speech text and is a positive integer; The step of training the large language model based on the predicted probability distribution, the predicted speech text and the standard speech text comprises: The large language model is trained based on the predicted probability distribution, the m predicted probabilities, the m standard probabilities and the standard speech text.

4. The method according to any one of claims 1 to 3, characterized in that: The trained acoustic model includes at least two layers of modeling networks; The step of inputting the sample speech features into the trained acoustic model to obtain the acoustic modeling features output by the trained acoustic model comprises: Inputting the sample speech features into the trained acoustic model to obtain at least two sample feature vectors respectively output by the at least two layers of modeling networks; The at least two sample feature vectors are concatenated or weighted to obtain the acoustic modeling feature.

5. A speech recognition method, characterized in that: The method comprises: Extract speech features of the target speech to be recognized to obtain target speech features; Inputting the target speech feature into the trained acoustic model to obtain the target modeling feature output by the trained acoustic model; Inputting the target modeling feature and the start label into the trained large language model, obtaining a target probability distribution and a first speech text output by the trained large language model, wherein the target probability distribution is used to indicate a probability distribution of predicted words corresponding to the target speech; Decoding the target probability distribution to obtain a second speech text; Based on the first voice text and / or the second voice text, a target voice text corresponding to the target voice is generated.

6. The method according to claim 5, characterized in that The step of inputting the target modeling features and the start tag into the trained large language model to obtain the target probability distribution and the first speech text output by the trained large language model includes: Inputting the target modeling features into the trained large language model to obtain the target probability distribution output by the trained large language model; Inputting the jth target word into the trained large language model to obtain the j+1th target word predicted by the trained large language model based on the jth target word, where j is an integer and when j=0, the jth target word is the start tag; The n target words obtained through n rounds of prediction are combined in order to form the first speech text, where n is determined by the number of words in the first speech text and n is a positive integer.

7. The method according to claim 5 or 6, characterized in that: The trained acoustic model includes at least two layers of modeling networks; The step of inputting the target speech feature into the trained acoustic model to obtain the target modeling feature output by the trained acoustic model includes: Inputting the target speech feature into the trained acoustic model to obtain at least two target feature vectors respectively output by the at least two layers of modeling networks; The at least two target feature vectors are concatenated or weighted to obtain the target modeling feature.

8. A speech recognition device, characterized in that: The device comprises: An acquisition module, used to acquire at least one sample speech and a standard speech text corresponding to the sample speech; A feature extraction module, used to extract speech features from the sample speech to obtain sample speech features; An acoustic processing module, used for inputting the sample speech features into the trained acoustic model to obtain acoustic modeling features output by the trained acoustic model; A speech recognition module, used for inputting the acoustic modeling features and the standard speech text into a large language model to obtain a predicted probability distribution and a predicted speech text output by the large language model, wherein the predicted probability distribution is used to indicate a probability distribution of predicted words corresponding to the sample speech; A training module is used to train the large language model based on the predicted probability distribution, the predicted speech text and the standard speech text, so that the trained large language model and the trained acoustic model have speech recognition function.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images