Voice recognition method, device, electronic device and storage medium

Abstract

The present application relates to the field of speech recognition technology and discloses a speech recognition method, device, electronic device, and storage medium. The speech recognition method includes: obtaining a speech to be recognized; recognizing the speech to be recognized using a speech recognition model to obtain a speech recognition result; wherein the speech recognition model is trained in the following manner: training an initial speech recognition model based on each initial training sample to obtain a preliminarily trained recognition model; obtaining recognition error rate representation information corresponding to each initial training sample when recognized by the preliminarily trained recognition model; selecting a target sample from each initial training sample based on the representation information; and training the preliminarily trained recognition model based on each target sample to obtain the speech recognition model. The solution provided by the present application can improve the recognition accuracy of the speech recognition model.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and more specifically, to a speech recognition method, device, electronic device, and storage medium. Background Art

[0002] In speech recognition scenarios, it is becoming increasingly common to use training models for speech recognition. However, since there are many characters with the same or similar pronunciations in a language, when the trained model recognizes speech, recognition errors such as grammatical errors or inadequate words often occur in the recognized sentences.

[0003] In order to improve the recognition accuracy of speech recognition models, in the existing technology, word error rate minimization training is often performed. When the word error rate minimization training is performed, all sample data in the model training process are involved in the word error rate minimization training process. However, for training sentences that can already be correctly recognized and have high discrimination, using them for word error rate minimization training will not bring any benefits, because such data has already achieved a small average word error rate. Minimizing the word error rate for such sentences will lead to a waste of training resources, the efficiency of word error rate minimization training is low, and the speech recognition effect of the trained model still needs to be improved. Summary of the Invention

[0004] The purpose of this application is to solve at least one of the above technical deficiencies, and the following technical solutions are proposed:

[0005] In one aspect of the present application, a speech recognition method is provided, comprising:

[0006] Get the speech to be recognized;

[0007] The speech recognition model is used to recognize the speech to be recognized and obtain a speech recognition result. The speech recognition model is trained in the following way:

[0008] Training the initial speech recognition model based on each initial training sample to obtain a preliminarily trained recognition model;

[0009] Obtaining the recognition error rate representation information corresponding to each initial training sample when it is recognized by the recognition model after preliminary training;

[0010] Select target samples from each initial training sample based on the representation information;

[0011] The recognition model after preliminary training is trained based on each target sample to obtain a speech recognition model.

[0012] Another aspect of the present application provides a speech recognition device, the device comprising:

[0013] A voice acquisition module to be recognized, used to acquire the voice to be recognized;

[0014] The recognition result acquisition module is used to recognize the speech to be recognized through the speech recognition model and obtain the speech recognition result;

[0015] The speech recognition model is obtained by training a training device, and the training device includes:

[0016] A preliminary training module is used to train the initial speech recognition model based on each initial training sample to obtain a recognition model after preliminary training;

[0017] The target sample screening module is used to obtain the corresponding recognition error rate representation information of each initial training sample when it is recognized by the recognition model after preliminary training, and select the target sample from each initial training sample based on the representation information;

[0018] The model retraining module is used to train the recognition model after preliminary training based on each target sample to obtain a speech recognition model.

[0019] In another aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the speech recognition method described in the first aspect of the present application is implemented.

[0020] In another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the speech recognition method shown in the first aspect of the present application is implemented.

[0021] The beneficial effects of the technical solution provided by this application are:

[0022] The speech recognition method provided in the present application uses recognition error rate representation information to screen initial training samples, and can screen out initial training samples with a higher probability of incorrect recognition as target samples, so as to train the speech recognition model based on the target samples. Compared with the method of using all initial training samples for training in the prior art, the amount of training data of the speech recognition model is greatly reduced, and the recognition accuracy of the speech recognition model is improved; moreover, by selecting target samples from the initial training samples based on the recognition error rate representation information, targeted training can be carried out for training samples with a higher recognition error rate, which is beneficial to improving the performance of the trained speech recognition model and improving the accuracy of speech recognition.

[0023] Additional aspects and advantages of the present application will be given in part in the following description, which will become apparent from the following description, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A flowchart of a speech recognition method provided in one embodiment of the present application;

[0026] Figure 2 A flowchart of obtaining the corresponding recognition error rate characterization information of each initial training sample when it is recognized by the recognition model after preliminary training, provided in one embodiment of the present application;

[0027] Figure 3 A flowchart of screening target samples and training a speech recognition model using the target samples provided in one embodiment of the present application;

[0028] Figure 4 A flowchart of iteratively training the preliminarily trained recognition model based on each target sample provided in one embodiment of the present application;

[0029] Figure 5 A flowchart for calculating the value of a first loss function based on the probability and the number of erroneous characters corresponding to each candidate recognition result of each target sample provided in one embodiment of the present application;

[0030] Figure 6 A schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0032] Table 1 is a recognition probability distribution table of the top five recognition results before normalization for the initial training sample "Navigate to the Forbidden City" provided by an embodiment of the present application;

[0033] Table 2 is a comparison table of information entropy corresponding to each character in the initial training sample "Navigate to the Forbidden City" provided by an embodiment of the present application;

[0034] Table 3 is a comparison table of experimental results provided in one embodiment of the present application. DETAILED DESCRIPTION

[0035] The following describes embodiments of the present application in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0036] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of this application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0037] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless specifically defined as herein, will not be interpreted in an idealized or overly formal sense.

[0038] Minimize word error rate: A way to reduce the model's error rate in recognizing characters.

[0039] Cross entropy is used to calculate the difference between the learned model distribution and the training distribution. The cross entropy loss function can measure the similarity between the distribution of true labels and the distribution of predicted labels after training.

[0040] Beam search: It is a heuristic method for solving optimization problems. It uses heuristic methods to estimate K better paths and only searches downward from these K paths, that is, only a limited number of K nodes are retained at each layer.

[0041] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning. This application obtains a recognition model and a speech recognition model after preliminary training through machine learning.

[0042] During the research process, the inventors discovered that during the word error rate minimization training process, if the target sample selection criterion is based on whether the predicted text corresponding to the maximum prediction probability is consistent with the sample text, only a limited number of target samples can be obtained. This insufficient number of target samples leads to inaccurate training results.

[0043] In order to solve the technical problems existing in the prior art, the speech recognition method, device, electronic device and storage medium provided in this application are intended to solve at least one of the above technical problems in the prior art.

[0044] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0045] The present application embodiment provides a possible implementation method, such as Figure 1 As shown, a flowchart of a speech recognition method is provided. The method can be executed by any electronic device, and optionally, can be executed on a server side, including the following steps:

[0046] Step S110, obtaining the speech to be recognized;

[0047] Step S120: Recognize the speech to be recognized by a speech recognition model to obtain a speech recognition result; wherein the speech recognition model is trained by the method provided in the following steps S130 to S140:

[0048] Step S130, training the initial speech recognition model based on each initial training sample to obtain a preliminarily trained recognition model;

[0049] Step S140, obtaining the corresponding recognition error rate representation information of each initial training sample when it is recognized by the recognition model after preliminary training;

[0050] Step S150, selecting a target sample from each initial training sample according to the representation information;

[0051] Step S160 , training the preliminarily trained recognition model based on each target sample to obtain the speech recognition model.

[0052] The solution provided in this application can be applied to, but not limited to, the following scenarios: an electronic device (such as a server or a user terminal) receives a speech recognition request for a segment of speech to be recognized. In response to the request, the server recognizes the speech to be recognized using a trained speech recognition model, obtains a speech recognition result, i.e., the text information corresponding to the speech to be recognized, and provides the speech recognition result to the user or the request sender. For example, the solution can be applied to an instant messaging application, through which two or more users can exchange information using text or voice. For a user, if the user receives a voice message sent by another user, the user can obtain the corresponding information by directly playing the voice. That is, when the user terminal or server receives a user's voice playback request (such as the user clicks on the received voice message), the user terminal or server can recognize the speech corresponding to the request and play the recognized speech to each user in the form of voice; or when the user initiates a request to convert the received speech into text, the user terminal or server can directly display the text corresponding to the speech to the user.

[0053] Optionally, the speech recognition model is trained in the following way:

[0054] A plurality of initial training samples for model training are obtained, and an initial speech recognition model is trained based on each initial training sample to obtain a recognition model after preliminary training; wherein, the recognition model after preliminary training is a model corresponding to when the convergence conditions in the model training process are met, that is, the recognition model after preliminary training can basically realize the correct recognition of speech samples, but the recognition accuracy can be further improved.

[0055] Obtain recognition error rate representation information corresponding to each initial training sample when it is recognized by the recognition model after preliminary training; wherein, the recognition error rate representation information is the error rate of the recognition result when the initial training sample is recognized by the recognition model after preliminary training, that is, the possibility of an error in the recognition result. The higher the possibility, the greater the probability that the initial training sample will be selected as the target sample.

[0056] Target samples are selected from each initial training sample according to the recognition error rate representation information, and the recognition model after preliminary training is trained based on each target sample to obtain a speech recognition model.

[0057] The present application screens the initial training samples based on the recognition error rate characterization information of the initial training samples to obtain target samples that meet the screening conditions. The screening conditions include: N initial training samples with the highest recognition error rate, or N initial training samples with a recognition error rate greater than a preset threshold, where N is a positive integer greater than 1. The recognition model is further trained based on the target samples, which is conducive to optimizing the recognition model in terms of recognition error rate. That is, compared with the recognition model after preliminary training, the speech recognition model trained using the target samples has a lower recognition error rate.

[0058] The present application screens target samples based on the characterization information of the recognition error rate of the initial training samples by the recognition model, and can screen out initial training samples with a higher recognition error rate as target samples, and train the speech recognition model based on the target samples. Compared with the method of using all the initial training samples for training in the prior art, the amount of training data of the speech recognition model is greatly reduced, and the recognition accuracy of the speech recognition model is improved; moreover, selecting target samples from the initial training samples based on the characterization information of the recognition error rate can realize targeted training on the recognition error rate, which is conducive to improving the performance of the trained speech recognition model and improving the accuracy of speech recognition.

[0059] In order to make the speech recognition solution provided by this application and its technical effects more clear, its specific implementation plan is described in detail with multiple examples below.

[0060] In an optional embodiment, each initial training sample includes a sample speech and a sample text corresponding to the sample speech. When obtaining the recognition error rate representation information corresponding to each initial training sample when it is recognized by the recognition model after preliminary training, for each initial training sample, it can be implemented in the following manner, as shown in the flowchart. Figure 2 Shown, including:

[0061] Step S210, using the preliminarily trained recognition model to recognize the sample speech of the sample, and obtaining at least two recognition probabilities that are ranked higher in descending order among the recognition probabilities corresponding to each character in the sample text of the sample;

[0062] Step S220, for each character in the sample text, determining the information entropy corresponding to the at least two recognition probabilities corresponding to the character;

[0063] Step S230 : determining the recognition error rate representation information corresponding to the initial training sample based on the information entropy corresponding to each character in the sample text.

[0064] For each initial training sample, the above-mentioned method can be used to obtain the corresponding recognition error rate characterization information of each sample. Among them, as a model training sample, the initial training sample includes a sample speech and a sample text corresponding to the sample speech. The sample text is the standard text of the sample speech, i.e., the correct text. The initial training sample is input into the recognition model after preliminary training. The sample speech of the initial training sample is recognized by the recognition model after preliminary training to obtain multiple recognition probabilities corresponding to each character in the sample text corresponding to the initial training sample. Based on the multiple recognition probabilities, the information entropy of the corresponding character is determined. The size of the information entropy corresponding to the character can indicate the probability of the character being misrecognized. The larger the information entropy, the greater the probability of the character being misrecognized. The smaller the information entropy, the higher the probability of the character being misrecognized.

[0065] Among them, the recognition probability refers to the possibility of the sample speech being recognized as a certain character. For example, if the sample speech is a speech signal of "navigation to the Forbidden City", the corresponding sample text is "navigation to the Forbidden City". When the sample speech is recognized by the recognition model after preliminary training, the prediction probabilities corresponding to each character in the sample text can be obtained. It can be understood that each character is predicted to be the probability of each word in the vocabulary (dictionary), such as the probability of the first "guide" being predicted as "guide", the probability of being predicted as "to", and so on. The higher the corresponding probability, the higher the probability that the first character in the final predicted text is the character. When performing speech recognition through the recognition model, the above recognition probability can be understood as the output results corresponding to the last hidden layer of the model (that is, the hidden layer cascaded with the output layer) with larger values ​​(that is, the probability value before normalization). By normalizing the output results with larger values, several recognition probabilities with higher rankings are obtained.

[0066] Optionally, at least two probability values ​​corresponding to the top-ranked characters are obtained according to the magnitude of the recognition probabilities before normalization, and the information entropy corresponding to each character is determined according to the at least two recognition probabilities corresponding to each character.

[0067] Characters with similar recognition probabilities often have the same or similar pronunciations. Since the recognition model is not discriminative due to the similar acoustic pronunciations, the information entropy of each corresponding character is obtained based on at least two recognition probabilities that are ranked first. If there are characters with the same or similar pronunciations, the recognition probabilities of the two characters are close. If the information entropy of the characters is calculated based on the recognition probabilities with close recognition probabilities, the calculated information entropy is larger.

[0068] The examples are as follows: An initial training sample is: (x, y*), where x is the sample voice and y* is the sample text. In this example, the sample text y* = ('guide', 'navigate', 'arrive', 'former palace'). The sample voice and the corresponding sample text are input into the recognition model after preliminary training, and the top 5 recognition probabilities corresponding to each character included in the training sample are obtained in descending order, as shown in Table 1. The recognition probabilities of the character 'arrive' in the sample text before normalization for the top 5 prediction results are: arrive - 9.111, dao - 9.011, guide - 8.444, tao - 7.345, ticket - 5.232. Since there are homophones and characters with similar pronunciations in the prediction results, the recognition probabilities before normalization corresponding to the prediction results are relatively close. The information entropy corresponding to the character 'arrive' obtained based on multiple recognition probabilities before normalization with similar probabilities is 1.2716. For the character 'former palace', the recognition probabilities before normalization for the top 5 in the probability ranking are: 11.123, 3.4, 1.2, 0.991, 0.7. The probability differences are relatively large, indicating that there are no identical or similar characters in the output results, and the model's recognition of this character is relatively accurate.

[0069] Table 1

[0070] Sample characters <![CDATA[O1]]> <![CDATA[O2]]> <![CDATA[O3]]> <![CDATA[O4]]> <![CDATA[O5]]> guide 10.032 (Guide) 2.22 2.100 0.14 0.111 Navigation 9.881 (Airline) 2.112 2.020 2.001 0.444 arrive 9.111 (arrival) 9.011 (Taobao) 8.444 (Guide) 7.345 (Tao) 5.232 (votes) Therefore 12.222 (therefore) 1.01 0.99 0.88 0.542 palace 11.123 (Palace) 3.4 1.2 0.991 0.7

[0071] According to the method provided in the above embodiment, the recognition probabilities before normalization corresponding to each character in the sample text 'navigate to the former palace' in the initial training sample are obtained in sequence. The information entropy of each character obtained based on this recognition probability is shown in Table 2. The information entropy of each character is: 0.0079, 0.0113, 1.2716, 0.0006, 0.0052. By comparing the data in the table, it can be seen that the information entropy of the character 'arrive' is the largest.

[0072] Table 2

[0073] Sample characters Entropy(b,i) guide 0.0079 Navigation 0.0113 arrive 1.2716 Therefore 0.0006 palace 0.0052

[0074] After that, according to the information entropy corresponding to each character in the initial training sample respectively, the recognition error-prone rate characterization information corresponding to the initial training sample is determined, so as to screen out the target data based on this characterization information subsequently.

[0075] For the solution provided in this embodiment, the recognition error rate characterization information of the initial training sample is determined based on the information entropy of each character in the initial training sample. The greater the information entropy, the greater the probability that there are characters with the same or similar pronunciations in the training sample, and the greater the probability of recognition errors. Based on the information entropy of the characters in the initial training sample, the probability of recognition errors of the recognition model for the initial training sample can be accurately determined. The initial training samples that meet the screening conditions are selected from multiple initial training samples based on the magnitudes of the information entropy corresponding to each character as the target samples, so as to achieve the purpose of reducing the recognition error rate of the speech recognition model based on the selected target samples.

[0076] Optionally, for an initial training sample, based on the information entropy corresponding to each character in the sample text, the recognition error rate characterization information corresponding to the initial training sample can be determined in the following manner:

[0077] The maximum information entropy among the information entropy corresponding to each character in the sample text of the initial training sample is used as the recognition error rate characterization information corresponding to the initial training sample.

[0078] For each initial training sample, all the characters included in the initial training sample are obtained, and the information entropy corresponding to all the characters in the initial training sample is obtained according to the method provided in the above embodiment, and the maximum information entropy among all the information entropy is used as the recognition error rate characterization information corresponding to the initial training sample.

[0079] Since the magnitude of the information entropy corresponding to a character characterizes the probability of misrecognition of the character, that is: the information entropy of a character can characterize the probability of misrecognition of the character. If any character corresponds to a relatively large information entropy, it indicates that the possibility of misrecognition of the character is greater. If there are recognition errors in the recognition result corresponding to the initial training sample, the character with the maximum information entropy has the greatest probability of misrecognition. Therefore, using the maximum information entropy corresponding to all characters as the information entropy of the initial training sample, this information entropy can accurately and intuitively characterize the recognition error rate of the initial training sample.

[0080] As shown in Table 2, among all the characters in the sample text corresponding to the initial training sample, the character with the maximum information entropy is '到' (arrive), so the information entropy 1.2716 corresponding to '到' is used as the information entropy of the initial training sample. Using this information entropy as the information entropy of the initial training sample can accurately and intuitively characterize the probability of misrecognition of the initial training sample.

[0081] This embodiment uses the maximum information entropy corresponding to the characters in the initial training sample as the recognition error rate representation information of the initial training sample, which can accurately characterize the recognition error rate of the initial training sample, and then select the initial training samples that are easily misrecognized by the initial speech recognition model as target samples based on the information entropy. According to these target samples, targeted training is carried out on the initial training samples with a higher recognition error rate, thereby improving the recognition probability of the speech recognition model and the efficiency of the trained speech recognition model.

[0082] In an optional embodiment, the recognition model is used to recognize the sample speech of the initial training sample, and at least two recognition probabilities corresponding to each character in the sample text of the sample are obtained, which are ranked higher in descending order. This can be achieved by:

[0083] A1, using the recognition model to recognize a sample speech of an initial training sample, to obtain at least two recognition probabilities before normalization corresponding to each character in the sample text of the sample, wherein the at least two recognition probabilities before normalization are at least two of the recognition probabilities before normalization that are ranked higher in descending order;

[0084] A2: normalize the at least two recognition probabilities before normalization to obtain at least two recognition probabilities corresponding to each character.

[0085] Obtain the predicted probability of each character in the sample text in the initial training sample after being processed by the recognition model. The predicted probability is the recognition probability before normalization. Select a preset number of predicted probabilities with a high ranking according to the size of the probability. The preset number can be set according to actual conditions and application requirements, such as 3, 5, 10, 20, etc. The selected preset number of prediction probabilities are normalized to obtain the preset number of recognition probabilities after normalization.

[0086] The preset number of predicted probabilities ranked at the top are normalized to obtain the normalized recognition probability. The recognition probability can intuitively reflect the characters in the training sample that are most likely to be misrecognized, thereby achieving standardization of the recognition probability. The information entropy of the characters obtained based on the recognition probability can more accurately characterize the recognition error rate of each character.

[0087] This embodiment also provides an implementation method to obtain the normalized recognition probability and determine the information entropy corresponding to the character and the information entropy corresponding to the initial training sample based on the recognition probability, as follows:

[0088] Assume that the number of settings is 5, that is, select the first 5 in the order from large to small, and assume that there are B pairs of training data B is the size of the current initial training sample batch. For a pair of initial training samples x b is the sample speech in the initial training sample b, For the sample text in the training sample b, calculate the predicted probability O(y k |x b ,y b,＜i ), which is the probability value before normalization, where k represents the kth dimension of the recognition model output after preliminary training, and y b,＜i ={y1,y2,…,y i-1}, for the prediction result of the i-th character in the b-th initial training sample, select the prediction probability O(y k |x b ,y b,＜i ) The top 5 with the largest values ​​are [O1, O2, O3, O4, O5], and are normalized using the following formula to obtain the normalized probability value, namely:

[0089]

[0090] Among them, O m is the predicted probability of the mth prediction result of the i-th character, P m is the recognition probability after normalization of the predicted probability. For the i-th character of the b-th initial training sample, the information entropy of the character Entropy(b,i) is calculated by the following formula:

[0091]

[0092] For this initial training sample, if the information entropy Entropy(b,i) of the i-th character is small, it means that the recognition model has made a correct prediction for that character with a high degree of confidence. This is because the recognition probability output by the recognition model is mainly concentrated on the correct recognition result, and the probability of the remaining recognition results is very low. In this case, the calculated information entropy is small. If the output result for a certain character produces a large information entropy, it means that the recognition model will have a high probability of making an incorrect prediction for that character. Therefore, as long as any character in the current training data produces a large information entropy, it can be considered that the current training sample has a high probability of recognition error. The current training sample can then be selected as the target sample for optimization training of the recognition model after preliminary training to obtain an optimized speech recognition model.

[0093] Preferably, the maximum information entropy among all characters in the initial training sample is selected as the recognition error rate representation information Entropy(b) of the initial training sample b, that is, as the error-proneness measure of the initial training sample:

[0094]

[0095] Repeat the above process for all initial training samples b = 1, 2, ..., B, and sort the information entropy of all initial training samples, select the initial training samples with information entropy greater than the preset information entropy threshold as the target samples, or sort the initial training samples according to the size of the information entropy, and select the initial training samples with the first θ part with larger information entropy as the target samples for optimization training of the recognition model.

[0096] Combine Figure 3 The flowchart shown in the figure illustrates the process of screening target samples and using the target samples to train the speech recognition model, and obtaining the training data consisting of the initial training samples. For the bth initial training sample, calculate the top-5 information entropy Entropy(b,i) of the recognition model output for each character after preliminary training. For each initial training sample, select the largest information entropy to measure the error-proneness of this initial training sample, which is expressed as Entropy(b). Sort the information entropy of all B training data in the initial training sample according to the size of the value, and select the first θ*B initial training samples with larger values ​​to perform word error rate minimization training. Repeat the above process for the initial training samples in the training data set until the model converges and a trained speech recognition model is obtained.

[0097] In the prior art, the target samples are screened by selecting the predicted sample with the highest recognition probability to see if it is consistent with the sample text. If they are consistent, they are used as target samples. If they are inconsistent, they are not used as target samples. Compared with this method, the present application selects several recognition probabilities with the highest ranking as the basis for screening target samples for each initial training sample, which greatly expands the number of target samples that can be screened, is conducive to increasing the number of target samples, and can avoid the problem of low model recognition accuracy caused by a small number of target samples.

[0098] According to the solution provided in the above embodiment, a target sample is selected from the initial training sample, wherein the initial training sample includes a sample speech and a sample text corresponding to the sample speech. Therefore, the target sample also includes a sample speech and a sample text corresponding to the sample speech. The following embodiment will describe how to train the recognition model after preliminary training based on the target sample, which can be performed in the following manner, including:

[0099] Iteratively train the preliminarily trained recognition model based on each target sample until the total loss function corresponding to the recognition model reaches the convergence condition;

[0100] Among them, the input of the recognition model is the sample speech of the target sample, and the output of the recognition model is the predicted text corresponding to the sample speech. The value of the total loss function represents the difference between the predicted text and the sample text of the sample speech of each target sample.

[0101] The embodiment of the present application utilizes the target sample to iteratively train the recognition model after preliminary training until the total loss function corresponding to the recognition model reaches the convergence condition. The value of the total loss function represents the difference between the predicted sample of the target sample and the sample text. The condition for the loss function to converge can be configured according to actual needs, that is, the condition for the end of model training can be configured according to actual needs. If the difference is within the preset threshold, that is, the difference is within an acceptable range, then the total loss function reaches the convergence condition. Since the target sample of the present application is selected based on the recognition error rate, the initial training sample that is easily misidentified is selected as the target sample, which is conducive to reducing the probability of the speech to be recognized being misidentified, and can achieve optimization in terms of the recognition error rate, that is, reducing the recognition error rate of the speech recognition model and improving the recognition accuracy of the speech recognition model.

[0102] When the total loss function of the model reaches the convergence condition, that is, the model's recognition error rate for the sample speech meets the preset conditions, compared with the initial speech recognition model, the corresponding recognition model that reaches the convergence condition greatly reduces the speech recognition error rate.

[0103] In an optional embodiment, the above-mentioned total loss function includes a first loss function and a second loss function, the value of the first loss function represents the character difference between the predicted text of the sample speech of each target sample and the sample text, and the value of the second loss function represents the text difference between the predicted text of the sample speech of the target sample and the sample text.

[0104] Optionally, the total loss function of the speech recognition model includes two parts, namely the first loss function and the second loss function mentioned above. The value of the first loss function represents the character difference between the predicted text corresponding to the target sample and the sample text, that is, the difference between each character in the predicted text and the character at each corresponding position in the sample text. It measures the difference between the predicted text and the sample text from the character perspective, while the second loss function measures the difference between the predicted text and the sample text from the text level, and measures the recognition error rate of the recognition model from the character and text levels, which is conducive to reducing the number of character recognition errors while improving the recognition accuracy of the overall text.

[0105] On this basis, in an optional embodiment, the iterative training of the recognition model after preliminary training based on each target sample can be obtained by the following method, the flow chart of which is as follows: Figure 4 Shown, including:

[0106] Step S410: For each training session, for each target sample, beam search decoding is performed on the sample speech of the target sample using the recognition model to obtain a set number of candidate recognition results corresponding to the sample speech and the probability corresponding to each candidate recognition result;

[0107] Step S420, obtaining the predicted text corresponding to each sample speech;

[0108] Step S430 , for each candidate recognition result, determining the number of incorrect characters in the candidate recognition result based on the candidate recognition result and the sample text;

[0109] Step S440, calculating the value of the first loss function according to the probability corresponding to each candidate recognition result of each target sample and the number of erroneous characters;

[0110] Step S450, calculating a value of a second loss function based on the predicted text and the sample text of each sample speech;

[0111] Step S460: Calculate the value of the total loss function according to the value of the first loss function and the value of the second loss function.

[0112] For each training, the value of the total loss function is obtained as follows:

[0113] For each target sample, the trained recognition model is subjected to beam search decoding to generate N candidate recognition results. The N candidate recognition results output after beam search decoding are the N optimal recognition results, where N is a positive integer greater than 1. The probability corresponding to each candidate recognition result is the recognition probability at the text level of the target sample, which can be obtained based on the recognition probability of each character.

[0114] The predicted text corresponding to the sample speech is the recognition result with the highest recognition probability output by the current recognition model, and the value of the second loss function is calculated based on the predicted text and the sample text corresponding to the sample speech.

[0115] The total loss function of the speech recognition model includes a first loss function and a second loss function. The value of the first loss function is obtained based on the probability corresponding to each candidate recognition result and the number of incorrectly identified characters in the candidate recognition result. The number of incorrect characters is determined by comparing the candidate recognition result with the sample text to determine the number of different characters at corresponding positions. The value of the first loss function is calculated using the probability corresponding to each candidate recognition result and the number of incorrect characters for the target sample, representing the character differences between the predicted text and the sample text. Calculating the value of the first loss function based on the N optimal recognition results is conducive to obtaining the optimal value of the first loss function.

[0116] The value of the second loss function can be calculated based on the predicted text and the sample text corresponding to the sample speech of the target sample to measure the text difference between the predicted sample and the sample text. The value of the second loss function can be calculated by at least one method such as cross entropy, Euclidean distance, etc.

[0117] After obtaining the values ​​of the first loss function and the second loss function, the value of the total loss function is calculated based on the values ​​of the first loss function and the second loss function, such as: accumulating or interpolating the values ​​of the first loss function and the second loss function to determine the value of the total loss function. The value of the total loss function represents the actual needs. The convergence direction of the speech recognition model can be adjusted by adjusting the processing method of the first loss function and the second loss function. Model training is performed based on the first loss function obtained based on N optimal recognition results, which is conducive to improving the recognition accuracy of the speech recognition model.

[0118] In an optional embodiment, the value of the first loss function is calculated based on the probability and the number of erroneous characters corresponding to each candidate recognition result of each target sample, which can be achieved by the following method, as shown in the flowchart. Figure 5 Shown, including:

[0119] Step S510, for each target sample, determining an average value of the erroneous characters corresponding to the target sample according to the number of erroneous characters corresponding to each candidate recognition result of the target sample;

[0120] Step S520 , for each candidate recognition result of each target sample, determining the edit distance between the candidate recognition result and the sample text;

[0121] Step S530: For each target sample, determine the training loss value corresponding to the target sample based on the probability and edit distance corresponding to each candidate recognition result of the target sample and the average value corresponding to the target sample;

[0122] Step S540: Obtain the value of the first loss function according to the training loss value of each target sample.

[0123] Each target sample corresponds to N candidate recognition results. First, the number of incorrectly recognized characters in each candidate recognition result, that is, the number of incorrect characters, is obtained. The average value of the incorrect characters of the corresponding target sample is determined based on the number of incorrect characters corresponding to each candidate recognition result and the number of candidate recognition results.

[0124] For each candidate recognition result of a target sample, the edit distance between the candidate recognition result and the sample text is calculated. The training loss value for the target sample is then determined based on the probability corresponding to each candidate recognition result, the edit distance, and the average number of incorrect characters corresponding to the target sample.

[0125] Specifically, given a pair of target samples (x,y * ), where x=(x1,x2,…,x T ) is the sample speech, y * =(y1,y2,…,y I ) is a sample text, which includes I characters, T and I are the lengths of the sample speech and sample text respectively. Let Y = {y1, y2, ..., y N} is the N optimal candidate paths obtained by beam search decoding for the target sample, each path corresponds to a candidate recognition result, the target sample (x, y * ) corresponds to the first loss function L MWER (x,y * ) is represented by the following formula:

[0126]

[0127] Among them, the right side of the equation in the formula of the first loss function can represent the training loss value corresponding to the target sample, where,

[0128]

[0129] W(y i ,y * ) is the edit distance between the i-th (i<=N) candidate recognition result and the sample text, is the average value of the error characters corresponding to the target sample, P(y n |x) is the recognition probability before normalization corresponding to the candidate recognition result. The solution provided in this embodiment determines the value of the first loss function based on the probability of each candidate recognition result, the edit distance, and the average value of the error characters corresponding to the target sample, thereby achieving accurate calculation of the first loss function.

[0130] In an optional embodiment, the first loss function is a word error rate minimization loss function, and the second loss function is a cross entropy loss function.

[0131] Optionally, the training process for minimizing word error rate is as follows: For each target sample, beam search decoding is performed using the preliminarily trained recognition model to generate N optimal recognition results, which can be referred to as N-Best. For each target sample, the N-Best list (i.e., a list of N optimal candidate recognition results) is then used to train the recognition model until the model reaches convergence. During this process, since the target samples have been screened, the number of target samples used for word error rate minimization training is significantly reduced compared to the number of initial training samples. In particular, when the initial training sample data volume is tens of thousands of hours, a large amount of invalid training data is filtered out, greatly improving the efficiency of word error rate minimization training.

[0132] The second loss function is the cross-entropy loss function. The cross-entropy can measure the degree of difference between two different probability distributions in the same random variable, and characterize the difference between the distribution of predicted text and sample text.

[0133] The solution provided in this embodiment determines the total loss function based on minimizing the word error rate loss function and the cross entropy loss function, and uses the total loss function to train the model. The speech recognition model finally trained can further reduce the recognition error rate of the speech to be recognized.

[0134] Optionally, the speech recognition method provided by this application further includes:

[0135] Based on the values ​​of the first loss function and the second loss function, the value of the total loss function is calculated, which can be achieved as follows:

[0136] B1, obtain a first weight of the first loss function and a second weight of the second loss function;

[0137] B2, calculate the value of the total loss function according to the value of the first loss function, the value of the second loss function, the first weight and the second weight.

[0138] Obtain a first weight of the first loss function and a second weight of the second loss function. The first weight and the second weight can be adjusted according to actual conditions or obtained through machine learning.

[0139] Optionally, during the training of the speech recognition model, the word error rate loss function and the cross entropy loss function L are minimized. CE Perform interpolation, the total loss function L is:

[0140]

[0141] Among them, the L MWER (x,y * ) is characterized as minimizing the word error rate loss function, L CEis the cross entropy loss function, (x,y * ) is a set of target samples, where x is the sample speech, y * is the sample text, and λ is the weight of the cross entropy loss function.

[0142] In this formula, the weight of minimizing the word error rate loss function is 1, and the weight of the cross entropy loss function is λ, and the value of λ can be obtained through machine learning.

[0143] Different weights are set for the first loss function and the second loss function, respectively, so that different weights are assigned to different loss functions according to the characteristics of the target sample, so as to accurately obtain the difference between the predicted text and the sample text of the target sample.

[0144] In the solution provided in this embodiment, the total loss function of the speech recognition model is a fusion of two loss functions. The weights of the first loss function and the second loss function can be adjusted according to actual conditions. If the value of the first loss function corresponding to a certain predicted text is large, it indicates that the probability of the predicted text being misrecognized is high. The first loss function is obtained based on the probability of each candidate recognition result and the number of misrecognized characters. Compared with calculating the loss function based on the probability of a single candidate recognition result, this method can more comprehensively and accurately measure the difference between the predicted text and the sample text, so as to improve the recognition accuracy of the speech recognition model and the number of correctly recognized characters based on the first loss function.

[0145] Optionally, the above-mentioned speech recognition model is an end-to-end speech recognition model based on a transformer structure.

[0146] The transformer-based end-to-end speech recognition model uses a self-attention mechanism to understand the current speech to be recognized through its context. It has a strong ability to extract semantic features, so that for words with the same or similar pronunciation in a sentence, the recognition results can be judged based on the surrounding words and the previous and next sentences, and the recognition results are more accurate. Moreover, the transformer-structured end-to-end speech recognition model solves the problem that each task in the traditional speech recognition method is independent and cannot be jointly optimized. The framework of a single neural network becomes simpler. As the number of model layers becomes deeper and the training data becomes larger, the accuracy increases. The transformer-structured end-to-end speech recognition model can better utilize and adapt to the new hardware parallel computing capabilities and has a faster computing speed, which means that the end-to-end speech recognition model based on the transformer structure can complete the transcription of speech of the same length in a shorter time and is more able to meet the needs of real-time transcription.

[0147] Based on the solution provided in the above embodiments of this application, the inventors conducted the following experiments, using 2000 hours of vehicle-mounted voice data and 1800 hours of voice data for training and testing. The experimental results are shown in Table 3.

[0148] Table 3

[0149]

[0150] Among them, the NWER percentage is the proportion of speech recognition models trained with the target sample to minimize the word error rate during the test. 0% refers to the recognition model that has not been trained to minimize the word error rate, that is, the recognition model after preliminary training. 100% (existing technology) refers to the use of existing technology to minimize the word error rate training. For 2000 hours of voice data on board, the training time of the existing technology is T1, and for 18,000 hours of voice data on board, the training time of the existing technology is T2. 20% refers to the use of the trained speech recognition model provided by this application to minimize the word error rate training, but the training time of the speech recognition model is T2. The training time of the existing technology is 20%, and 40% is the training time for minimizing the word error rate using the trained speech recognition model provided by this application, but the training time of the speech recognition model is 40% of the training time of the existing technology. It can be seen from the experimental data in Table 3 that when the recognition error rates are equal, the training time of the speech recognition model provided by this application is 40% of the existing technology, which greatly reduces the training time of the speech recognition model; compared with the recognition model after preliminary training, the recognition error rate of the speech recognition model after target data training decreases; when training the speech recognition model, the larger the amount of training data, the lower the recognition error rate of the speech recognition model.

[0151] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application also provides a speech recognition device 600, such as Figure 6 As shown, the device may include: a speech acquisition module 610 for identifying speech, and a recognition result acquisition module 620, wherein the speech recognition model in the recognition result acquisition module is obtained by training with a training device 700, and the training device 700 includes: a preliminary training module 710, a target sample screening module 720, and a model retraining module 730, wherein:

[0152] The speech acquisition module 610 is used to acquire the speech to be recognized;

[0153] A recognition result obtaining module 620 is used to recognize the speech to be recognized through a speech recognition model to obtain a speech recognition result;

[0154] The speech recognition model is obtained by training with a training device 700, which includes:

[0155] A preliminary training module 710 is used to train the initial speech recognition model based on each initial training sample to obtain a recognition model after preliminary training;

[0156] The target sample screening module 720 is used to obtain the corresponding recognition error rate representation information of each initial training sample when it is recognized by the recognition model after preliminary training, and select the target sample from each initial training sample based on the representation information;

[0157] The model retraining module 730 is used to train the recognition model after preliminary training based on each target sample to obtain a speech recognition model.

[0158] The speech recognition device provided in the present application uses recognition error rate characterization information to screen initial training samples to obtain target samples. It can screen out initial training samples with higher recognition error rates as target samples, and train the speech recognition model based on the target samples. Compared with the method of using all initial training samples for training in the prior art, the amount of training data of the speech recognition model is greatly reduced, and the recognition accuracy of the speech recognition model is improved.

[0159] Optionally, the initial training sample includes a sample speech and a sample text corresponding to the sample speech. The target sample screening module 720 further includes:

[0160] The recognition sample speech unit is used to recognize the sample speech of an initial training sample using the recognition model, and obtain at least two recognition probabilities that are ranked higher in descending order among the recognition probabilities corresponding to each character in the sample text of the sample;

[0161] a character information entropy determination unit, configured to determine the information entropy corresponding to each character in the sample text according to the at least two recognition probabilities corresponding to each character;

[0162] Determine a representation information unit for determining recognition error rate representation information corresponding to the initial training sample based on the information entropy corresponding to each character in the sample text.

[0163] Optionally, for an initial training sample, a representation information unit is determined, specifically for:

[0164] The maximum information entropy among the information entropies corresponding to the characters in the sample text of the initial training sample is used as the recognition error rate representation information corresponding to the initial training sample.

[0165] Optionally, identifying sample speech units is specifically used to:

[0166] Recognizing the sample speech of the initial training sample using the recognition model to obtain at least two recognition probabilities before normalization corresponding to each character in the sample text of the sample, wherein the at least two recognition probabilities before normalization are at least two of the recognition probabilities before normalization that are ranked higher in descending order;

[0167] Normalization is performed on the at least two recognition probabilities before normalization to obtain at least two recognition probabilities corresponding to each character.

[0168] Optionally, the initial training sample includes a sample speech and a sample text corresponding to the sample speech. The model retraining module 730 is specifically configured to:

[0169] Iteratively train the preliminarily trained recognition model based on each target sample until the total loss function corresponding to the recognition model reaches the convergence condition;

[0170] Among them, the input of the recognition model is the sample speech of the target sample, the output of the recognition model is the predicted text corresponding to the sample speech, and the value of the total loss function represents the difference between the predicted text and the sample text of the sample speech of each target sample.

[0171] Optionally, the total loss function in the model retraining module 730 includes a first loss function and a second loss function, the value of the first loss function represents the character difference between the predicted text and the sample text of the sample speech of each target sample, and the value of the second loss function represents the text difference between the predicted text and the sample text of the sample speech of the target sample.

[0172] Optionally, the training device 700 further includes:

[0173] A beam search decoding module is used to perform beam search decoding on the sample speech of each target sample using the recognition model for each training session, and obtain a set number of candidate recognition results corresponding to the sample speech and the probability corresponding to each candidate recognition result;

[0174] A predicted text acquisition module is used to acquire the predicted text corresponding to the sample speech;

[0175] A module for determining the number of incorrect characters is used to determine the number of incorrect characters in each candidate recognition result based on the candidate recognition result and the sample text;

[0176] A first loss function module, configured to calculate a value of a first loss function based on the probability corresponding to each candidate recognition result of each target sample and the number of erroneous characters;

[0177] A second loss function module, configured to calculate a value of a second loss function based on the predicted text and the sample text of each sample speech;

[0178] The total loss function module is used to calculate the value of the total loss function according to the value of the first loss function and the value of the second loss function.

[0179] Optionally, the first loss function module is specifically used to:

[0180] For each target sample, determining an average value of the erroneous characters corresponding to the target sample based on the number of erroneous characters corresponding to each candidate recognition result of the target sample;

[0181] For each candidate recognition result of each target sample, determining the edit distance between the candidate recognition result and the sample text;

[0182] For each target sample, determine a training loss value corresponding to the target sample based on the probability and edit distance corresponding to each candidate recognition result of the target sample and the average value corresponding to the target sample;

[0183] The value of the first loss function is obtained according to the training loss value of each target sample.

[0184] Optionally, the first loss function in the training device 700 is a word error rate minimization loss function, and the second loss function is a cross entropy loss function.

[0185] Optionally, the training device 700 further includes:

[0186] A weight acquisition module, configured to acquire a first weight of the first loss function and a second weight of the second loss function;

[0187] The total loss function is calculated as follows:

[0188] The value of the total loss function is calculated according to the first weight, the second weight, the value of the first loss function, and the value of the second loss function.

[0189] Optionally, the speech recognition model in the speech recognition device is an end-to-end speech recognition model based on a transformer structure.

[0190] The speech recognition device of the embodiment of the present application can execute the speech recognition method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module and unit in the speech recognition device in each embodiment of the present application correspond to the steps in the speech recognition method in each embodiment of the present application. For the detailed functional description of each module of the speech recognition device, please refer to the description of the corresponding speech recognition method shown in the previous text, and will not be repeated here.

[0191] Based on the same principle as the method shown in the embodiment of the present application, an electronic device is also provided in the embodiment of the present application, which may include but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the speech recognition method shown in any optional embodiment of the present application by calling the computer program. Compared with the prior art, the present application uses recognition error rate characterization information to screen the initial training samples, and can screen out initial training samples with a higher recognition error rate as target samples, so as to train the speech recognition model based on the target samples. Compared with the method of using all initial training samples for training in the prior art, the amount of training data of the speech recognition model is greatly reduced, and the recognition accuracy of the speech recognition model is improved; moreover, the target samples are selected from the initial training samples according to the recognition error rate, so as to realize targeted training for the recognition error rate, which is conducive to improving the efficiency of obtaining the trained speech recognition model.

[0192] In an alternative embodiment, an electronic device is provided, such as Figure 7 As shown, Figure 7 The electronic device 4000 shown may be a server, including a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0193] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0194] Bus 4002 may include a path for transmitting information between the above components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0195] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0196] The memory 4003 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0197] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0198] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0199] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0200] It should be noted that the computer-readable medium mentioned above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0201] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0202] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0203] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the speech recognition method provided in the various optional implementations described above.

[0204] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0205] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0206] The modules described in the embodiments of the present application may be implemented in software or hardware. In some cases, the name of a module does not necessarily limit the module itself. For example, a recognition result obtaining module may also be described as a "speech recognition result obtaining module."

[0207] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A speech recognition method, characterized in that: include: Get the speech to be recognized; The speech to be recognized is recognized by a speech recognition model to obtain a speech recognition result; wherein the speech recognition model is trained in the following manner: Training the initial speech recognition model based on each initial training sample to obtain a preliminarily trained recognition model; Obtaining recognition error rate representation information corresponding to each of the initial training samples when recognized by the recognition model after the preliminary training; Selecting a target sample from each of the initial training samples according to the representation information; Training the preliminarily trained recognition model based on each of the target samples to obtain the speech recognition model; The initial training sample includes a sample speech and a sample text corresponding to the sample speech; The obtaining of the recognition error rate characterization information corresponding to each of the initial training samples when recognized by the recognition model after the preliminary training includes: For one of the initial training samples, the sample speech of the sample is recognized using the recognition model to obtain at least two recognition probabilities that are ranked higher in descending order among the recognition probabilities corresponding to each character in the sample text of the sample; Determining the information entropy corresponding to each character according to the at least two recognition probabilities corresponding to each character in the sample text; Determining recognition error rate representation information corresponding to the initial training sample based on the information entropy corresponding to each character in the sample text; For an initial training sample, determining the recognition error rate representation information corresponding to the initial training sample based on the information entropy corresponding to each character in the sample text includes: The maximum information entropy among the information entropies corresponding to the characters in the sample text of the initial training sample is used as the recognition error rate representation information corresponding to the initial training sample.

2. The method according to claim 1, characterized in that The method of using the recognition model to recognize the sample speech of the sample and obtaining at least two recognition probabilities corresponding to each character in the sample text of the sample that are ranked higher in descending order includes: Recognizing a sample speech of an initial training sample using the recognition model to obtain at least two recognition probabilities before normalization corresponding to each character in the sample text of the sample, wherein the at least two recognition probabilities before normalization are at least two of the recognition probabilities before normalization that are ranked higher in descending order; Normalization is performed on the at least two recognition probabilities before normalization to obtain at least two recognition probabilities corresponding to each character.

3. The method according to claim 1, characterized in that The initial training sample includes a sample speech and a sample text corresponding to the sample speech; The training of the preliminarily trained recognition model based on each of the target samples includes: Iteratively training the preliminarily trained recognition model based on each of the target samples until the total loss function corresponding to the recognition model reaches a convergence condition; Among them, the input of the recognition model is the sample speech of the target sample, and the output of the recognition model is the predicted text corresponding to the sample speech. The value of the total loss function represents the difference between the predicted text and the sample text of the sample speech of each target sample.

4. The method according to claim 3, characterized in that The total loss function includes a first loss function and a second loss function, the value of the first loss function represents the character difference between the predicted text and the sample text of the sample speech of each target sample, and the value of the second loss function represents the text difference between the predicted text and the sample text of the sample speech of the target sample.

5. The method according to claim 4, characterized in that The iterative training of the preliminarily trained recognition model based on each target sample includes: For each training session, for each target sample, beam search decoding is performed on the sample speech of the target sample using the recognition model to obtain a set number of candidate recognition results corresponding to the sample speech and a probability corresponding to each candidate recognition result; Obtaining predicted text corresponding to the sample speech; For each candidate recognition result, determining the number of incorrect characters in the candidate recognition result based on the candidate recognition result and the sample text; Calculating a value of a first loss function based on the probability and the number of erroneous characters corresponding to each candidate recognition result of each target sample; Calculating a value of a second loss function based on the predicted text and the sample text of each of the sample speech; The value of the total loss function is calculated according to the value of the first loss function and the value of the second loss function.

6. The method according to claim 5, characterized in that Calculating the value of the first loss function according to the probability and the number of erroneous characters corresponding to each candidate recognition result of each target sample includes: For each target sample, determining an average value of the erroneous characters corresponding to the target sample based on the number of erroneous characters corresponding to each candidate recognition result of the target sample; For each candidate recognition result of each target sample, determining the edit distance between the candidate recognition result and the sample text; For each target sample, determining a training loss value corresponding to the target sample based on the probability and edit distance corresponding to each candidate recognition result of the target sample and the average value corresponding to the target sample; The value of the first loss function is obtained according to the training loss value of each target sample.

7. The method according to any one of claims 4 to 6, characterized in that The first loss function is a word error rate minimization loss function, and the second loss function is a cross entropy loss function.

8. The method according to any one of claims 4 to 6, characterized in that Also includes: Obtaining a first weight of the first loss function and a second weight of the second loss function; The value of the total loss function is calculated as follows: The value of the total loss function is calculated according to the first weight, the second weight, the value of the first loss function, and the value of the second loss function.

9. The method according to claim 1, characterized in that The speech recognition model is an end-to-end speech recognition model based on the transformer structure.

10. A speech recognition device, characterized in that: include: A voice acquisition module to be recognized, used to acquire the voice to be recognized; A recognition result obtaining module is used to recognize the speech to be recognized through a speech recognition model to obtain a speech recognition result; The speech recognition model is obtained by training with a training device, and the training device includes: A preliminary training module is used to train the initial speech recognition model based on each initial training sample to obtain a recognition model after preliminary training; a target sample screening module, configured to obtain the corresponding recognition error rate characterization information of each of the initial training samples when the initial training sample is recognized by the recognition model after the preliminary training, and select a target sample from each of the initial training samples according to the characterization information; A model retraining module, configured to train the preliminarily trained recognition model based on each target sample to obtain the speech recognition model; The initial training sample includes a sample speech and a sample text corresponding to the sample speech; The target sample screening module includes: A sample speech recognition unit is configured to recognize the sample speech of an initial training sample using the recognition model, and obtain at least two recognition probabilities that are ranked higher in descending order among the recognition probabilities corresponding to each character in the sample text of the sample; a character information entropy determination unit, configured to determine the information entropy corresponding to each character in the sample text according to the at least two recognition probabilities corresponding to each character; Determine a representation information unit, which is used to determine the recognition error rate representation information corresponding to the initial training sample based on the information entropy corresponding to each character in the sample text; For one of the initial training samples, determining the representation information unit is specifically used to: The maximum information entropy among the information entropies corresponding to the characters in the sample text of the initial training sample is used as the recognition error rate representation information corresponding to the initial training sample.

11. The device according to claim 10, characterized in that The recognition sample speech unit is specifically used for: Recognizing a sample speech of an initial training sample using the recognition model to obtain at least two recognition probabilities before normalization corresponding to each character in the sample text of the sample, wherein the at least two recognition probabilities before normalization are at least two of the recognition probabilities before normalization that are ranked higher in descending order; Normalization is performed on the at least two recognition probabilities before normalization to obtain at least two recognition probabilities corresponding to each character.

12. The device according to claim 10, characterized in that The initial training sample includes a sample speech and a sample text corresponding to the sample speech; The model retraining module is specifically used to: Iteratively training the preliminarily trained recognition model based on each of the target samples until the total loss function corresponding to the recognition model reaches a convergence condition; Among them, the input of the recognition model is the sample speech of the target sample, and the output of the recognition model is the predicted text corresponding to the sample speech. The value of the total loss function represents the difference between the predicted text and the sample text of the sample speech of each target sample.

13. The device according to claim 12, characterized in that The total loss function in the model retraining module includes a first loss function and a second loss function. The value of the first loss function represents the character difference between the predicted text and the sample text of the sample speech of each target sample, and the value of the second loss function represents the text difference between the predicted text and the sample text of the sample speech of the target sample.

14. The device according to claim 13, characterized in that The training device further comprises: A beam search decoding module is configured to, for each training session, perform beam search decoding on the sample speech of each target sample using the recognition model to obtain a set number of candidate recognition results corresponding to the sample speech and a probability corresponding to each candidate recognition result; A predicted text acquisition module is used to acquire the predicted text corresponding to the sample speech; a module for determining the number of incorrect characters, configured to determine, for each candidate recognition result, the number of incorrect characters in the candidate recognition result based on the candidate recognition result and a sample text; A first loss function module, configured to calculate a value of a first loss function based on the probability and the number of erroneous characters corresponding to each candidate recognition result of each target sample; A second loss function module, configured to calculate a value of a second loss function based on the predicted text and the sample text of each of the sample speech; The total loss function module is used to calculate the value of the total loss function according to the value of the first loss function and the value of the second loss function.

15. The device according to claim 14, characterized in that The first loss function module is specifically used to: For each target sample, determining an average value of the erroneous characters corresponding to the target sample based on the number of erroneous characters corresponding to each candidate recognition result of the target sample; For each candidate recognition result of each target sample, determining the edit distance between the candidate recognition result and the sample text; For each target sample, determining a training loss value corresponding to the target sample based on the probability and edit distance corresponding to each candidate recognition result of the target sample and the average value corresponding to the target sample; The value of the first loss function is obtained according to the training loss value of each target sample.

16. The device according to any one of claims 13 to 15, characterized in that The first loss function in the training device is a word error rate minimization loss function, and the second loss function is a cross entropy loss function.

17. The device according to any one of claims 13 to 15, characterized in that The training device further comprises: A weight acquisition module, configured to acquire a first weight of the first loss function and a second weight of the second loss function; The value of the total loss function is calculated as follows: The value of the total loss function is calculated according to the first weight, the second weight, the value of the first loss function, and the value of the second loss function.

18. The device according to claim 10, characterized in that The speech recognition model in the speech recognition device is an end-to-end speech recognition model based on a transformer structure.

19. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the speech recognition method according to any one of claims 1 to 9 is implemented.

20. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the speech recognition method according to any one of claims 1 to 9.

21. A computer program product, characterized in that The computer program product includes computer instructions, and a processor executes the computer instructions to implement the speech recognition method according to any one of claims 1 to 9.

Images