A speech recognition processing method and apparatus

Abstract

The application discloses a speech recognition processing method and device, and relates to the technical field of computers. A specific implementation manner of the method comprises the following steps: in response to monitoring input speech information, performing speech recognition transcription on the speech information to obtain a transcription result, calling an error detector to detect error characters in the transcription result; calling a pre-trained language model, performing mask processing on the positions of the error characters in the transcription result, predicting candidate characters at the mask positions, and calculating the probability of each candidate character appearing at the mask positions; calculating the pronunciation feature distance between each candidate character and the error character, combining the probability of each candidate character appearing at the mask positions, and obtaining a selection probability value of each candidate character; screening a target candidate character with the largest selection probability value, replacing the error character with the target candidate character, obtaining a corrected transcription text, and returning the corrected transcription text for display. The implementation manner considers context semantics and pronunciation features, and effectively corrects the transcription result of speech recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a speech recognition processing method and device. BACKGROUND

[0002] The quality of speech recognition is often measured by the error rate, i.e., the word error rate (WER). Although the automatic speech recognition (ASR) technology is constantly developing, and the amount of available data is also expanding, there are still many errors in the transcription results of the ASR system, which are similar in speech semantics. Therefore, the post-processing of the speech recognition system is very important, and the error correction can well solve the recognition error problem of speech semantics.

[0003] At present, the error correction methods mainly include two types: 1) a grammar rule-based error correction method, which mainly adopts handwritten grammar rules to correct the transcription results of the ASR by using a rule matching method. 2) a semantic error correction method using a large pre-training model in natural language processing, such as using BERT (Pre-training of Deep Bidirectional Transformers for Language Understanding, an open-source pre-training language model based on a large amount of corpus) to correct the transcription results of the ASR.

[0004] In the process of implementing the present application, the inventors have found that the method 1 can only correct the errors that meet the grammar rules, without considering the information of speech semantics, and the method 2 ignores the speech information and cannot effectively correct the words with the same or similar pronunciation. SUMMARY

[0005] Therefore, the embodiments of the present application provide a speech recognition processing method and device, which can at least solve the problem that the speech semantic features are not considered in the prior art, resulting in low effective error correction efficiency.

[0006] To achieve the above object, according to one aspect of the embodiments of the present application, a speech recognition processing method is provided, which comprises:

[0007] In response to monitoring the input speech information, performing speech recognition transcription on the speech information to obtain a transcription result, and calling an error detector to detect error characters in the transcription result;

[0008] Calling a pre-training language model to perform mask processing on the positions of the error characters in the transcription result, predicting candidate characters at the mask positions, and calculating the probability of each candidate character appearing at the mask position;

[0009] calculate a pronunciation feature distance of each candidate character and the error character, combine a probability of each candidate character appearing at the mask position to obtain a selection probability value of each candidate character;

[0010] select a target candidate character with the largest selection probability value, replace the error character with the target candidate character to obtain a corrected transcription text and return for display.

[0011] Optionally, before the error detector is called to detect the error character in the transcription result, the method further comprises:

[0012] obtain a set of sample voice information, call a voice recognition program to perform voice recognition transcription on each sample voice information to obtain a plurality of sample transcription results;

[0013] receive a labeling operation on an error character in each sample transcription result, and input the sample voice information and the corresponding labeled error character to fine-tune the pre-trained language model to obtain the error detector.

[0014] The fine-tuning of the pre-trained language model comprises: adding a linear layer neural network based on the trained pre-trained language model, and only adjusting the parameters of the linear layer neural network during training.

[0015] Optionally, the calculation of the pronunciation feature distance of each candidate character and the error character comprises:

[0016] obtain a first initial consonant, a first vowel, and a first tone of the error character in pinyin, and a second initial consonant, a second vowel, and a second tone of the candidate character in pinyin;

[0017] calculate an initial consonant Euler distance of the first initial consonant and the second initial consonant, a vowel Euler distance of the first vowel and the second vowel, and a pronunciation tone distance of the first tone and the second tone to obtain the pronunciation feature distance of the candidate character and the error character.

[0018] Optionally, the obtaining of the corrected transcription text and the returning for display further comprises:

[0019] calling a rule transcription program to search for a character in the transcription text in a rule transcription dictionary; wherein the rule transcription dictionary comprises characters that need to be transcribed by grammar rules;

[0020] in response to a search result being present, determining a to-be-replaced character corresponding to the character in the transcription text in the rule transcription dictionary to replace the character in the transcription text with the to-be-replaced character.

[0021] To achieve the above objectives, according to another aspect of the present invention, a speech recognition processing apparatus is provided, comprising:

[0022] The detection module is used to respond to the detection of input voice information, perform speech recognition and transcription on the voice information, obtain the transcription result, and call the error detector to detect erroneous characters in the transcription result;

[0023] The prediction module is used to call a pre-trained language model to mask the position of the erroneous character in the transcription result, predict candidate characters at the mask position, and calculate the probability of each candidate character appearing at the mask position.

[0024] The calculation module is used to calculate the pronunciation feature distance between each candidate character and the erroneous character, and combine the probability of each candidate character appearing at the mask position to obtain the selection probability value of each candidate character;

[0025] The replacement module is used to filter out the target candidate character with the highest selection probability value, replace the erroneous character with the target candidate character, obtain the corrected transcribed text, and return it for display.

[0026] Optionally, a training module is also included for:

[0027] Obtain a set of sample speech information, call the speech recognition program to perform speech recognition and transcription on each sample speech information, and obtain multiple sample transcription results;

[0028] It receives the annotation operation for erroneous characters in the transcription result of each sample, and uses the sample speech information and the corresponding annotated erroneous characters as input to fine-tune the pre-trained language model to obtain the error detector;

[0029] The fine-tuning of the pre-trained language model includes adding a linear layer neural network to the pre-trained language model, and adjusting the parameters of the linear layer neural network only during training.

[0030] Optionally, the computing module is used for:

[0031] Obtain the first initial, first final, and first tone of the erroneous character in Pinyin, and the second initial, second final, and second tone of the candidate character in Pinyin;

[0032] Calculate the initial consonant Euler distance between the first initial consonant and the second initial consonant, the vowel Euler distance between the first final vowel and the second final vowel, and the pronunciation tone distance between the first tone and the second tone, and sum them to obtain the pronunciation feature distance between the candidate character and the erroneous character.

[0033] Optionally, the replacement module is further configured to:

[0034] The rule-based transcription program is invoked to search for characters in the transcribed text in the rule-based transcription dictionary; wherein, the rule-based transcription dictionary includes characters that need to be transcribed as defined by grammatical rules;

[0035] In response to a search result indicating existence, the character to be replaced in the rule transcription dictionary corresponding to the character in the transcribed text is determined, so as to replace the character in the transcribed text with the character to be replaced.

[0036] To achieve the above objectives, according to another aspect of the present invention, a speech recognition processing electronic device is provided.

[0037] The electronic device of this invention includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the above-described speech recognition processing methods.

[0038] To achieve the above objectives, according to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the above-described speech recognition processing methods.

[0039] According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: A neural network layer is added to the pre-trained language model for training to obtain an error detector, which is used to detect errors in the ASR transcription results and obtain erroneous characters; the pre-trained language model is called to consider contextual semantic features to predict candidate characters for the position of the erroneous character; the pronunciation features of each character are considered, and the pronunciation features and contextual semantics are effectively fused through a balancing function to reorder the candidate characters; furthermore, grammatical rule matching is performed considering the application scenario to obtain the final speech recognition transcription result.

[0040] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0041] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0042] Figure 1 This is a schematic diagram of the main flow of a speech recognition processing method according to an embodiment of the present invention;

[0043] Figure 2 This is a schematic flowchart of a specific speech recognition processing method according to an embodiment of the present invention;

[0044] Figure 3It is a schematic diagram of the main modules of a voice recognition processing device according to an embodiment of the present invention;

[0045] Figure 4 It is an exemplary system architecture diagram to which an embodiment of the present invention can be applied;

[0046] Figure 5 It is a schematic diagram of the structure of a computer system of a mobile device or a server suitable for implementing an embodiment of the present invention. Detailed implementation manners

[0047] The following describes exemplary embodiments of the present invention with reference to the accompanying drawings. Various details of the embodiments of the present invention are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted below.

[0048] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other. The acquisition, storage, use, processing, etc. of data in the technical solution of this application all comply with the relevant regulations of national laws and regulations.

[0049] The transcription results of existing ASR systems may have recognition errors where the speech semantics are similar. For semantic similarity errors, such as the use of words like "de", "děi", "dì", etc., in some sentences, the original word may be "noon", but through context and speech recognition, the result is "midday", and "midday" needs to be corrected to "noon" through post-processing; for speech recognition errors, such as "chóng lái" and "cóng lái", "lǚ xíng" and "lǚ xíng", these words are prone to recognition errors, so post-processing is required for error correction.

[0050] See Figure 1 , which shows the main flowchart of a voice recognition processing method provided by an embodiment of the present invention, including the following steps:

[0051] S101: In response to detecting input voice information, perform voice recognition transcription on the voice information to obtain a transcription result, and call an error detector to detect error characters in the transcription result;

[0052] S102: Call a pre-trained language model to mask the positions of the error characters in the transcription result, predict candidate characters at the masked positions, and calculate the probability of each candidate character appearing at the masked position;

[0053] S103: Calculate the pronunciation feature distance between each candidate character and the erroneous character, and combine the probability of each candidate character appearing at the mask position to obtain the selection probability value of each candidate character;

[0054] S104: Select the target candidate character with the highest selection probability value, replace the erroneous character with the target candidate character, obtain the corrected transcribed text, and return it for display.

[0055] In the above embodiments, the scope of application for step S101 is quite broad, including terminals such as intelligent customer service, smart speakers, smart vehicles, and intelligent chatbots. The speech-to-text function in these terminals utilizes speech recognition technology. Furthermore, a crucial technology in this solution is acquiring pinyin pronunciation features, thus it is preferably applied in Chinese speech recognition scenarios.

[0056] When a user inputs information into the terminal, the information can be text, images, video, or voice. When the terminal detects that the input is voice, it uses Automatic Speech Recognition (ASR) technology to perform speech recognition and obtain a transcription result. An error detector is then used to detect erroneous characters in the transcription result.

[0057] Before proceeding, a binary classification error detector needs to be built, for example, 1 represents an incorrect character and 0 represents a correct character. The characteristics of a typical trained model are: 1) training from scratch requires a large amount of data, computation time, and computational resources; 2) it carries risks such as model non-convergence, insufficient parameter optimization, low accuracy, low model generalization ability, and susceptibility to overfitting. This solution uses a pre-trained model (a model already trained using the dataset) fine-tuning approach. Based on the existing trained BERT model, a linear layer neural network is added according to the binary classification task. During training, only the parameters of the newly added neural network layer are updated, without updating the original parameters of the BERT model or modifying its network structure, effectively avoiding the aforementioned potential problems.

[0058] Specifically, a sample speech information set is acquired, and Automatic Speech Recognition (ASR) technology is used to perform speech recognition on each sample speech information set, resulting in multiple sample transcription results. Based on manual annotation of erroneous Chinese characters in the sample transcription results, the BERT model is fine-tuned and trained using the sample speech information set and the erroneous Chinese characters in the transcription results as positive examples, resulting in an error detector. The BERT model is a pre-trained language model applicable to multiple languages; considering that this scheme mainly involves Chinese speech recognition, a Chinese BERT model is adopted.

[0059] For step S102, after calling the error detector to detect erroneous characters in the current transcription result, the position of the erroneous character in the transcription result is determined, and the input at this position is represented by a MASK. Here, MASK: a mask, is a prediction method used in BERT. In computer science and digital logic, it refers to a string of binary numbers that, through bitwise operations, mask a specified position to achieve the desired result. The BERT model then re-predicts the character at the masked position, obtaining a series of candidate character results, which serve as the candidate character set for that position.

[0060] For example, in {0 1 0 0 0 0 1 0 0 0 1}, for the first erroneous character, the candidate character set 1 is {candidate character 1, candidate character 2, candidate character 3...candidate character 10}; for the second erroneous character, the candidate character set 2 is {candidate character 11, candidate character 12, candidate character 13}; and for the third erroneous character, the candidate character set 3 is {candidate character 14, candidate character 15, candidate character 16, candidate character 17}.

[0061] It's important to note that the input to the aforementioned error detector is the transcription result of speech recognition, and the output is a classification result, indicating whether it's an error character. Here, the input to the BERT model is the sentence after masking the classification result. Although both use the BERT model, they are two BERT models with completely different purposes. Furthermore, this method leverages the contextual modeling capability of the BERT model to predict the mask position, resulting in a probability. This probability corresponds to a vector composed of the probabilities of all words in the dictionary during training. For example, if the dictionary contains 10 words, the model will output 10 numbers, each representing a probability, indicating how likely the character is at that position. This is typically a value between 0 and 1, such as 0.5 for A, 0.2 for B, and 0.3 for C. Therefore, we obtain candidate character 1 - probability 1, candidate character 2 - probability 2, ..., candidate character 10 - probability 10.

[0062] For step S103, the open-source method DIMSIM (an open-source algorithm for accurate Chinese speech similarity based on learning high-dimensional encoding) is used to calculate the pronunciation feature distance between each candidate character in the candidate character set and the transcribed error character. The formula is as follows:

[0063]

[0064] in, These represent the initial consonant, final vowel, and tone of the character in Chinese Pinyin, respectively. For S... 声母 S 韵母 This scheme preferably uses the Euler distance, while S 声调That is the pronunciation tone distance. Among them, the Euclidean distance is the most intuitive distance measurement method, and the distance between two points encountered in the calculation in space generally refers to the Euclidean distance.

[0065] The pronunciation tone distance of homophonic characters is 0, while the remaining S 声母 , S 韵母 are all greater than or equal to 0. Therefore, S(character 1, character 2) obtained by adding these three parts is also greater than or equal to 0. The greater the pronunciation feature distance between two characters with more obvious voice differences.

[0066] For example, the input voice information is "Nice to serve you". Through the ASR system, it is recognized that "兴" is "行". Then, the error detector is called to detect the error character "行". Assuming that the candidate characters include "兴", then calculate the pronunciation feature distance between the two characters "兴" and "行", that is, s(xing2, xing4). Among them, 行(xing2) is the error character and 兴(xing4) is the candidate character. Therefore, both the Euler distance of the initial consonant and the Euler distance of the final consonant are 0, and the pronunciation tone distance of the tone part is 2. Thus, the final calculated pronunciation feature distance between these two characters is 2.

[0067] The existing BERT model represents the feature of semantic information, and DIMSIM represents the feature of Chinese pinyin, which can provide effective feature supplementation for acoustic tasks such as speech recognition. And the existing error correction methods basically only consider the features of the semantic part. Therefore, integrating pronunciation features into semantic error correction can significantly improve speech recognition error correction.

[0068] The selection probability value of each candidate character is calculated through the balance function to balance the influence of the semantic features and pronunciation features of the candidate characters and the error characters in the final selection of candidate characters. The balance function is defined as follows:

[0069] Φ(P 候选字符 , S(c 错误字符 , c 候选字符 )) = P 候选字符 ×exp(-α×S(c 错误字符 , c 候选字符 ))

[0070] Among them, P 候选字符 represents the probability that the candidate character predicted by the BERT model through the context appears at the masked position, S(c 错误字符 , c 候选字符 ) represents the pronunciation feature distance between each candidate character and the error character, and α is a hyperparameter specified by the empirical value, representing the balance parameter.

[0071] After obtaining the selection probability values of each candidate character, the candidate character set can be re-sorted to determine the target candidate character with the largest selection probability value in each candidate character set. Then, the incorrect character can be replaced with the target candidate character to obtain the transcription text after pronunciation correction. It should be noted that the transcription text is the final result of speech recognition. Regardless of the number of masked positions, there will be only one transcription text for a single sentence of speech, that is, the final recognition result of the speech.

[0072] Taking the sentence "Nice to serve you" as an example again, the ASR recognition result is "Very high behavior to serve you", and the error detector identifies that "behavior" and "serve" are recognized incorrectly. Taking "behavior" as an example, the candidate character set at the position of "behavior" obtained by masked prediction BERT is {行, 性, 星, 兴} (xing, xing, xing, xing in Chinese pinyin), and the corresponding occurrence probabilities are {0.4, 0.25, 0.1, 0.25}, where 0.4 is the maximum probability. After calculating the features of each candidate character in the candidate character set, new selection probabilities {0.2, 0.1, 0.15, 0.55} may be obtained, which achieves the purpose of re-sorting the candidate characters, screening out the candidate character with the largest selection probability value, that is, "兴" (xing), and replacing "行" (xing) with "兴" (xing).

[0073] Through the above steps, a transcription text with a significantly reduced error rate can be obtained, so that the speech recognition result viewed by the user is relatively correct. In addition, considering the actual application scenario, the speech recognition result is normal characters, and in some scenarios, further conversion is required. Therefore, it needs to be customized according to different scenarios. For example, customize the words and phrases to be transcribed through grammar rules to generate a rule transcription dictionary, such as 杯具 -> 悲剧 (cup -> tragedy), 香菇 -> 想哭 (mushroom -> want to cry), 女孩纸 -> 女孩子 (girl -> girl), 稀饭 -> 喜欢 (porridge -> like), 偶 -> 我 (I).

[0074] After obtaining the transcription text through the above steps, use the transcription text as input and input it into the rule transcription program to determine the characters to be replaced corresponding to each character in the transcription text through the rule transcription dictionary, and perform character replacement to obtain the final output result. For example: 偶稀饭你 -> 我喜欢你 (I like you); 杯具啊 -> 悲剧啊 (cup ah -> tragedy ah); 为一个女孩纸香菇 -> 为一个女孩子想哭 (for a girl mushroom -> for a girl want to cry).

[0075] The above are cases of matching, such as 杯具, 香菇, 女孩纸, but for cases of non-matching, such as "我想要$旅行$" (I want $travel$), there are no matching characters in the rule transcription dictionary, which means that no processing needs to be done on this transcription text.

[0076] The method provided in the above embodiments trains an error detector to detect errors in ASR transcription results, predicts candidate characters based on the BERT model considering contextual semantic features to predict the position of the erroneous character, and reorders the candidate characters by combining the pronunciation features of each character, so as to effectively correct errors in the transcription results in a scientific way and reduce the word error rate of the transcribed text.

[0077] See Figure 2 The diagram illustrates a specific speech recognition processing method according to an embodiment of the present invention, including the following steps:

[0078] S201: Obtain a set of sample speech information, call the speech recognition program to perform speech recognition and transcription on each sample speech information, and obtain multiple sample transcription results;

[0079] S202: Receive the annotation operation for erroneous characters in the transcription result of each sample, and use the sample speech information and the corresponding annotated erroneous characters as input to fine-tune the pre-trained language model to obtain an error detector; wherein, fine-tuning the pre-trained language model includes: adding a linear layer neural network on the basis of the already trained pre-trained language model, and adjusting the parameters of the linear layer neural network only during training;

[0080] S203: In response to detecting input voice information, perform speech recognition and transcription on the voice information to obtain a transcription result, and call an error detector to detect erroneous characters in the transcription result;

[0081] S204: Call the pre-trained language model to mask the position of the erroneous character in the transcription result, predict the candidate character at the mask position, and calculate the probability of each candidate character appearing at the mask position;

[0082] S205: Obtain the first initial, first final, and first tone of the erroneous character in Chinese Pinyin, and the second initial, second final, and second tone of the candidate character in Chinese Pinyin;

[0083] S206: Calculate the initial consonant Eulerian distance between the first initial consonant and the second initial consonant, the vowel Eulerian distance between the first final vowel and the second final vowel, and the pronunciation tone distance between the first tone and the second tone, and sum them to obtain the pronunciation feature distance between the candidate character and the erroneous character;

[0084] S207: Based on the pronunciation feature distance and the probability of each candidate character appearing at the mask position, obtain the selection probability value of each candidate character;

[0085] S208: Select the target candidate character with the highest selection probability value, replace the erroneous character with the target candidate character, obtain the corrected transcribed text, and return it for display.

[0086] See Figure 3 The diagram shows the main modules of a speech recognition processing device 900 provided in an embodiment of the present invention, including:

[0087] The detection module 301 is used to respond to the detection of input voice information, perform speech recognition and transcription on the voice information, obtain the transcription result, and call the error detector to detect erroneous characters in the transcription result;

[0088] The prediction module 302 is used to call a pre-trained language model to mask the position of the erroneous character in the transcription result, predict candidate characters at the mask position, and calculate the probability of each candidate character appearing at the mask position.

[0089] The calculation module 303 is used to calculate the pronunciation feature distance between each candidate character and the erroneous character, and combine the probability of each candidate character appearing at the mask position to obtain the selection probability value of each candidate character;

[0090] The replacement module 304 is used to filter out the target candidate character with the highest selection probability value, replace the erroneous character with the target candidate character, obtain the corrected transcribed text, and return it for display.

[0091] The apparatus for implementing this invention further includes a training module, used for:

[0092] Obtain a set of sample speech information, call the speech recognition program to perform speech recognition and transcription on each sample speech information, and obtain multiple sample transcription results;

[0093] It receives the annotation operation for erroneous characters in the transcription result of each sample, and uses the sample speech information and the corresponding annotated erroneous characters as input to fine-tune the pre-trained language model to obtain the error detector;

[0094] The fine-tuning of the pre-trained language model includes adding a linear layer neural network to the pre-trained language model, and adjusting the parameters of the linear layer neural network only during training.

[0095] In the apparatus of this invention, the calculation module 303 is used for:

[0096] Obtain the first initial, first final, and first tone of the erroneous character in Pinyin, and the second initial, second final, and second tone of the candidate character in Pinyin;

[0097] Calculate the initial consonant Euler distance between the first initial consonant and the second initial consonant, the vowel Euler distance between the first final vowel and the second final vowel, and the pronunciation tone distance between the first tone and the second tone, and sum them to obtain the pronunciation feature distance between the candidate character and the erroneous character.

[0098] In the apparatus of this invention, the replacement module 304 is further configured to:

[0099] The rule-based transcription program is invoked to search for characters in the transcribed text in the rule-based transcription dictionary; wherein, the rule-based transcription dictionary includes characters that need to be transcribed as defined by grammatical rules;

[0100] In response to a search result indicating existence, the character to be replaced in the rule transcription dictionary corresponding to the character in the transcribed text is determined, so as to replace the character in the transcribed text with the character to be replaced.

[0101] Furthermore, the specific implementation details of the device described in the embodiments of the present invention have been described in detail in the above-described method, so the details will not be repeated here.

[0102] Figure 4 An exemplary system architecture 400 to which embodiments of the present invention can be applied is shown, including terminal devices 401, 402, 403, network 404, and server 405 (this is merely an example).

[0103] Terminal devices 401, 402, and 403 can be various electronic devices with displays and support for web browsing, and have various communication client applications installed. Users can use terminal devices 401, 402, and 403 to interact with server 405 through network 404 to receive or send messages, etc.

[0104] Network 404 is a medium used to provide a communication link between terminal devices 401, 402, 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0105] Server 405 can be a server providing various services. It should be noted that the method provided in this embodiment of the invention is generally executed by server 405. Correspondingly, the device is generally set in server 405. Specifically, it performs the following: in response to detecting input voice information, it performs speech recognition and transcription on the voice information to obtain a transcription result; it calls an error detector to detect erroneous characters in the transcription result; it calls a pre-trained language model to mask the position of the erroneous character in the transcription result, predicts candidate characters at the mask position, and calculates the probability of each candidate character appearing at the mask position; it calculates the pronunciation feature distance between each candidate character and the erroneous character, and combines the probability of each candidate character appearing at the mask position to obtain the selection probability value of each candidate character; it filters out the target candidate character with the highest selection probability value, replaces the erroneous character with the target candidate character, obtains the corrected transcribed text, and returns it for display.

[0106] It should be understood that Figure 4 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0107] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 500 suitable for implementing a terminal device of the present invention. Figure 5 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0108] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the system 500. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0109] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0110] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined above in the system of this invention.

[0111] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0113] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor may be described as including a detection module, a prediction module, a calculation module, and a replacement module. The names of these modules do not necessarily limit the module itself; for example, the replacement module may also be described as a "character replacement module."

[0114] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to perform any of the above-described speech recognition processing methods.

[0115] According to the technical solution of this embodiment of the invention, a neural network layer is added to the pre-trained language model for training to obtain an error detector, which is used to detect errors in the ASR transcription results and obtain erroneous characters; the pre-trained language model is called to consider the contextual semantic features to predict the candidate characters at the position of the erroneous character; the pronunciation features of each character are considered, and the pronunciation features and contextual semantics are effectively fused through a balancing function to reorder the candidate characters; furthermore, grammar rule matching is performed considering the application scenario to obtain the final speech recognition transcription result.

[0116] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A speech recognition processing method, characterized in that, include: In response to the detection of input speech information, speech recognition and transcription are performed on the speech information to obtain the transcription result, and an error detector is called to detect erroneous characters in the transcription result; wherein, the error detector is obtained by fine-tuning the pre-trained language model, and fine-tuning the pre-trained language model includes: adding a linear layer neural network on the basis of the already trained pre-trained language model, and only adjusting the parameters of the linear layer neural network during training, without updating the original parameters and network structure of the pre-trained language model; A pre-trained language model is invoked to mask the position of the erroneous character in the transcription result, predict candidate characters at the masked position, and calculate the probability of each candidate character appearing at the masked position; wherein, the probability corresponds to a vector composed of the probabilities of all words in the dictionary; Calculate the pronunciation feature distance between each candidate character and the erroneous character, and combine the probability of each candidate character appearing at the mask position to obtain the selection probability value of each candidate character; wherein, the pronunciation feature distance is weighted by a balance parameter, the exponent is taken, and multiplied by the occurrence probability to obtain a balance function, and the selection probability value of each candidate character is calculated through the balance function; The target candidate character with the highest selection probability value is selected, and the erroneous character is replaced with the target candidate character to obtain the corrected transcribed text, which is then returned for display.

2. The method according to claim 1, characterized in that, Before invoking the error detector to detect erroneous characters in the transcription result, the method further includes: Obtain a set of sample speech information, call the speech recognition program to perform speech recognition and transcription on each sample speech information, and obtain multiple sample transcription results; It receives the annotation operation for erroneous characters in the transcription result of each sample, and uses the sample speech information and the corresponding annotated erroneous characters as input to fine-tune the pre-trained language model to obtain the error detector.

3. The method according to claim 1 or 2, characterized in that, The calculation of the pronunciation feature distance between each candidate character and the erroneous character includes: Obtain the first initial, first final, and first tone of the erroneous character in Pinyin, and the second initial, second final, and second tone of the candidate character in Pinyin; Calculate the initial consonant Euler distance between the first initial consonant and the second initial consonant, the vowel Euler distance between the first final vowel and the second final vowel, and the pronunciation tone distance between the first tone and the second tone, and sum them to obtain the pronunciation feature distance between the candidate character and the erroneous character.

4. The method according to claim 1, characterized in that, The process of obtaining the corrected transcribed text and returning it for display also includes: The rule-based transcription program is invoked to search for characters in the transcribed text in the rule-based transcription dictionary; wherein, the rule-based transcription dictionary includes characters that need to be transcribed as defined by grammatical rules; In response to a search result indicating existence, the character to be replaced in the rule transcription dictionary corresponding to the character in the transcribed text is determined, so as to replace the character in the transcribed text with the character to be replaced.

5. A speech recognition processing device, characterized in that, include: The detection module is used to respond to the detected input speech information, perform speech recognition and transcription on the speech information to obtain the transcription result, and call an error detector to detect erroneous characters in the transcription result; wherein, the error detector is obtained by fine-tuning the pre-trained language model, and fine-tuning the pre-trained language model includes: adding a linear layer neural network on the basis of the already trained pre-trained language model, and only adjusting the parameters of the linear layer neural network during training, without updating the original parameters and network structure of the pre-trained language model; The prediction module is used to call a pre-trained language model to mask the position of the erroneous character in the transcription result, predict candidate characters at the masked position, and calculate the probability of each candidate character appearing at the masked position; wherein, the probability corresponds to a vector composed of the probabilities of all words in the dictionary; The calculation module is used to calculate the pronunciation feature distance between each candidate character and the erroneous character, and combine the probability of each candidate character appearing at the mask position to obtain the selection probability value of each candidate character; wherein, the pronunciation feature distance is weighted by a balance parameter, the exponent is taken, and multiplied by the occurrence probability to obtain a balance function, and the selection probability value of each candidate character is calculated by the balance function; The replacement module is used to filter out the target candidate character with the highest selection probability value, replace the erroneous character with the target candidate character, obtain the corrected transcribed text, and return it for display.

6. The apparatus according to claim 5, characterized in that, It also includes a training module for: Obtain a set of sample speech information, call the speech recognition program to perform speech recognition and transcription on each sample speech information, and obtain multiple sample transcription results; It receives the annotation operation for erroneous characters in the transcription result of each sample, and uses the sample speech information and the corresponding annotated erroneous characters as input to fine-tune the pre-trained language model to obtain the error detector.

7. The apparatus according to claim 5 or 6, characterized in that, The computing module is used for: Obtain the first initial, first final, and first tone of the erroneous character in Pinyin, and the second initial, second final, and second tone of the candidate character in Pinyin; Calculate the initial consonant Euler distance between the first initial consonant and the second initial consonant, the vowel Euler distance between the first final vowel and the second final vowel, and the pronunciation tone distance between the first tone and the second tone, and sum them to obtain the pronunciation feature distance between the candidate character and the erroneous character.

8. The apparatus according to claim 5, characterized in that, The replacement module is also used for: The rule-based transcription program is invoked to search for characters in the transcribed text in the rule-based transcription dictionary; wherein, the rule-based transcription dictionary includes characters that need to be transcribed as defined by grammatical rules; In response to a search result indicating existence, the character to be replaced in the rule transcription dictionary corresponding to the character in the transcribed text is determined, so as to replace the character in the transcribed text with the character to be replaced.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-4.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-4.

Images