Speech recognition error correction method and related equipment

Abstract

The invention discloses a speech recognition error correction method and device, a server, a computer readable storage medium and a computer program product. According to the method, a large language model is applied to a speech recognition error correction task, frame-level phoneme probability distribution generated by an acoustic model is directly and completely input into the pre-trained large language model, useful information is reserved as much as possible, and the large language model performs speech recognition correction according to global information of front and back frames. Compared with the prior art, the voice recognition correction is carried out according to the global information, the context understanding is more comprehensive, and more complex grammar errors, logic errors and fact errors can be detected in the error correction task, so that a more comprehensive, consistent and coherent result is provided.

Description

Technical Field

[0001] The present disclosure generally relates to the field of speech recognition technology, and in particular to a speech recognition error correction method, device, server, computer-readable storage medium, and computer program product. Background Art

[0002] Speech recognition is a technology that converts speech into text. Currently, speech recognition technology has been widely used in devices such as smartphones, smart speakers, and smart TVs. Speech recognition can generally be set to output multiple candidate results. The candidate results will be scored by the speech recognition system. The higher the score, the more likely the candidate is. Normally, most of the candidates with the highest scores are correct results, but there are still many correct results that are not the candidates with the highest scores. How to repair the results of speech recognition to get a better recognition rate is the speech recognition error correction task.

[0003] Common speech recognition error correction methods include language model re-scoring based methods, Attention-based Decoder re-scoring based methods, NBest modeling based speech recognition error correction methods, ChatLLM based speech recognition error correction methods, etc. However, due to the limitations of data volume and model modeling capabilities, these methods have poor robustness and are prone to overfitting. In actual applications, their performance is average and they cannot achieve accurate recognition in rich corpus environments. Summary of the invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, it is desired to provide a speech recognition error correction method, device, server, computer-readable storage medium and computer program product, which apply a large language model to the speech error correction task. Through the fine-tuning method, the robustness of the large language model is guaranteed and it is adapted to the speech recognition error correction task.

[0005] In a first aspect, an embodiment of the present application provides a speech recognition error correction method, comprising:

[0006] Receive user voice, extract voice features, and obtain voice features;

[0007] Inputting the speech features into an acoustic model to generate a phoneme probability distribution at a frame level;

[0008] Mapping the phoneme probability distribution to an input length of a large language model;

[0009] The mapped phoneme probability distribution is used as input data, the large language model is called to perform speech recognition correction according to the global information of the previous and next frames, and the output data is used as the recognition result of the user's speech.

[0010] In one embodiment, before mapping the phoneme probability distribution to the input length of the large language model, the method further includes:

[0011] Identifying redundant frames that do not contain useful information in the frame-level phoneme probability distribution;

[0012] The redundant frames are discarded.

[0013] In one embodiment, identifying redundant frames that do not contain useful information in the frame-level phoneme probability distribution includes:

[0014] identifying a blank frame in the frame-level phoneme probability distribution;

[0015] Accordingly, the redundant frame is discarded, specifically: the blank frame is discarded.

[0016] In one embodiment, identifying a blank frame in the frame-level phoneme probability distribution includes:

[0017] Determine whether the probability of a blank label in a CTC peak of each frame in the frame-level phoneme probability distribution exceeds a set threshold;

[0018] If exceeded, the frame exceeding the threshold is determined as a blank frame.

[0019] In one embodiment, before discarding the blank frame, the method further includes:

[0020] Determine whether the blank frame is a silent interval before and after a valid frame;

[0021] If yes, keep the blank frame;

[0022] If not, the step of discarding the blank frame is performed.

[0023] In one embodiment, it also includes:

[0024] Based on the input sequence expansion, calculating the loss function between the recognition result and the real speech label;

[0025] The large language model is optimized and fine-tuned according to the loss function.

[0026] In a second aspect, an embodiment of the present application provides a speech recognition error correction device, comprising:

[0027] A feature extraction unit, used to receive user speech, extract speech features, and obtain speech features;

[0028] A first model extraction unit, used for inputting the speech features into an acoustic model to generate a phoneme probability distribution at a frame level;

[0029] A data processing unit, configured to map the phoneme probability distribution to an input length of a large language model;

[0030] The second model extraction unit is used to use the mapped phoneme probability distribution as input data, call the large language model to perform speech recognition correction according to the global information of the previous and next frames, and use the output data as the recognition result of the user's speech.

[0031] In a third aspect, an embodiment of the present application provides a server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the embodiment of the present application when executing the program.

[0032] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the embodiment of the present application.

[0033] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in the embodiment of the present application.

[0034] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0036] Figure 1 A flow chart of a speech recognition error correction method provided by an embodiment of the present application is shown;

[0037] Figure 2 A schematic diagram of a large language model optimization process provided by an embodiment of the present application is shown;

[0038] Figure 3 An exemplary structural block diagram of a speech recognition error correction device provided in an embodiment of the present application is shown;

[0039] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing a server of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0040] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0041] It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and in combination with the embodiments. Although the embodiments of the present application provide the method operation instruction steps shown in the following embodiments or drawings, more or fewer operation instruction steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of the present application. The method may be executed in the order of the methods shown in the embodiments or drawings or in parallel during the actual processing process or when the device is executed.

[0042] Please refer to Figure 1 , Figure 1 FIG. 1 is a flow chart of a speech recognition error correction method provided by an embodiment of the present application. Figure 1 As shown, the method includes:

[0043] S101, receiving user voice, performing voice feature extraction, and obtaining voice features;

[0044] Receive the user voice to be recognized, and extract the continuous voice as voice features through some voice signal processing methods. The specific processing method is not limited in this embodiment, and you can refer to the relevant technology. It should be noted that this step needs to extract the features required by the speech recognition alignment model and the unsupervised pre-training model respectively. The features required by the two models may be the same or different, and there is no restriction here. Some restrictions need to be made on the duration of each voice frame, that is, the duration of the two features needs to be a multiple relationship, otherwise the voice frames cannot be aligned.

[0045] S102, inputting the speech features into the acoustic model to generate a phoneme probability distribution at the frame level;

[0046] The acoustic model in this method can use an encoder-decoder structure based on the attention mechanism, using an encoder decoder. In addition, other structures can also be used, which is not limited in this embodiment.

[0047] After the speech features are input into the acoustic model, the output of each frame of the acoustic model is the phoneme (the smallest speech unit, which can also be called a modeling unit in this method, such as a character, syllable or word) probability distribution at the frame level. This probability distribution represents the probability that the frame audio belongs to each phoneme.

[0048] S103, mapping the phoneme probability distribution to the input length of a large language model (LLM);

[0049] The obtained phoneme probability distribution is an N-frame D-dimensional vector. Before being input into the large language model, an adaptation layer (adapter) structure is used to map the dimensions of the phoneme probability distribution and the large language model to the same length.

[0050] S104: using the mapped phoneme probability distribution as input data, calling the large language model to perform speech recognition correction according to the global information of the previous and next frames, and using the output data as the recognition result of the user's speech.

[0051] This method applies a large language model to speech recognition error correction tasks. Compared with other speech recognition error correction methods, this method does not need to obtain multiple candidates through search and then perform error correction. Instead, the probability distribution output by the acoustic model is directly input into the large language model, retaining as much useful information as possible. In addition, the pre-trained large language model is called to perform speech recognition correction based on the global information of the previous and next frames. Compared with the traditional language model that only sees past information but not future information, the error correction task of the large language model in this method can see the full text information including the previous and next frames. Therefore, when the large language model calculates the forward direction, there is no need to hide the subsequent information. Speech recognition correction is performed based on the global information, which is more comprehensive for context understanding. The large language model can detect more complex grammatical errors, logical errors and factual errors in the error correction task, and can provide more comprehensive, consistent and coherent results.

[0052] The output data of the large language model is obtained, and the best result is searched as the recognition result of the user's speech.

[0053] It should be noted that although the operations of the method of the present invention are described in a particular order in the drawings, this does not require or imply that the operations must be performed in this particular order or that all illustrated operations must be performed to achieve desired results.

[0054] Based on the above introduction, the method provided in this embodiment applies a large language model to the speech recognition error correction task, and directly inputs all the frame-level phoneme probability distribution generated by the acoustic model into the pre-trained large language model, retaining as much useful information as possible. The large language model performs speech recognition correction based on the global information of the previous and next frames. It performs speech recognition correction based on the global information, has a more comprehensive understanding of the context, and can detect more complex grammatical errors, logical errors, and factual errors in error correction tasks, thereby providing more comprehensive, consistent and coherent results.

[0055] In one embodiment, before executing step S103 to map the phoneme probability distribution to the input length of the large language model, the following steps may be further performed: identifying redundant frames that do not contain useful information in the frame-level phoneme probability distribution; and discarding redundant frames.

[0056] The length of a speech frame is much longer than the length of a text, so when the output of the acoustic model is used as the input of a large language model, a huge amount of computation will be generated. In order to reduce the computation of the large language model, improve the efficiency of speech recognition error correction, and reduce resource usage, redundant frames can be identified for the phoneme probability distribution generated by the acoustic model. If a redundant frame is identified, the information is removed. Redundant frames include but are not limited to: repeated frames, static frames, blank frames, noise frames, and transition frames.

[0057] In order to avoid valid information being accidentally deleted, optionally, the redundant frame that does not contain useful information in the phoneme probability distribution at the recognition frame level can specifically be: a blank frame (blank, meaningless frame) in the phoneme probability distribution at the recognition frame level, and the blank frame information is correspondingly discarded.

[0058] At present, most of the speech frames are output as blank frames. For example, a speech label is: The weather is very good today. The speech frames are f1 f2 f3 f4 f5 f6 f7....f22. The corresponding situations may be as follows: f1: blank f2: blank f3: today f4: blank f5: blank f6: blank f7: blank f8: weather f9: blank f10: blank f11: weather f12: blank f13: blank f14: weather f15: blank f16: blank f17: very f18: blank f19: blank f20: good f21: blank f22: blank. It can be seen that the blank frames account for a huge proportion. The data of the blank frames in the acoustic model output is discarded and only the data of the non-blank frames is retained. The meaningless data is discarded and only the meaningful data is retained. It can save more than 90% of the calculation amount and reduce data transmission.

[0059] In this embodiment, the specific implementation method for identifying blank frames is not limited. Specifically, the identification of blank frames in the phoneme probability distribution at the frame level can be performed according to the following steps: determine whether the probability of blank labels in the CTC (Connectionist temporal classification) peak of each frame in the phoneme probability distribution at the frame level exceeds a set threshold; if exceeded, the frames exceeding the threshold are determined as blank frames.

[0060] The peaks generated by CTC are used for recognition to reduce the amount of calculation. If the probability of blank (blank frame, meaningless frame) in the output of the acoustic model is greater than the set threshold (such as 0.99), the frame is considered meaningless and discarded directly, and only meaningful frames are retained as the input of the large language model. Using the characteristics of CTC to trim the output of the acoustic model is simple and easy to operate, and has high recognition accuracy. It should be noted that when blank frame recognition is performed based on CTC peaks, the acoustic model needs to use the CTC error function on the encoder side. Specifically, the acoustic model can use the Encoder-CTC structure, that is, the Encoder decoder is used as the acoustic model, and the CTC loss is used as the error criterion.

[0061] Furthermore, before discarding the blank frame, the following steps may be performed first: determine whether the blank frame is a silent interval before or after a valid frame; if so, retain the blank frame; if not, perform the step of discarding the blank frame.

[0062] In a recording or audio stream, silence intervals refer to those parts where there is no actual speaking content. When these parts appear in the frames before and after the valid frames, they may be used for pauses between sentences or words in actual speech. Identifying and retaining these silence intervals is more in line with the input habits of the language model and helps to improve the accuracy of speech recognition.

[0063] In one embodiment, in the application of the large language model, its model parameters can be further optimized and fine-tuned, which ensures the robustness of the large language model and enables the large language model to adapt to the speech recognition error correction task.

[0064] Specifically, in addition to the above steps, the following steps may be performed:

[0065] Step S105, based on the input sequence expansion, calculating the loss function between the recognition result and the real speech label;

[0066] Step S106: Optimize and fine-tune the large language model according to the loss function.

[0067] like Figure 2 The figure shows a schematic diagram of the optimization process of a large language model. During training, the output of the large language model and the true speech label are used to calculate the error, and then the parameters are updated in reverse.

[0068] Since the input length and output length may be inconsistent, for example, there may be insertion and deletion errors, which may cause the length of the recognition model output to be inconsistent with the actual label length. When calculating the error function of the large language model, this method does not use alignment or other methods to align the input and output to calculate the error function. Instead, it calculates the loss function between the recognition result and the actual speech label based on the input sequence expansion. It uses methods such as CTC loss or a simpler method of lengthening the input to solve the unequal length problem, which is both simple and effective.

[0069] Based on the input sequence expansion, the specific method of calculating the loss function between the recognition result and the real speech label is not limited. Optionally, loss functions such as CTC loss and cross entropy function can be used. Among them, CTC loss can solve the problem of unequal length of input and output, but it requires that the input length is much larger than the output length. Targetedly, some blank frames can be retained when discarding blank frames to facilitate the calculation of CTC loss. Of course, other length expansion methods can also be adopted. CTC effectively solves the problem of mismatch between input and output sequence lengths by introducing blank labels and flexible path search mechanisms. It allows the length of the input sequence to be arbitrary, and achieves end-to-end sequence-to-sequence learning by expanding the output sequence and using a dynamic programming algorithm to find the optimal alignment method.

[0070] It should be noted that in this embodiment, only the parameters of the large language model need to be updated, and there is no need to update the parameters of the acoustic model.

[0071] Further references Figure 3 , which shows an exemplary structural block diagram of a speech recognition error correction device according to an embodiment of the present application, which mainly includes:

[0072] The feature extraction unit 101 is used to receive user speech, extract speech features, and obtain speech features;

[0073] A first model extraction unit 102, used to input speech features into an acoustic model to generate a phoneme probability distribution at a frame level;

[0074] A data processing unit 103, used to map the phoneme probability distribution to the input length of the large language model;

[0075] The second model extraction unit 104 is used to use the mapped phoneme probability distribution as input data, call the large language model to perform speech recognition correction according to the global information of the previous and next frames, and use the output data as the recognition result of the user's speech.

[0076] It should be understood that the units described in the above device are similar to those in the reference Figure 1The steps in the method described above correspond to each other. Therefore, the operations and features described above for the method are also applicable to the device and the units contained therein, and will not be repeated here. The device can be pre-implemented in the browser or other security application of the server, or loaded into the browser or its security application of the server by downloading or the like. The corresponding units in the device can cooperate with the units in the server to implement the solution of the embodiment of the present application.

[0077] For the several units mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units described above can be embodied in one unit. On the contrary, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0078] It should be noted that for details not disclosed in the speech recognition error correction device of the embodiment of the present application, please refer to the details disclosed in the above embodiments of the present application, which will not be repeated here.

[0079] Reference below Figure 4 , Figure 4 A schematic diagram of the structure of a computer system suitable for implementing a server of an embodiment of the present application is shown.

[0080] like Figure 4 As shown, the computer system includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 into the random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation instructions of the system are also stored. The CPU 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0081] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.

[0082] In particular, according to an embodiment of the present application, the above reference flow chart Figure 1The described process can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the above-mentioned functions defined in the system of the present application are executed.

[0083] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium such as a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0084] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operating instructions of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the aforementioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operating instruction, or can be implemented with a combination of dedicated hardware and computer instructions.

[0085] The units or modules involved in the embodiments described in the present application may be implemented by software or hardware. The units or modules described may also be arranged in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0086] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the server described in the above embodiment, or may exist independently without being assembled into the server. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the speech recognition error correction method described in the present application.

[0087] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A speech recognition error correction method, characterized in that: include: Receive user voice, extract voice features, and obtain voice features; Inputting the speech features into an acoustic model to generate a phoneme probability distribution at a frame level; Mapping the phoneme probability distribution to an input length of a large language model; The mapped phoneme probability distribution is used as input data, the large language model is called to perform speech recognition correction according to the global information of the previous and next frames, and the output data is used as the recognition result of the user's speech.

2. The method according to claim 1, characterized in that Before mapping the phoneme probability distribution to the input length of the large language model, the method further includes: Identifying redundant frames that do not contain useful information in the frame-level phoneme probability distribution; The redundant frames are discarded.

3. The method according to claim 2, characterized in that Identifying redundant frames that do not contain useful information in the frame-level phoneme probability distribution, including: identifying a blank frame in the frame-level phoneme probability distribution; Accordingly, the redundant frame is discarded, specifically: the blank frame is discarded.

4. The method according to claim 3, characterized in that Identifying a blank frame in the frame-level phoneme probability distribution, comprising: Determine whether the probability of a blank label in a CTC peak of each frame in the frame-level phoneme probability distribution exceeds a set threshold; If exceeded, the frame exceeding the threshold is determined as a blank frame.

5. The method according to claim 3, characterized in that Before discarding the blank frame, the method further includes: Determine whether the blank frame is a silent interval before and after a valid frame; If yes, keep the blank frame; If not, the step of discarding the blank frame is performed.

6. The method according to any one of claims 1 to 5, characterized in that: Also includes: Based on the input sequence expansion, calculating the loss function between the recognition result and the real speech label; The large language model is optimized and fine-tuned according to the loss function.

7. A speech recognition error correction device, characterized in that: include: A feature extraction unit, used to receive user speech, extract speech features, and obtain speech features; A first model extraction unit, used for inputting the speech features into an acoustic model to generate a phoneme probability distribution at a frame level; A data processing unit, configured to map the phoneme probability distribution to an input length of a large language model; The second model extraction unit is used to use the mapped phoneme probability distribution as input data, call the large language model to perform speech recognition correction according to the global information of the previous and next frames, and use the output data as the recognition result of the user's speech.

8. A server comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images