Speech recognition method and related equipment

Abstract

The invention discloses a voice recognition method and device, a server, a computer readable storage medium and a computer program product. According to the method, modeling units of a large language model are unified into an acoustic model, so that the problem of word modeling sparsity in speech training data can be greatly relieved; besides, according to the method, the output of the acoustic model is directly input into the large language model, so that the process of mapping by using an adapter is reduced, the fusion of the acoustic model and the language model is more direct, and the speech recognition performance can be greatly improved. By using the scheme of applying the large language model to the acoustic model provided by the method, the method is more direct and efficient, and meanwhile, the speech recognition performance can be greatly improved.

Description

Technical Field

[0001] The present disclosure generally relates to the field of speech recognition technology, and in particular to a speech recognition method, device, server, computer-readable storage medium, and computer program product. Background Art

[0002] Speech recognition refers to the conversion of audio signals into corresponding text, thereby achieving automated processing and interaction. It is widely used in business scenarios such as intelligent customer service, robot credit review, and robot debt collection. End-to-end speech recognition systems usually use only one model to model speech recognition tasks, that is, input speech and output text. However, end-to-end speech recognition systems have the problem of weak semantic content, that is, the model tends to output words that are the same as the pronunciation of the speech and ignores semantic information. The solution to this problem is generally to add a language model to strengthen the semantic information. Generally speaking, the language model can be added when the acoustic model is decoded (lm decoding) or after the acoustic model is decoded (lm rescoring).

[0003] The powerful capabilities of large language models have been proven in various tasks. How to add large language models to speech recognition systems to improve speech recognition performance is currently a hot research project.

[0004] The current mainstream application solution for large language models is to use the output of the acoustic model as the input of the large language model, input the speech frame as prompt information into the large language model, and then the large language model autoregressively outputs the recognition result.

[0005] However, since the speech frame is used as the prompt information input, the speech frame and the large language model information do not match, so an adaptation layer (adapter) is needed to convert the speech information into text information. At the same time, the modeling unit of the large language model is different from the recognition model, which causes the two to be mismatched, which will affect the performance when combined. In addition, when the speech frame is used as the prompt information, the input of the large language model is very long, resulting in a very large amount of calculation. Summary of the invention

[0006] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a speech recognition method, apparatus, server, computer-readable storage medium, and computer program product that can achieve direct adaptation of the acoustic model to the large language model.

[0007] In a first aspect, an embodiment of the present application provides a speech recognition method, comprising:

[0008] Receive user voice, extract voice features, and obtain voice features;

[0009] Inputting the speech features into an acoustic model to generate a modeling unit and a probability thereof for each frame;

[0010] Select the top N modeling units with the highest probability, and perform weighted processing according to the corresponding probabilities to obtain the weighted result of each frame;

[0011] Inputting the weighted results of each frame into a large language model for text conversion to generate recognized text;

[0012] The acoustic model and the large language model application characters are used as modeling units.

[0013] In one embodiment, before inputting the weighted results of each frame into a large language model for text conversion, the method further includes: identifying and discarding redundant frames.

[0014] In one embodiment, the identifying and discarding redundant frames includes: identifying and discarding blank frames.

[0015] In one embodiment, identifying and discarding blank frames includes:

[0016] Determine whether the probability of blank labels in the CTC peak of each frame exceeds the set threshold;

[0017] If it exceeds, it is considered as a blank frame and discarded.

[0018] In one embodiment, it also includes:

[0019] Obtaining the actual text corresponding to the user's voice;

[0020] The weighted result of each frame and the actual text are input into the large language model, and the parameters of the large language model are updated according to the loss between the actual text and the recognized text.

[0021] In one embodiment, before receiving the user voice, the method further includes: independently training the acoustic model and the large language model.

[0022] In a second aspect, an embodiment of the present application provides a speech recognition device, including:

[0023] A feature extraction unit, used to receive user speech, extract speech features, and obtain speech features;

[0024] A first model extraction unit, used for inputting the speech features into an acoustic model to generate a modeling unit and a probability thereof for each frame;

[0025] A data processing unit is used to select the top N modeling units with the highest probability, and perform weighted processing according to the corresponding probabilities to obtain a weighted result for each frame;

[0026] The second model extraction unit is used to input the weighted results of each frame into the large language model for text conversion to generate recognized text; wherein the acoustic model and the large language model application characters are modeling units.

[0027] In a third aspect, an embodiment of the present application provides a server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the embodiment of the present application when executing the program.

[0028] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the embodiment of the present application.

[0029] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in the embodiment of the present application.

[0030] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0032] Figure 1 A flow chart of a speech recognition method provided by an embodiment of the present application is shown;

[0033] Figure 2 A schematic diagram of the processing results of the top5 multiple frames provided in an embodiment of the present application is shown;

[0034] Figure 3 A schematic diagram of a modeling unit of a large language model provided by an embodiment of the present application is shown;

[0035] Figure 4 A schematic diagram of multi-frame output provided by an embodiment of the present application is shown;

[0036] Figure 5 A schematic diagram of a large language model optimization process provided by an embodiment of the present application is shown;

[0037] Figure 6 An exemplary structural block diagram of a speech recognition device provided in an embodiment of the present application is shown;

[0038] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing a server of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0039] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0040] It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and in combination with the embodiments. Although the embodiments of the present application provide the method operation instruction steps shown in the following embodiments or drawings, more or fewer operation instruction steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of the present application. The method may be executed in the order of the methods shown in the embodiments or drawings or in parallel during the actual processing process or when the device is executed.

[0041] Please refer to Figure 1 , Figure 1 FIG. 1 is a flow chart of a speech recognition method provided by an embodiment of the present application. Figure 1 As shown, the method includes:

[0042] S101, receiving user voice, performing voice feature extraction, and obtaining voice features;

[0043] Receive the user voice to be recognized, and extract the continuous voice as voice features through some voice signal processing methods. The specific processing method is not limited in this embodiment, and reference can be made to related technologies. Generally speaking, the voice with a duration of t1 can be extracted into T M-dimensional feature vectors, which are called T-frame voice vectors. Voice features are usually Fbank (Filter Bank) and MFCC (Mel-Frequency Cepstral Coefficients) features. 1 second of voice can usually be converted into a 100*80 (T*M) matrix, where the specific values ​​of T and M are not limited.

[0044] S102, inputting the speech features into the acoustic model to generate a modeling unit and its probability for each frame;

[0045] The acoustic model in this method can use an encoder-decoder structure based on the attention mechanism. In addition, other structures can also be used, which is not limited in this embodiment.

[0046] After inputting the speech features into the acoustic model, the output of each frame of the acoustic model is the probability distribution of the modeling units, including the modeling units (words) and their probabilities. This probability distribution represents the probability that the frame of audio belongs to each modeling unit. Among them, the modeling unit is the smallest unit used to represent and process basic language units.

[0047] S103. Select the top N modeling units with the highest probabilities and perform weighted processing according to the corresponding probabilities to obtain the weighted result of each frame;

[0048] After the acoustic model processes each frame of audio and outputs the probability distribution of the modeling units, select the top N modeling units with the highest probabilities. N is less than or equal to the number of modeling units and is a positive integer. The specific value is not limited in this embodiment. Perform weighted processing on these unit probabilities and use these weighted results as the input to the large language model.

[0049] Specifically, each Chinese character can be mapped to its corresponding embedding vector, and then these embedding vectors are combined according to the weighted probabilities.

[0050] For example, the output of the first frame may be: {"今": 0.90, "仅": 0.03, "近": 0.02, "禁": 0.02, "令": 0.02, "斤": 0.01}

[0051] Taking N as 5 as an example, select the top 5 words with the highest probabilities and their probabilities: {"今": 0.90, "仅": 0.03, "近": 0.02, "禁": 0.02, "令": 0.02}

[0052] Perform weighted processing on these probability values. Usually, the probability values can be directly used as weights, that is, [0.90, 0.03, 0.02, 0.02, 0.02], and use this weighted vector as the input to the large language model. Specifically, each Chinese character can be mapped to its corresponding embedding vector, and then these embedding vectors are combined according to the weighted probabilities.

[0053] Assume that the embedding vectors of "今", "仅", "近", "禁", "令" are E_今, E_仅, E_近, E_禁, E_令 respectively. Then the weighted input vector can be expressed as: Input = 0.9 * E_今 + 0.03 * E_仅 + 0.02 * E_近 + 0.02 * E_禁 + 0.02 * E_令.

[0054] The above is introduced by taking the processing process of a single frame as an example. As Figure 2 shown is a schematic diagram of the processing result of top5 multi-frames.

[0055] After the above processing, the acoustic model and the large language model are directly connected, and there is no need for an adapter structure to map the acoustic model to the large language model.

[0056] S104: Input the weighted results of each frame into the large language model for text conversion to generate recognized text.

[0057] At present, the modeling unit of the acoustic model is generally Chinese characters, while the modeling unit of the large language model is usually words, so there will be mismatches. The common application of large language models to speech recognition tasks will unify the modeling units into words of the large language model, which will ensure the performance of the large language model. However, this will cause the problem of sparse speech training data, because many words appear less frequently in speech data, resulting in poor recognition performance.

[0058] In this method, the modeling unit of the large language model and the acoustic model are unified into a character sequence, because the modeling unit of the large language model generally includes the modeling unit of the acoustic model. Although the modeling unit of the large language model is a word, in order to avoid exceeding the scope of the dictionary, the characters are included in the modeling unit, such as Figure 3 The figure shows a schematic diagram of a modeling unit of a large language model. In this method, only the character part of the modeling unit of the large language model is used, which ensures that the modeling units of the language model and the acoustic model are consistent and the performance of the large language model is guaranteed.

[0059] After the weighted results of each frame are input into the large language model for text conversion, the large language model will generate recognition text. The recognition text data can be used for subsequent tasks as the result of user speech recognition.

[0060] It should be noted that although the operations of the method of the present invention are described in a particular order in the drawings, this does not require or imply that the operations must be performed in this particular order or that all illustrated operations must be performed to achieve desired results.

[0061] Based on the above introduction, the method provided in this embodiment proposes to unify the modeling units of the large language model into the acoustic model, which can greatly alleviate the problem of sparse word modeling in speech training data; in addition, this method directly inputs the output of the acoustic model into the large language model, which reduces the process of using adapter mapping, making the fusion of the acoustic model and the language model more direct, and can greatly improve the speech recognition performance. The solution provided by this method for applying the large language model to the acoustic model can be more direct and efficient, and can also greatly improve the performance of speech recognition.

[0062] In one embodiment, before executing step S104 to input the weighted results of each frame into the large language model for text conversion, the following steps may be further performed: identifying and discarding redundant frames.

[0063] The length of a speech frame is much longer than the length of a text, so when the acoustic model is used as the input of a large language model, a huge amount of calculation will be generated. In order to reduce the amount of calculation of the large language model, improve speech recognition efficiency, and reduce resource usage, redundant frames can be identified for each frame generated by the acoustic model. If they are identified as redundant frames, they will be removed. Redundant frames include but are not limited to: repeated frames, static frames, blank frames, noise frames, and transition frames.

[0064] It should be noted that the identification and elimination of redundant frames need to be completed before being input into the large language model. This can be done after the acoustic model generates the modeling units and their probabilities for each frame in step S102, and before step S103 selects the top N modeling units with the highest probabilities and performs weighted processing according to the corresponding probabilities. This can also reduce the workload of weighting the modeling units and the corresponding probabilities.

[0065] In order to avoid valid information from being mistakenly deleted, optionally, identifying and discarding redundant frames may specifically include: identifying and discarding blank frames (blank, meaningless frames).

[0066] like Figure 4 The figure shows a schematic diagram of multi-frame output, where each rectangle represents the output of each frame, and the rectangle with a white background is a blank frame. Only the topN data of the black frame is input into the large language model. Only meaningful frames are retained as input to the large language model, which can save more than 90% of the calculation amount. At the same time, only part of the data of the meaningful frame needs to be input into the large language model during calculation, which is also very beneficial for the separate deployment of the acoustic model and the large language model, and can greatly reduce the amount of transmitted data.

[0067] In this embodiment, the specific implementation method for identifying blank frames is not limited. Specifically, identifying and discarding blank frames can be performed according to the following steps: determining whether the probability of blank labels in the CTC (Connectionist temporal classification) peak of each frame exceeds a set threshold; if exceeded, determining it as a blank frame and discarding it.

[0068] The peaks generated by CTC are used for identification to reduce the amount of calculation. If the probability of blank (blank frame, meaningless frame) in the output of the acoustic model is greater than the set threshold (such as 0.99), the frame is considered meaningless and discarded directly, and only meaningful frames are retained as the input of the large language model. Using the characteristics of CTC to trim the output of the acoustic model is simple and easy to operate, and has high recognition accuracy. It should be noted that when blank frame recognition is performed based on CTC peaks, the acoustic model needs to use the CTC error function on the encoder side.

[0069] In one embodiment, in the actual application of the large language model, its model parameters can be further optimized. Specifically, in addition to the above steps, the following steps can be performed:

[0070] Step S105: Obtain the actual text corresponding to the user's voice;

[0071] Step S106: input the weighted results of each frame and the actual text into the large language model, and update the parameters of the large language model according to the loss between the actual text and the recognized text.

[0072] like Figure 5 The figure shows a schematic diagram of the optimization process of a large language model. In this embodiment, a trained large language model is used as initialization. The input of the large language model consists of two parts. The first part is the weighted result after the output of the acoustic model, and the second part is the text corresponding to the speech as input. The acoustic input (the output of the acoustic model) has the context of the acoustic input. In addition to the context of the text, the text input also has all the information of the acoustic input, which can help the large language model improve its understanding ability and enhance coherence. The output of the large language model is the probability distribution of the word-level modeling unit. Various fine-tuning algorithms can be used to update the large language model, such as Figure 5 The figure shows that the parameters are updated by CE Loss (Cross-Entropy Loss), but it is not limited thereto. It should be noted that in this embodiment, only the parameters of the large language model need to be updated, and the parameters of the acoustic model do not need to be updated.

[0073] The acoustic model and the large language model called in the above embodiments are both pre-trained models. The above embodiments do not limit the training process of the two models. In one embodiment, before receiving the user voice in step S101, it also includes: independently training the acoustic model and the large language model.

[0074] The required acoustic model and large language model are trained separately. The training processes of the two are completely independent and have no connection with each other. They can achieve better generalization in their respective fields and improve the flexibility of optimization and updating.

[0075] Further references Figure 6 , which shows an exemplary structural block diagram of a speech recognition device according to an embodiment of the present application, which mainly includes:

[0076] The feature extraction unit 101 is used to receive user speech, extract speech features, and obtain speech features;

[0077] A first model extraction unit 102, used to input speech features into an acoustic model to generate a modeling unit and its probability for each frame;

[0078] The data processing unit 103 is used to select the first N modeling units with the highest probability, and perform weighted processing according to the corresponding probabilities to obtain a weighted result for each frame;

[0079] The second model extraction unit 104 is used to input the weighted results of each frame into the large language model for text conversion to generate recognized text; wherein the acoustic model and the large language model application characters are modeling units.

[0080] It should be understood that the units described in the above device are similar to those in the reference Figure 1 The steps in the method described above correspond to each other. Therefore, the operations and features described above for the method are also applicable to the device and the units contained therein, and will not be repeated here. The device can be pre-implemented in the browser or other security application of the server, or loaded into the browser or its security application of the server by downloading or the like. The corresponding units in the device can cooperate with the units in the server to implement the solution of the embodiment of the present application.

[0081] For the several units mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units described above can be embodied in one unit. On the contrary, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0082] It should be noted that for details not disclosed in the speech recognition device in the embodiment of the present application, please refer to the details disclosed in the above embodiment of the present application, which will not be repeated here.

[0083] Reference below Figure 7 , Figure 7 A schematic diagram of the structure of a computer system suitable for implementing a server of an embodiment of the present application is shown.

[0084] like Figure 7 As shown, the computer system includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation instructions of the system are also stored. The CPU 701, the ROM 702 and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0085] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0086] In particular, according to an embodiment of the present application, the above reference flow chart Figure 1 The described process can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present application are executed.

[0087] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium such as a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0088] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operating instructions of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the aforementioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operating instruction, or can be implemented with a combination of dedicated hardware and computer instructions.

[0089] The units or modules involved in the embodiments described in the present application may be implemented by software or hardware. The units or modules described may also be arranged in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0090] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the server described in the above embodiment, or may exist independently without being assembled into the server. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the speech recognition method described in the present application.

[0091] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A speech recognition method, characterized in that: include: Receive user voice, extract voice features, and obtain voice features; Inputting the speech features into an acoustic model to generate a modeling unit and a probability thereof for each frame; Select the top N modeling units with the highest probability, and perform weighted processing according to the corresponding probabilities to obtain the weighted result of each frame; Inputting the weighted results of each frame into a large language model for text conversion to generate recognized text; The acoustic model and the large language model application characters are used as modeling units.

2. The method according to claim 1, characterized in that Before inputting the weighted results of each frame into the large language model for text conversion, the method further includes: identifying and discarding redundant frames.

3. The method according to claim 2, characterized in that The identifying and discarding redundant frames includes: identifying and discarding blank frames.

4. The method according to claim 3, characterized in that The identifying and discarding of blank frames comprises: Determine whether the probability of blank labels in the CTC peak of each frame exceeds the set threshold; If it exceeds, it is considered as a blank frame and discarded.

5. The method according to claim 1, characterized in that Also includes: Obtaining the actual text corresponding to the user's voice; The weighted result of each frame and the actual text are input into the large language model, and the parameters of the large language model are updated according to the loss between the actual text and the recognized text.

6. The method according to claim 1, characterized in that Before receiving the user voice, the method further includes: independently training the acoustic model and the large language model.

7. A speech recognition device, characterized in that: include: A feature extraction unit, used to receive user speech, extract speech features, and obtain speech features; A first model extraction unit, used for inputting the speech features into an acoustic model to generate a modeling unit and a probability thereof for each frame; A data processing unit is used to select the top N modeling units with the highest probability, and perform weighted processing according to the corresponding probabilities to obtain a weighted result for each frame; The second model extraction unit is used to input the weighted results of each frame into the large language model for text conversion to generate recognized text; wherein the acoustic model and the large language model application characters are modeling units.

8. A server comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images