A multimodal speech recognition error correction method and system

Abstract

The present invention discloses a multimodal speech recognition error correction method and system, comprising: obtaining original sample data from a corpus, generating error samples from the original samples using a fuzzy sound generator; annotating the initials and finals of the error sample text to construct fuzzy sound texts with different similarity levels; adjusting the parameters of the different similarity ratios of the fuzzy sound texts of the error sample data based on the annotated error sample data; constructing a speech and text fusion feature vector based on the original correct sample and error sample data, inputting the fusion feature vector into an error correction model for training, and outputting the word with the highest correct probability for each speech position through a fully connected layer and an activation function. The method and system utilize the multimodal fusion features of speech and text for training to obtain a model for customer service speech error correction. The error correction model based on the combination of text and speech can reduce the impact of dialects and environmental noise, thereby improving the accuracy of customer service speech quality inspection.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a multimodal speech recognition error correction method and system. Background Art

[0002] Today, customer service primarily relies on hotlines. Customer service voice quality inspection is a crucial quality control step in hotline operations. Traditional customer service quality inspection is conducted manually through monitoring customer service call recordings, which is time-consuming and labor-intensive, and relies solely on random sampling. The emergence of intelligent quality inspection has revolutionized the current state of quality inspection in traditional customer service centers. Full-scale quality inspection, simultaneous on-screen quality inspection, and online customer notifications mean that data analysis reports are instantly accessible. However, customer service voice is plagued by issues such as ambient noise and dialects, and automatic speech recognition text errors are inevitable. Without text error correction, quality inspection accuracy cannot be guaranteed. Therefore, text error correction technology is fundamental to the reliability of intelligent quality inspection results. The main challenges currently faced in voice error correction for quality inspection are: 1. Most error correction solutions only correct the text data after speech recognition, which loses the original speech data information; 2. Currently, there are few annotated error samples for speech recognition in customer service scenarios, making it difficult to achieve effective training results. Summary of the Invention

[0003] One of the inventive objectives of the present invention is to provide a multimodal speech recognition error correction method and system. The method and system use the multimodal fusion features of speech and text for training to obtain a model for customer service speech error correction. The error correction model based on the combination of text and speech can reduce the influence of dialects and environmental noise, thereby improving the accuracy of customer service speech quality inspection.

[0004] Another object of the present invention is to provide a multimodal speech recognition error correction method and system, which utilizes Chinese fuzzy sound matching technology to perform data enhancement on multimodal fusion features. Through the fuzzy sound data enhancement, the training data of the model is expanded, thereby solving the problem of insufficient data in existing model training and the problem of speech data being discarded in the text error correction model.

[0005] Another object of the present invention is to provide a multimodal speech recognition error correction method and system, which generates enhanced data of fuzzy sounds by analyzing relevant data such as flat and retroflex consonants, front and back nasal sounds, and pronunciation tones in a language. The enhanced data is configured with a similarity level based on the fuzzy sounds, and the proportion parameter is adjusted according to the fuzzy sound similarity level based on the overall situation of the annotated data in reality, so that the training enhanced data samples are closer to the actual situation.

[0006] In order to achieve at least one of the above-mentioned objects, the present invention further provides a multimodal speech recognition error correction method, the method comprising:

[0007] Obtaining original sample data from a corpus, and generating error samples from the original samples using a fuzzy sound generator;

[0008] The error sample data is labeled according to the initial consonants and finals of the characters, and fuzzy phonetic characters with different similarity levels are constructed;

[0009] According to the marked error sample data, adjust the parameters of the different similarity level ratios of the fuzzy phonetic characters of the error sample data;

[0010] A speech and text fusion feature vector is constructed based on the original correct sample and error sample data, and the fusion feature vector is input into the error correction model for training. The word with the highest correct probability for each speech position is output through the fully connected layer and activation function.

[0011] According to one preferred embodiment of the present invention, the error sample marking method includes: marking the flat and retroflex consonants, tones, and front and back nasals of each character in the error sample respectively to obtain marked sample data.

[0012] According to another preferred embodiment of the present invention, the method for generating error samples includes: processing the original sample based on the fuzzy pinyin of the fuzzy sound generator to obtain error sample data with different similar pinyin characters.

[0013] According to another preferred embodiment of the present invention, a method for constructing similarity levels of fuzzy sound characters based on error samples includes configuring similarity levels of similar characters of at least one existing flat and retroflex consonant, tone, and front and back nasal sounds according to the number of similarity types, wherein the more similarity types there are, the higher the similarity level.

[0014] According to another preferred embodiment of the present invention, the error sample generation method includes: adjusting the proportion of different levels of fuzzy sounds according to the overall situation of fuzzy sounds in the error sample, so that the error sample is closer to the real data enhancement sample.

[0015] According to another preferred embodiment of the present invention, the error correction model construction method includes: constructing respective speech vectors and text vectors for the error sample and the original correct sample, wherein the speech vector includes the fuzzy speech vector of the error sample, and the text vector includes the Chinese character vector obtained from the error sample, to generate the speech and text fusion feature vector.

[0016] According to another preferred embodiment of the present invention, the method for constructing a speech and text fusion feature vector includes: using a bidirectional LSTM model to model two sample speech and text respectively, constructing a token sequence after inputting the sample speech and text, and performing an attention mechanism to screen the vector of each token position to obtain a bimodal fusion vector with the same length of Chinese characters and speech.

[0017] According to another preferred embodiment of the present invention, the error correction model includes a pre-trained Albert model. In the embedding layer of the pre-trained Albert model, the word embedding of the corresponding position of the sample original text includes a weighted sum of the word embedding of the word at the corresponding position of the sample original text and the word embedding of the mask of the position, wherein the weight of the mask is the probability of typos at the corresponding position.

[0018] In order to achieve at least one of the above-mentioned objects of the invention, the present invention further provides a multimodal speech recognition error correction system, which executes the above-mentioned multimodal speech recognition error correction method.

[0019] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the above-mentioned multimodal speech recognition error correction method. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Shown is a flow chart of a multimodal speech recognition error correction method of the present invention.

[0021] Figure 2 Shown is a schematic diagram of a fuzzy sound matching method in a multimodal speech recognition and error correction system of the present invention.

[0022] Figure 3 Shown is a schematic diagram of constructing a fusion vector in a multimodal speech recognition and error correction system of the present invention.

[0023] Figure 4 Shown is a schematic diagram of error correction model training in a multimodal speech recognition error correction system of the present invention. DETAILED DESCRIPTION

[0024] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are for illustrative purposes only, and those skilled in the art will readily appreciate other obvious variations. The basic principles of the present invention defined in the following description may be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0025] It is to be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the elements may be multiple, and the term "one" should not be understood as a limitation on the quantity.

[0026] Please combine Figure 1-Figure 4The present invention provides a multimodal speech recognition error correction method and system, the method comprising: obtaining original customer service conversation data from a corpus, wherein the corpus includes but is not limited to public data sets and historical customer service conversation data, the customer service conversation data extracted from the corpus is used as the original sample, and data expansion is performed on the original sample. In the present invention, a fuzzy sound generator is used to expand the customer service conversation data of the original sample to obtain error sample data, wherein the error sample data is obtained based on fuzzy pinyin. The correct original customer service conversation data and the error sample data are used as constructed complete sample data for model training. In the present invention, different types of fuzzy sound annotations are performed on each word in the sample to obtain different types of fuzzy sound audio and text, and the similarity level is further calculated based on the fuzzy sound similarity type of the selected word, and the fuzzy sound parameters are adjusted according to the similarity level to achieve the expanded error sample to be more consistent with the real enhanced data. The speech and text of the error sample and the correct sample are respectively constructed as feature vectors, and a multimodal fusion feature vector is established. The multimodal fusion feature vector is input into the error correction model, and the probability of the typo at each position is output. The word with the highest probability of being correct is output as the output result of the current voice dialogue.

[0027] Specifically, the present invention uses a fuzzy sound generator to perform data enhancement corresponding to fuzzy sound matching after obtaining the original sample customer service conversation data. Please refer to Figure 2 The original dialogue input includes: Who are you? Report loss, try restarting, and the input to the fuzzy sound generator includes "Can you try restarting?" The fuzzy sound parameters are adjusted according to the required similarity level, and the text or words to be fuzzified are selected through the mask mechanism, such as Figure 2 Using the word "try" in the input sentence as a mask, the fuzzy sound generator generates a set of fuzzy sound candidate characters based on similarity levels, including (implementation, fact, real-time, and epic). The fuzzy sound character closest to the original text in the fuzzy sound candidate set is selected and embedded into the masked position of the original text to form an error sample. Using the fuzzy sound generator's fuzzy sound candidate set for masking and matching replacement effectively expands the corpus, greatly enhancing the effectiveness of subsequent model training.

[0028] It is worth mentioning that in the present invention, error samples need to be marked, and the hyperparameters of the fuzzy sound generator are adjusted through the way of marking feedback, so that the text of the fuzzy sound candidate set of the fuzzy sound generator can be closer to the real scenario. Among them, the text replaced by the masking mechanism in the error sample is marked according to the initial consonant and final vowel of the text, and the similarity level of the text replaced by the masking mechanism in the error sample is constructed, where the similarity level has the same meaning as the fuzzy level in the attached drawing, indicating the similarity degree with the original text in terms of speech. The calculation method of the similarity level includes: obtaining the flat and retroflex sounds, tones, and front and back nasal sounds of the replaced text, comparing the flat and retroflex sounds, tones, and front and back nasal sounds of the text replaced by the masking mechanism with the correct original text speech, judging the number of the same pronunciation types in the flat and retroflex sounds, tones, and front and back nasal sounds, and configuring the similarity level according to the number of the pronunciation types. For example: it can be divided into N levels according to the similarity of the fuzzy sound. The higher the level, the less similar the pronunciation. For example, level0 is that the pronunciation is exactly the same ("ma" and "ma"), the pronunciation of level1 is the same, but the tones are different ("ke" and "ke"), and level2 is the common fuzzy of flat and retroflex sounds and front and back nasal sounds ("shi" and "si"). According to the overall situation of the marked data, adjust the proportion parameters of different-level fuzzy sounds to generate a data augmentation sample closer to the real data. Or in some other preferred embodiments of the present invention, weight parameters can be configured for different pronunciation types, calculate the weighted sum of the similarities of different types of pronunciations of the masked replacement text and the correct text, and determine the similarity level of the text by setting a level threshold. After adjusting the hyperparameters of the fuzzy sound generator, an error sample closer to the real scenario is obtained.

[0029] After obtaining the error sample, correct sample speech data and text data close to the real scenario, further construct speech feature vectors and text feature vectors by using the speech data and text data respectively. Among them, the constructed speech feature vectors and text feature vectors are fused to obtain a multi-modal fusion feature vector, and the multi-modal fusion feature vector is fused by means including but not limited to vector splicing. Thus, a fusion feature vector that can describe both speech and text can be obtained. After completing the fusion of the speech and text feature vectors, further input the fused feature vector into an error correction model for error judgment. In the present invention, the error correction model is preferably configured as the pre-trained model albert.

[0030] Please combine Figure 3The method for constructing a multimodal fusion feature vector includes: using a bidirectional LSTM network to model the erroneous sample data and the correct sample data at the input, constructing a token sequence containing both speech and text features, and then performing an attention mechanism to filter the vector at each token position. It should be noted that the attention mechanism filtering method mainly queries all audio features at each token position in the text. Because the length of audio and text differs, the length after LSTM network feature extraction is also different. If we can query the audio with text, the output features after the attention mechanism filtering can be aligned with the text to generate a bimodal fusion vector.

[0031] After generating a fused feature vector of audio and text, the detection network further performs a binary classification task on the fused feature vector using a bidirectional LSTM + fully connected layer, calculating the probability of whether each position in the original text contains an error. The predicted probability of a typo at each position is p, and the probability of no typo is 1-p.

[0032] It is worth mentioning that the core technical feature of the error correction model in the present invention lies in the embedding layer of the Albert model. In the present invention, the embedding layer of the Albert model is obtained by the weighted sum of the word embedding of the character at the corresponding position in the original text and the word embedding of the mask [MASK]. The weight value of the mask [MASK] is the probability that the current position is predicted to be a typo by the bidirectional LSTM+fully connected layer of the detection network. Therefore, when the detection network judges that the probability of the current position being a typo is high, then the weight of the mask [MASK] in the word embedding of the position in the correction network is higher. Conversely, if the detection network judges that the probability of the current position being a typo is very low, then the weight of the mask [MASK] in the word embedding of the position in the correction network is lower.

[0033] After using the Albert model to represent each speech position, the output of the last layer of the Albert model is further added to the word embedding of the corresponding position in the original text as the final representation at each moment. The word at each position is predicted through the fully connected layer + Softmax, and finally the word with the highest prediction probability is selected as the output of the current result.

[0034] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium mentioned above in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of an electrical, magnetic, optical, electromagnetic, infrared segment, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wire segments, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, electrical wire, optical fiber cable, RF, etc., or any suitable combination thereof.

[0035] The flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0036] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments. Without departing from the principles, the implementation methods of the present invention may be subject to any deformation or modification.

Claims

1. A multimodal speech recognition error correction method, characterized in that: The method comprises: Obtaining original sample data from a corpus, and generating error samples from the original samples using a fuzzy sound generator; The error samples are labeled according to the initials and finals of the characters, and fuzzy phonetic characters with different similarity levels are constructed; According to the marked error sample data, adjust the parameters of the different similarity level ratios of the fuzzy phonetic characters of the error sample data; Constructing a speech and text fusion feature vector based on the original correct sample and error sample data, inputting the fusion feature vector into the error correction model for training, and outputting the word with the highest correct probability for each speech position through a fully connected layer and activation function; The error correction model construction method comprises: constructing a speech vector and a text vector for the error sample and the original correct sample, respectively, wherein the speech vector comprises a fuzzy speech vector of the error sample, and the text vector comprises a Chinese character vector obtained from the error sample, and generating the speech and text fusion feature vector; The method for constructing a speech and text fusion feature vector includes: using a bidirectional LSTM model to model two sample speech and text respectively, constructing a token sequence after inputting the sample speech and text, and performing an attention mechanism screening on the vector of each token position to obtain a bimodal fusion vector with the same length of Chinese characters and speech.

2. A multimodal speech recognition error correction method according to claim 1, characterized in that: The error sample marking method comprises: marking the flat and retroflex consonants, tones, and front and back nasals of each word in the error sample respectively to obtain marked sample data.

3. The multimodal speech recognition error correction method according to claim 1, characterized in that: The method for generating an error sample comprises: processing an original sample based on the fuzzy pinyin of a fuzzy sound generator to obtain error sample data with different similar pinyin characters.

4. The multimodal speech recognition error correction method according to claim 1, characterized in that: The method for constructing the similarity level of fuzzy sound characters based on error samples includes configuring similarity levels of similar characters of at least one flat and retroflex consonant, tone, and front and back nasal sounds according to the number of similarity types, wherein the more similarity types there are, the higher the similarity level.

5. The multimodal speech recognition error correction method according to claim 1, characterized in that: The error sample generation method includes: adjusting the proportion of different levels of fuzzy sounds according to the overall situation of fuzzy sounds in the error sample, so that the error sample is closer to the real data enhancement sample.

6. A multimodal speech recognition error correction method according to claim 1, characterized in that: The error correction model includes a pre-trained Albert model. In the embedding layer of the pre-trained Albert model, the word embedding of the corresponding position of the sample original text includes a weighted sum of the word embedding of the word at the corresponding position of the sample original text and the word embedding of the mask of the position, wherein the weight of the mask is the probability of the typo at the corresponding position.

7. A multimodal speech recognition error correction system, characterized in that: The system executes a multimodal speech recognition error correction method as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the multimodal speech recognition error correction method according to any one of claims 1 to 6.

Images