Speech recognition model training method and speech processing method of intelligent customer service

Abstract

The invention provides a voice recognition model training method, an intelligent customer service voice processing method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining voice sample data, generating a label for each piece of voice sample data, training a pre-training model through the voice sample data, obtaining a trained pre-training model, and carrying out dialect adaptability training on the pre-training model by using the voice sample data with the dialect label to obtain a dialect understanding model; and carrying out model fusion on the pre-training model and the dialect understanding model to obtain a language understanding model. According to the method, the label is generated for the voice sample data, the pre-training model is trained by using the dialects, so that the dialect understanding model can accurately identify the dialects, the model fused with the pre-training model can accurately identify mandarin and dialects at the same time, answers of the user during question making can be more accurately matched, the communication efficiency is improved, and the user experience is improved. And better service experience is brought to the user.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition, and in particular to a speech recognition model training method, a speech processing method and device for intelligent customer service, an electronic device and a computer-readable storage medium. Background Art

[0002] In recent years, intelligent customer service systems have been widely developed due to their wide coverage time and rapid response. Many manual customer service systems have been integrated with intelligent customer service to serve more users and have received many positive reviews.

[0003] Nowadays, the design and optimization of intelligent customer service are mostly focused on Mandarin, serving Mandarin users, who account for a relatively large proportion of the user population. Intelligent customer service uses voice recognition to determine the user's question and give a reply. Voice recognition has a good recognition accuracy for Mandarin.

[0004] In addition to a large number of users who speak Mandarin, there are also many users who speak dialects. The speech recognition effect of these users after voice input is not very good, which affects the question and answer effect of users. Summary of the invention

[0005] The embodiment of the present invention provides a speech recognition model training method to solve the problem of insufficient recognition accuracy of speech recognition for dialects in the prior art.

[0006] Correspondingly, the embodiments of the present invention also provide a voice processing method, device, electronic device and computer-readable storage medium for intelligent customer service to ensure the implementation and application of the above method.

[0007] In a first aspect, an embodiment of the present invention provides a method for training a speech model, the method comprising:

[0008] Acquire speech sample data, and generate a label for each of the speech sample data; the speech sample data includes Mandarin and various dialects, the labels include Mandarin labels and dialect labels, and the dialect labels are used to indicate the type of the dialect;

[0009] Using the speech sample data to train the pre-trained model to obtain a trained pre-trained model;

[0010] Using the speech sample data with the dialect label, the pre-trained model is trained for dialect adaptability to obtain a dialect understanding model; the dialect understanding model is used to identify the text recognition result of the dialect speech data, and the pre-trained model is used to identify the text recognition result of the Mandarin speech data;

[0011] The pre-trained model and the dialect understanding model are subjected to model fusion to obtain a language understanding model; the language understanding model is used to identify text recognition results of dialect speech data and Mandarin speech data.

[0012] In a second aspect, an embodiment of the present invention provides a speech recognition method, the method comprising:

[0013] Get the audio data to be processed input by the user;

[0014] Inputting the audio data to be processed into the trained language understanding model to obtain a text recognition result corresponding to the audio data to be processed;

[0015] According to the text recognition result and the preset knowledge base, a reply sentence matching the text recognition result is determined;

[0016] The reply sentence is sent to the device of the user so that the reply sentence serves as a reply to the audio data to be processed.

[0017] In a third aspect, an embodiment of the present invention provides a speech recognition model training device, the device comprising:

[0018] A first acquisition module, used to acquire speech sample data and generate a label for each speech sample data;

[0019] A pre-training module, used to train a pre-training model using the speech sample data to obtain a trained pre-training model;

[0020] A dialect training module, used to perform dialect adaptability training on the pre-trained model using speech sample data with the dialect label to obtain a dialect understanding model;

[0021] The fusion module performs model fusion on the pre-trained model and the dialect understanding model to obtain a language understanding model.

[0022] In a fourth aspect, an embodiment of the present invention provides a voice processing device for intelligent customer service, the device comprising:

[0023] The second acquisition module acquires the audio data to be processed input by the user;

[0024] A recognition module, used for inputting the audio data to be processed into the trained language understanding model to obtain a text recognition result corresponding to the audio data to be processed;

[0025] The question-answering module is used to determine a reply sentence matching the text recognition result based on the text recognition result and a preset knowledge base;

[0026] The reply module is used to send the reply statement to the user's device so that the reply statement serves as a reply to the audio data to be processed.

[0027] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of one or more methods described in the embodiments of the present invention are implemented.

[0028] In a sixth aspect, an embodiment of the present invention provides a readable storage medium. When instructions in the readable storage medium are executed by a processor of an electronic device, the electronic device can execute one or more methods described in the embodiments of the present invention.

[0029] Compared with the related art, the embodiments of the present invention have the following advantages:

[0030] In an example of the present invention, speech sample data is obtained, and a label is generated for each of the speech sample data, and a pre-trained model is trained using the speech sample data to obtain a trained pre-trained model. The speech sample data with the dialect label is used to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model; the pre-trained model and the dialect understanding model are fused to obtain a language understanding model. The present invention generates labels for speech sample data, classifies speech sample data to be trained, and after generating labels for the speech sample data, uses the speech sample data to train the pre-trained model, so that the trained pre-trained model has a basis for speech recognition. Since the speech sample data contains a large number of audio data with Mandarin labels, the recognition accuracy of Mandarin will continue to increase during training, so the pre-trained model has a good understanding ability for Mandarin. After the pre-training model is trained, all the voice sample data with dialect labels are used to perform dialect adaptation training on the pre-training model. Since the voice sample data used are all voice sample data with dialect labels, the training results are close to the dialect training results, so that the pre-trained model has the ability to recognize dialects and obtains a dialect understanding model. Finally, the dialect understanding model and the pre-training model are fused so that the fused language understanding model has the ability to accurately recognize both Mandarin and dialects. By accurately recognizing dialects and Mandarin, it can more accurately match the answers to user questions, improve communication efficiency, and bring better service experience to users.

[0031] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 is a flowchart of a method for training a speech recognition model provided by an embodiment of the present invention;

[0034] Figure 2 is a flowchart of another method for training a speech recognition model provided by an embodiment of the present invention;

[0035] Figure 3 This is a flowchart of a voice processing method for intelligent customer service provided by an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram of the overall process of an intelligent customer service system interacting with a user provided by an embodiment of the present invention;

[0037] Figure 5 is a structural diagram of a speech recognition model training device provided by an embodiment of the present invention;

[0038] Figure 6 It is a structural diagram of a voice processing device for intelligent customer service provided by an embodiment of the present invention;

[0039] Figure 7 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Usually when users interact with computers, they need to convert their input into computer-readable input information, such as keystrokes, binary codes, or character sequences. For users, voice communication is a more convenient and quick way of communication. To use this communication method between users and computers, it is necessary to recognize the voice information input by users and convert the audio information into computer-readable input through voice recognition.

[0042] Voice recognition function is used in all aspects of our lives. For example, when it is inconvenient to input information manually, it is very convenient and fast to input voice information through voice recognition function. Especially in recent years, the intelligent customer service system has been rapidly developed. It has become an important means of communicating with users. When users encounter problems, they need to contact customer service, and voice communication is the most convenient and fastest way for users to communicate. Users explain their problems through voice input, and then the intelligent customer service performs voice recognition, understands the user's problem, searches for answers and provides feedback to complete customer service work.

[0043] The speech recognition model training method of the present invention is further described below in conjunction with the relevant drawings and embodiments:

[0044] Figure 1 is a flowchart of a method for training a speech recognition model provided by an embodiment of the present invention. Figure 1 As shown, the method may include:

[0045] Step 101: Acquire speech sample data and generate a label for each speech sample data.

[0046] In the embodiment of the present invention, the voice sample data refers to the audio data used for communication, and the audio data stores the information of the user's language communication. When the user communicates with the language, he or she may not necessarily use Mandarin, but may also use various local dialects. The specific language used is affected by the user's life manager.

[0047] Dialects are variants of languages ​​in different regions. They are usually pronounced differently from Mandarin. Some regions also have words with specific meanings that are only circulated locally. Sometimes the same word has different usage scenarios and specific meanings in different regions. The speech sample data obtained in the embodiment of the present invention includes Mandarin speech sample data and dialect speech sample data.

[0048] The label of the speech sample data is an identification symbol carried by the speech sample data, and the label stores the dialect information of the speech sample, which is used to indicate whether the speech sample data is Mandarin or a dialect, and to which dialect the speech sample data belongs.

[0049] For example, the content of the audio information of a voice sample data is "The weather is really nice today". When the user who recorded the voice sample data spoke Sichuan dialect, the identification symbol information carried by the label is {dialect; Sichuan dialect}; when the user who recorded the voice sample data spoke Mandarin, the identification symbol information carried by the label is {Mandarin; null}.

[0050] After obtaining the speech sample data, a label is generated for each speech sample data, indicating the dialect type of the speech sample data, that is, different dialect speech sample data are distinguished, so that the sample data has clear branches in the dialect category, avoiding confusion between dialects and ensuring that the model can fully learn the characteristics of various dialects.

[0051] Step 102: Use the speech sample data to train a pre-trained model to obtain a trained pre-trained model.

[0052] In an embodiment of the present invention, the pre-trained model is trained by supervised classification, and the speech sample data is sample data provided to the model for supervised classification. By training the pre-trained model with the speech sample data, the pre-trained model has the function of recognizing speech data.

[0053] Specifically, a pre-trained model is a model that is pre-trained using a large amount of data sets in an original specific task. The model can be used on the target task later. The model can be fine-tuned according to the characteristics of the target task, so as to achieve the purpose of improving the execution effect of the target task. Its essence utilizes the theory of transfer learning. By inputting speech sample data into the pre-trained model, outputting the text recognition result of the speech sample, calculating the difference between the recognition result and the sample, and determining the loss function, the training of the pre-trained model is completed, so that the pre-trained model has the function of speech recognition. These models usually have millions to billions or even more parameters, and can handle extremely complex tasks, such as natural language processing, computer vision, speech recognition, etc.

[0054] In addition, the pre-trained model only has the speech recognition function of Mandarin, and the speech recognition effect of dialects is not ideal. However, the number of dialect speech sample data is relatively scarce compared to the number of Mandarin speech sample data. It is difficult to have a good training effect on the pre-trained model using a small number of sample data, so that it can recognize dialect speech data. Therefore, the embodiment of the present invention first trains a pre-trained model for recognizing Mandarin speech data, and then uses the dialect speech data to further adjust the pre-trained model to make up for the difficulties in dialect speech recognition caused by the scarcity of dialect speech sample data.

[0055] Step 103: Use the speech sample data with the dialect label to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model.

[0056] In the embodiment of the present invention, dialect adaptive training is an improved training method for improving the training effect of the pre-training model on the dialect by using limited dialect speech sample data and adjusting specific layers in the pre-training model.

[0057] The pre-trained model is usually a pre-trained model trained on a large-scale general dataset. The model usually used for speech recognition includes the following layers:

[0058] Input layer: used to receive original audio signals or other features.

[0059] Convolutional layer: used to extract local features, such as edges and textures in the spectrogram, and is often used to capture the time and frequency characteristics of audio signals.

[0060] Recurrent layer: It is used to capture long-term dependencies of time series data, which is particularly important for speech recognition because speech signals are temporally continuous.

[0061] Attention mechanism: used to focus on the important parts of the input sequence to improve the model's interpretability and performance.

[0062] Fully connected layer: used to map the extracted features to the final output categories, usually used for hierarchical tasks.

[0063] Output layer: Generates the final recognition result, such as a text sequence.

[0064] Among them, by adding more feature dimensions to the input layer, the unique acoustic features of the dialect can be better captured. Increasing the number of convolutional layers or adjusting the size of the convolution kernel in the convolutional layer can better capture the unique frequency and time features of the dialect. Optimizing the parameters of the attention mechanism can also enable the model to better focus on the unique features of the dialect.

[0065] In an embodiment of the present invention, a pre-trained model is obtained, and then a specific layer of the pre-trained model is adjusted, and speech sample data with dialect labels is input to obtain an output value of a training process. The output value and the speech sample data with dialect labels are used to calculate the loss value, and the loss value of this training can be obtained. The loss function is determined based on the loss value, and the parameters of the pre-trained model can be trained, so that after multiple rounds of training operations are completed and the training objectives are met, a dialect understanding model is obtained. The embodiment of the present invention does not limit the selection of the loss function. By training the pre-trained model, the model with basic speech recognition function can be improved to obtain the ability to recognize dialect speech data, and the training of the dialect understanding model can be realized in the limited speech sample data with dialect labels, which makes up for the lack of speech sample data with dialect labels.

[0066] In addition, in addition to adjusting specific layers in the pre-trained model, the ability to recognize and understand dialects can also be optimized through joint training. Through multi-task learning, parameters are shared in the first four layers of the model so that the model can learn common features, and a multi-task loss function is designed to combine the loss terms of multiple tasks. A multi-task data set is created to receive the data set for multi-task learning, and the weight of the task loss is dynamically adjusted according to the difficulty of the task. During the training process, the losses of different tasks are alternately optimized so that the model can achieve better performance on different tasks. Step 104, the pre-trained model and the dialect understanding model are model-fused to obtain a language understanding model.

[0067] In the related art, due to the imbalance in the data volume of dialect speech data and Mandarin speech data, the data volume of dialect speech data is not as rich as that of Mandarin speech data, resulting in limited performance of the model in recognizing and understanding dialect speech data.

[0068] In the embodiment of the present invention, the pre-trained model is trained by voice sample data so that the pre-trained model has a certain ability to recognize universal features and realizes the recognition of Mandarin voice data. Although there are certain differences in the features of dialects and Mandarin, some features are relatively similar. At this time, dialect voice sample data with a smaller amount of data is used to perform dialect adaptability training on the pre-trained model, and the pre-trained model is adjusted so that the pre-trained model pays more attention to and captures the features of the dialect on the basis of basic Mandarin voice understanding, so that the trained dialect understanding model can recognize the dialect voice data, which solves the problem that it is difficult to train a dialect understanding model for understanding dialect voice with insufficient dialect voice sample data, and improves the utilization efficiency of dialect voice sample data.

[0069] In an embodiment of the present invention, the pre-trained model is trained by using speech sample data with dialect labels and speech sample data with Mandarin labels, so that the pre-trained model can have a better effect on the recognition of the general features of the speech data, so the pre-trained model can have accurate recognition results when recognizing Mandarin speech data, and the dialect understanding model is trained using speech sample data with dialect labels on the basis of the pre-trained model, so that the dialect understanding model has a better effect on the recognition of dialect features, so the dialect understanding model has accurate recognition results for dialect speech data, in order to make the model have both accurate recognition of Mandarin speech data and accurate recognition of dialect speech data, the pre-trained model and the dialect understanding model are fused, and the recognition effects of the general features and the recognition effects of the dialect features are integrated into one model, so as to generate a language understanding model that has the function of accurately recognizing both Mandarin speech data and dialect speech data.

[0070] For example, model A has a good capture effect on feature a, and when model A receives data a, it can accurately identify it; model B has a good capture effect on feature b, and when model B receives data b, it can accurately identify it. At this time, a model C is needed that can recognize both feature a and feature b. By using model fusion, model C can capture both feature a and feature b well, so as to achieve accurate identification of data a and data b.

[0071] In addition, through model fusion, the functions of understanding Mandarin speech data and dialect speech data, which require pre-trained models and dialect understanding models, can be integrated into one language understanding model. Not only is there no need to determine whether the speech data is a dialect and input it into the corresponding model in order to identify the speech data, but the computing cost and storage requirements of the two models can also be saved to the computing cost and storage requirements of one model, reducing the operation steps, improving efficiency and saving costs.

[0072] In an example of the present invention, speech sample data is obtained, and a label is generated for each of the speech sample data, and a pre-trained model is trained using the speech sample data to obtain a trained pre-trained model. The speech sample data with the dialect label is used to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model; the pre-trained model and the dialect understanding model are fused to obtain a language understanding model. The present invention generates labels for speech sample data, classifies speech sample data to be trained, and after generating labels for the speech sample data, uses the speech sample data to train the pre-trained model, so that the trained pre-trained model has a basis for speech recognition. Since the speech sample data contains a large number of audio data with Mandarin labels, the recognition accuracy of Mandarin will continue to increase during training, so the pre-trained model has a good understanding ability for Mandarin. After the pre-training model is trained, all the voice sample data with dialect labels are used to perform dialect adaptation training on the pre-training model. Since the voice sample data used are all voice sample data with dialect labels, the training results are close to the dialect training results, so that the pre-trained model has the ability to recognize dialects and obtains a dialect understanding model. Finally, the dialect understanding model and the pre-training model are fused so that the fused language understanding model has the ability to accurately recognize both Mandarin and dialects. By accurately recognizing dialects and Mandarin, it can more accurately match the answers to user questions, improve communication efficiency, and bring better service experience to users.

[0073] Reference Figure 2 , the speech recognition model training method may include the following steps:

[0074] Step 201, obtaining voice sample data.

[0075] This step may specifically refer to the above step 101, which will not be described in detail here.

[0076] Optionally, step 201 may specifically include:

[0077] Sub-step 2011, performing text transcription according to the acquired voice sample data to obtain a text transcription result.

[0078] In an embodiment of the present invention, a large amount of voice sample data can be collected through recording equipment or an online platform, and the voice sample data can be transcribed into text by professionals to obtain text information corresponding to the voice sample data. Through text transcription, the information of the voice sample data is specifically implemented in the form of text, and the specific words and expressions in the dialect are more clearly reflected, so that the voice sample data adds information of the text dimension, and obtains sample data with more rich information, adding more features to the limited dialect voice sample data, providing sample data with richer dialect information for the subsequent use of the dialect voice sample data training model, and improving the utilization rate of the dialect voice sample data.

[0079] For example, when the pronunciation of a voice sample data is "jīn tiān de tiān qìzhēn hǎo yā", through text transcription, the text content of "Today's weather is really good" is obtained, and the text content is the text content information corresponding to the voice information contained in the voice sample data.

[0080] In addition, in some life scenarios, we often encounter situations where many pronunciations correspond to multiple characters. Text transcription of speech sample data can more directly determine the text information that the speech sample data should correspond to in the current context. These text information can be determined by professionals through the contextual information content. Correspondingly, using speech sample data with text transcription information to train the model can link the speech sample data with the contextual information when it appears. When the model recognizes the contextual information content, it can make a clear judgment on the meaning carried by the appearing speech data, allowing the model to clarify the content corresponding to the current speech data through the contextual information of the speech data, thereby forming a more accurate recognition.

[0081] For example, when the pronunciation of a voice sample data is "bào fù", it is difficult to determine the corresponding text information solely based on the pronunciation. For such homophonic words with different meanings, it is necessary to refer to the context information to determine the specific meaning it intends to express and match the corresponding text result. If the context information is discussing a person's ambition, then the text information corresponding to this voice is likely to be "抱负" (ambition). If the context information is about money, then the corresponding text information is likely to be "暴富" (sudden wealth). If the context information is about some unfortunate experiences, the corresponding text information may be "报复" (retaliation).

[0082] Sub-step 2012: Combine the text transcription result with the initials and pinyin of the text to obtain the text sample data.

[0083] In the embodiments of the present invention, the pinyin and initials are generated in combination with the voice sample data. The pinyin and initials are also the literal manifestations of the voice. If the voice sample data is in Mandarin, the pinyin and initials correspond to the standard pinyin of Mandarin. If the voice sample data is in a dialect, then the pinyin and initials are adjusted approximately according to the specific pronunciation. For example, in Cantonese, the pronunciation of "一" is relatively similar to "yā", so "yā" is used as the pinyin of "一", while in Mandarin, the pronunciation of "一" is "yī", and both the pinyin and initials change. At this time, the pinyin and initials should be recorded according to the voice sample data.

[0084] Among them, for the convenience of recording pinyin and initials, the pinyin of "你好" (nǐhǎo) can be marked as "ni3hao3", and the initials of "你好" (nǐhǎo) can be marked as "i" and "ao".

[0085] In addition, recording the pinyin and initials of the voice sample data is also to better improve the utilization rate of dialect voice sample data, so that the dialect voice sample data has more features. By marking the pinyin, the model can understand the features at the pinyin level. Especially when using dialect voice sample data to train a pre-trained model, since there are some similarities between the pinyin of Mandarin and dialects, it can help the pre-trained model better master the function of recognizing dialect voice data. And by marking the initials, the model can better understand the features at the phoneme level. By learning the initials, the model can better master the pronunciation rules in the dialect, help the model extract more refined voice features, improve the recognition accuracy, and distinguish the subtle differences in different dialects, thus avoiding the confusion of dialects.

[0086] Sub-step 2013: Generate labels for the text sample data and the voice sample data respectively.

[0087] The labels generated by the text sample data are similar to the labels of the speech sample data. The labels store the dialect information of the speech sample, which is used to indicate whether the speech sample data is Mandarin or a dialect, and to which dialect the speech sample data belongs.

[0088] In the example of the present invention, the text sample data includes the text content, pinyin and final vowel information transcribed from the corresponding voice sample data, and both the text sample data and the voice sample data have information for distinguishing between Mandarin and various dialects, forming a unified format for easy input into the model for training.

[0089] Step 202: Use the speech sample data to train a pre-trained model to obtain a trained pre-trained model.

[0090] This step may specifically refer to the above step 102, which will not be described in detail here.

[0091] Optionally, before step 202, the method further includes:

[0092] Sub-step 2021, performing noise filtering on the speech sample data to obtain noise-filtered speech sample data.

[0093] The noise in the voice sample data may include, for example, background noise, small human voices in the background, current sounds generated by the sound receiving device when it is disturbed, and the like.

[0094] In an embodiment of the present invention, by performing noise filtering on the speech sample data, the influence of noise on the speech sample data is eliminated as much as possible, so that when the speech sample data is used to train the model, the speech features extracted by the model are affected by the noise, thereby improving the accuracy of the model for speech recognition.

[0095] The noise may be filtered by linear filtering, Wiener filtering, subspace algorithm or machine learning method, and the embodiment of the present invention does not limit the selection of noise filtering.

[0096] Sub-step 2022, performing audio segmentation on the noise-filtered speech sample data word by word, and obtaining single-word speech data after segmentation.

[0097] In an embodiment of the present invention, the speech sample data after noise filtering is audio segmented word by word, so that the speech sample data forms the smallest language understanding unit, and single-word speech data is obtained. After training the pre-training model, the pre-training model can recognize the smallest language understanding unit, that is, a single word. After recognition, the content corresponding to the speech data can be better recognized through understanding of the context. If the audio data is not segmented into the smallest language understanding unit, some sentences with the same meaning will change their sentence structure, which will require the pre-training model to input richer speech sample data for training before it can recognize the content.

[0098] For example, in daily spoken language, people often greet others with "Have you eaten?" In some specific regions, people are used to using inversion in language expression, expressing it in the form of "Have you eaten?" This form does not affect understanding and use in daily communication. If the audio is segmented by the smallest comprehensible unit, the audio data obtained is "you", "eat", "meal", "have", "have", and the recognition effect can be achieved when encountering an inverted sentence like "Have you eaten?" If the audio data is not segmented or segmented by grammatical structure to obtain sentences like "you", "meal", "have", the recognition effect for inverted sentences will be reduced.

[0099] Sub-step 2022, after detecting that the audio segmentation is completed, training the pre-trained model according to the single-word voice data and the label of the voice sample data to which the single-word voice data belongs, to obtain a trained pre-trained model.

[0100] In an embodiment of the present invention, by automatically detecting the end of the audio segmentation process in a voice sample data, single-word voice data arranged according to the audio content of the voice sample data is obtained, and the single-word voice data is automatically input into the pre-training model in sequence, so that the voice sample data is input into the pre-training model without manual intervention, thereby improving the efficiency of the training model and reducing the labor cost of the training model.

[0101] For example, the audio information of a speech sample data is "jīn tiān de tiān qìzhēn hǎo yā". After audio segmentation of the speech sample data, "jīn", "tiān", "de", "tiān", "qì", "zhēn", "hǎo", and "yā" are obtained. After the segmentation process is completed, the 8 single-word speech data segmented are input into the pre-training model in sequence.

[0102] Step 203: Use the speech sample data with the dialect label to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model.

[0103] This step may be specifically referred to as the above step 103, which will not be described in detail here.

[0104] Optionally, step 203 may specifically include:

[0105] Sub-step 2031, combining the text sample data with the dialect label with the speech sample data with the dialect label to obtain multimodal dialect data containing the speech sample data and the corresponding text sample data.

[0106] In an embodiment of the present invention, multimodal dialect data refers to text sample data with dialect labels and speech sample data with dialect labels, wherein the label with the speech sample data carries dialect type information, and the text sample data carries the text transcription of the speech sample data close to the pinyin and finals of the dialect pronunciation, and the label with the text sample data carries dialect type information.

[0107] In addition, corresponding text sample data with dialect labels are generated for speech sample data with dialect labels. On the one hand, because there are certain differences between dialect pronunciation and standardized pronunciation, the connection between the different pronunciations and standardized pronunciations can be established through text sample information; on the other hand, some words and sentences in dialects have certain regional characteristics, which do not exist in other regions or the meanings of the expressions are very different. Through text, a concept unique to the region can be formed to avoid contaminating the meanings of expressions in other regions.

[0108] Sub-step 2032, using the multimodal dialect data, performing dialect adaptation training on the pre-trained model to obtain a dialect understanding model.

[0109] Among them, dialect adaptive training is to train the pre-trained model through sample data with dialect characteristics, and at the same time adjust the layer structure of the pre-trained model to enable it to better capture the dialect characteristics.

[0110] In an embodiment of the present invention, a pre-trained model is trained by training a pre-trained model with speech sample data, and then a dialect-adaptive training is performed on the pre-trained model using multimodal data to obtain a dialect understanding model. Since the multimodal data has as rich dialect data features as possible, including the pronunciation of the dialect, the corresponding characters, pinyin, vowels, and the dialect usage area, the pre-trained model function with basic speech data understanding is developed in the direction of understanding dialect speech data. The pronunciation, characters, pinyin, vowels, and regional information of the dialect provide the pre-trained model with a connection with Mandarin speech sample information, so that a dialect understanding model is trained in limited dialect speech sample data. At the same time, the speech sample data and text sample data can enable the model to better understand the features at the phoneme level, master the pronunciation rules in the dialect, help the model extract more refined speech features, improve recognition accuracy, distinguish subtle differences in different dialects, and thus avoid confusion of dialects.

[0111] Optionally, step 203 may specifically include:

[0112] Sub-step 2033, when it is detected that the speech sample data with the dialect label is updated, the pre-trained model is subjected to dialect adaptation training using the speech sample data with the dialect label to obtain a dialect understanding model.

[0113] Currently, the existing speech sample data is unbalanced. The amount of Mandarin speech sample data is abundant, while the amount of dialect speech sample data is scarce. Therefore, when new dialect speech sample data is received, the pre-trained model is trained in time to obtain a dialect understanding model, thereby improving the recognition accuracy of the dialect understanding model for the dialect.

[0114] In addition, the speed of information dissemination is getting faster and faster nowadays, which also has a great impact on language. Timely training of new dialect speech sample data to obtain a dialect understanding model is also to enable the dialect understanding model to keep up with the evolution and development of dialects.

[0115] Sub-step 2034, when it is detected that the processing concurrent task volume of the dialect understanding model is less than a preset threshold, the pre-trained model is subjected to dialect adaptive training using speech sample data with the dialect label to obtain a dialect understanding model.

[0116] In the embodiment of the present invention, the processing performance of the dialect understanding model is also monitored.

[0117] For example, before performing dialect adaptation training, the pre-trained model recognizes 100 records of speech data under concurrent tasks. For the dialect understanding model, if the number of speech data processed by the dialect understanding model under concurrent tasks is less than 100 records, it is considered that the processing performance of the dialect understanding model does not meet the requirements and further training is required to improve the response speed of recognizing speech data.

[0118] Sub-step 2035, when receiving the fine-tuning instruction, corresponding to the fine-tuning instruction, using the speech sample data to train the pre-trained model, using the speech sample data with the dialect label to perform dialect adaptation training on the pre-trained model to obtain a dialect understanding model.

[0119] In a specific embodiment of the present invention, the user experience can be used to feedback whether the recognition accuracy of the dialect recognition model is recognized by the user. When unsatisfactory feedback is received from the user, a fine-tuning instruction is issued to continue to obtain the dialect understanding model through dialect adaptive training.

[0120] Optionally, step 203 may specifically include:

[0121] Sub-step 2036, increasing the number of convolutional layers of the pre-trained model to obtain a convolutional layer that extracts dialect features.

[0122] Among them, the convolution layer is used to extract local features, usually used to extract the time and frequency features of audio signals. By increasing the number of convolution layers, the dialect-specific time and frequency features can be better captured.

[0123] In an embodiment of the present invention, by freezing some convolutional layers, such as the convolutional layers used to extract features of Mandarin speech sample data in an early pre-trained model, the convolutional layers for extracting features of dialect speech sample data are kept working normally, so that the convolutional layers can increase the recognition accuracy of dialect features while retaining common features.

[0124] Sub-step 2037, increasing the number of loop layers of the loop layer of the pre-trained model to obtain a bidirectional loop layer.

[0125] Among them, the recurrent layer extracts the long-term dependencies of time series data, while speech data has temporal continuity.

[0126] In the embodiment of the present invention, by increasing the number of recurrent layers and using bidirectional recurrent layers, the ability of the dialect recognition model to extract long-term dependencies is enhanced while taking into account the context information of the speech sample data.

[0127] Sub-step 2038, increasing the number of fully connected layers of the pre-trained model to obtain a fully connected layer that maps dialect features.

[0128] Among them, the fully connected layer is used to map the extracted features to the final output category.

[0129] In the embodiment of the present invention, the number of fully connected layers is increased so that the additional fully connected layers are used to specifically map dialect features, thereby ensuring the classification task of dialect features.

[0130] Step 204: input the speech sample data into the pre-trained model, and output a first predicted text recognition result of the speech sample data.

[0131] Among them, the speech sample data is input into the pre-trained model to obtain the recognition result of the speech sample data by the pre-trained model, which is the first predicted text recognition result. The speech sample data is re-recognized using the pre-trained model in order to have a specific recognition result for the speech recognition effect of the pre-trained model. At the same time, the recognition result after recognition by the pre-trained model makes the feature extraction recognition result of the pre-trained model more in line with the model training habits. When the target model is trained using the first predicted text recognition result and the speech sample data associated with it, it helps the target model to learn richer features.

[0132] Step 205: input the speech sample data into the dialect understanding model, and output a second predicted text recognition result.

[0133] The speech sample data is input into the dialect understanding model, and the recognition result of the dialect understanding model on the speech sample data is obtained, which is the second predicted text recognition result.

[0134] Step 206: Fusing the first predicted text recognition result and the second predicted text recognition result to form a target predicted text recognition result of the speech sample data.

[0135] The first predicted text recognition result and the second predicted text recognition result are fused, that is, the data of the two are integrated to form a target predicted text recognition result of the speech sample data. The target predicted text recognition result includes recognition data of a pre-trained model for recognizing Mandarin and recognition data of a dialect understanding model for recognizing dialects.

[0136] Step 207: training the initial model based on the speech sample data with the Mandarin label, the speech sample data with the dialect label, and the speech sample data associated with the target predicted text recognition result to obtain the language understanding model.

[0137] The speech sample data associated with the target predicted text recognition result refers to the speech sample data corresponding to the predicted text recognition result. Since it is the result of recognition by two models, one speech sample data corresponds to two target predicted text recognition results.

[0138] The initial model refers to the initial model used by the language understanding model trained using speech sample data with Mandarin labels, speech sample data with dialect labels, and speech sample data associated with target predicted text recognition results. Specifically, a pre-trained model can be used as the initial model.

[0139] Optionally, step 207 may specifically include:

[0140] Sub-step 2071, inputting the speech sample data with the Mandarin label, the speech sample data with the dialect label, and the speech sample data associated with the target predicted text recognition result into the initial model to obtain a third predicted text recognition result output by the initial model.

[0141] In an embodiment of the present application, an initial model is used to recognize speech sample data with Mandarin labels and speech sample data with dialect labels, and speech sample data associated with target predicted text recognition results, so that the initial model has initial recognition results for the above three sample data.

[0142] Sub-step 2072, determining a first loss value according to the difference between the third predicted text recognition result and the target predicted text recognition result.

[0143] Among them, the difference represented by the first loss value indicates the difference between the speech recognition function of the initial model and the trained pre-trained model and dialect understanding model. The smaller the first loss value, the more the function of the initial model is biased towards the pre-trained model and the dialect understanding model.

[0144] Sub-step 2073, determining a second loss value based on the difference between the third predicted text recognition result, the Mandarin label, and the dialect label.

[0145] Among them, the difference represented by the second loss value indicates the accuracy of the initial model in recognizing the real speech sample data. The smaller the second loss value, the higher the accuracy of the initial model in recognizing the real speech sample data.

[0146] Sub-step 2074 trains the initial model according to the first loss value, the second loss value and a preset loss function to obtain the language understanding model.

[0147] The loss function is calculated by using the first loss value and the second loss value to balance the recognition accuracy of the initial model in the real speech sample data and the recognition similarity of the pre-trained model and the dialect understanding model after training, so as to obtain a language understanding model.

[0148] In addition, the loss function is used to balance the impact of the two trained models and the real speech sample data on the language understanding model, so that while the language understanding model learns the speech features of the two trained models, there will be no distortion in the recognition of the real speech sample data, thereby ensuring the accuracy of the language understanding model in speech recognition. The language understanding model has the ability to recognize Mandarin speech data and dialect speech data, and the functions of the two models are integrated into the language understanding model, so that the computational cost and storage requirements of the two models are reduced to the computational cost and storage requirements of one model, thereby saving costs.

[0149] For example, the computing power of the pre-trained model is 100 operations (Ops) and the storage requirement is 10GB, which is used to realize the recognition of Mandarin speech data. The computing power of the dialect understanding model is also 100Ops and the storage requirement is 10GB, which is used to realize the recognition of dialect speech data. The two together complete the recognition of Mandarin and dialect speech data. The computing power used by the fused language understanding model to realize the recognition of Mandarin and dialect speech data is still 100Ops, and the storage requirement is still 10GB, but it can well retain the functions of the two models.

[0150] In an example of the present invention, by acquiring speech sample data and generating a label for each of the speech sample data, the speech sample data is used to train a pre-trained model to obtain a trained pre-trained model, and the speech sample data with the dialect label is used to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model; the pre-trained model and the dialect understanding model are model-fused to obtain a language understanding model. The present invention generates labels for speech sample data to classify speech sample data to be trained. In addition to the basic dialect and Mandarin categories, the specific dialect system to which the speech sample data belongs is also subdivided under the dialect label, so that the label classification of the speech sample data is more detailed, providing a basis for training a dialect understanding model that can recognize multiple dialects, and generating text sample data for the speech sample data at the same time. After the speech sample data and the text sample data are labeled, the pre-trained model is trained so that the trained pre-trained model has the basis for speech recognition. Since the speech sample data contains a large number of audio data with Mandarin labels, the recognition accuracy of Mandarin will continue to increase during training, so the pre-trained model has a good understanding ability for Mandarin. After the pre-training model is trained, the multimodal data formed by all the voice sample data and text sample data with dialect labels are used to perform dialect adaptation training on the pre-training model. Since the text sample data better links the dialect and Mandarin, the dialect understanding model obtained through dialect adaptation training of the pre-training model has higher accuracy in recognizing dialects. Finally, the dialect understanding model and the pre-training model are fused so that the fused language understanding model has the ability to accurately recognize both Mandarin and dialects. By accurately recognizing dialects and Mandarin, it can more accurately match the answers to users' questions, improve communication efficiency, and bring better service experience to users.

[0151] Figure 3 is a flowchart of a method for voice processing of intelligent customer service provided by an embodiment of the present invention. Figure 3 As shown, the method may include:

[0152] Step 301: Obtain audio data to be processed input by a user.

[0153] Among them, the method of obtaining user input can be that the user uses a recording device to record the user, or the user records the phone call when communicating with the intelligent customer service.

[0154] Step 302: input the audio data to be processed into the trained language understanding model to obtain a text recognition result corresponding to the audio data to be processed.

[0155] In an embodiment of the present invention, a user inputs audio data to be processed, and the audio data may be Mandarin or a dialect. The dialect or Mandarin is recognized as text through a trained language understanding model to obtain a recognition result.

[0156] Step 303, based on the text recognition result and a preset knowledge base, determine a reply sentence that matches the text recognition result.

[0157] The text recognition results are matched with the preset knowledge base for questions. The knowledge base returns the reply through the answers corresponding to the stored questions, which are then relayed to the user by the intelligent customer service.

[0158] Step 304: Send the reply statement to the user's device so that the reply statement serves as a reply to the audio data to be processed.

[0159] For example, when a user asks what the weather will be like tomorrow, the intelligent customer service will use the language understanding model to recognize the user's voice input, obtain computer-readable input information, and match it with the preset knowledge base based on the content of the input information. After querying the knowledge base, it will find out that tomorrow will be sunny, and the information that tomorrow will be sunny will be returned to the intelligent customer service. At this time, the intelligent customer service will convert the information returned by the knowledge base into audio data that the user can understand, and reply to the user, "Hello, it will be sunny tomorrow, and it is a good time to go out for exercise."

[0160] Figure 4 The overall process of the intelligent customer service system interacting with users:

[0161] When a user makes a call to the intelligent customer service, the user inputs voice data during the communication and the intelligent customer service receives it. On the one hand, the intelligent customer service identifies the user's dialect, calls up the language understanding model for identification, obtains computer-readable input information, matches relevant knowledge in the preset knowledge base based on the computer-readable input information after voice recognition, obtains the answer to the user's question, and responds to the user by converting the answer text into audio data using the corresponding voice. On the other hand, when the user speaks, the dialect voice sample data is updated, and after generating labels for the dialect voice sample data, the pre-trained model is trained to obtain the dialect understanding model. After model fusion, the language understanding model is formed. When communicating with the user, voice sample data with dialect labels is continuously accumulated, so that the final language understanding model can accurately recognize and understand the dialect voice data, so that subsequent users can have better voice communication effects when communicating.

[0162] In an example of the present invention, speech sample data is obtained, and a label is generated for each of the speech sample data, and a pre-trained model is trained using the speech sample data to obtain a trained pre-trained model. The speech sample data with the dialect label is used to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model; the pre-trained model and the dialect understanding model are fused to obtain a language understanding model. The present invention generates labels for speech sample data, classifies speech sample data to be trained, and after generating labels for the speech sample data, uses the speech sample data to train the pre-trained model, so that the trained pre-trained model has a basis for speech recognition. Since the speech sample data contains a large number of audio data with Mandarin labels, the recognition accuracy of Mandarin will continue to increase during training, so the pre-trained model has a good understanding ability for Mandarin. After the pre-training model is trained, all the voice sample data with dialect labels are used to perform dialect adaptation training on the pre-training model. Since the voice sample data used are all voice sample data with dialect labels, the training results are close to the dialect training results, so that the pre-trained model has the ability to recognize dialects and obtains a dialect understanding model. Finally, the dialect understanding model and the pre-training model are fused so that the fused language understanding model has the ability to accurately recognize both Mandarin and dialects. By accurately recognizing dialects and Mandarin, it can more accurately match the answers to user questions, improve communication efficiency, and bring better service experience to users.

[0163] Figure 5 A speech recognition model training device is provided in an embodiment of the present invention. Figure 5 As shown, the device may include:

[0164] The first acquisition module 401 is used to acquire speech sample data and generate a label for each speech sample data.

[0165] The pre-training module 402 is used to train a pre-training model using the speech sample data to obtain a trained pre-training model.

[0166] The dialect training module 403 is used to use the speech sample data with the dialect label to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model.

[0167] The fusion module 404 performs model fusion on the pre-trained model and the dialect understanding model to obtain a language understanding model.

[0168] Optionally, the first acquisition module 401 may specifically include:

[0169] The text transcription submodule performs text transcription based on the acquired voice sample data to obtain a text transcription result.

[0170] The text sample submodule combines the text transcription result with the word finals and pinyin to obtain text sample data.

[0171] The label submodule generates labels for the text sample data and the voice sample data respectively.

[0172] Optionally, the pre-training module 402 may specifically include:

[0173] The noise filtering submodule performs noise filtering on the speech sample data to obtain noise-filtered speech sample data.

[0174] The audio segmentation submodule performs audio segmentation on the noise-filtered speech sample data word by word to obtain single-word speech data after segmentation.

[0175] The sample processing submodule, after detecting that the audio segmentation is completed, trains the pre-training model according to the single-word voice data and the label of the voice sample data to which the single-word voice data belongs to obtain a trained pre-training model.

[0176] Optionally, the dialect training module 403 may specifically include:

[0177] The update detection submodule, when detecting that the speech sample data with the dialect label is updated, uses the speech sample data with the dialect label to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model.

[0178] The performance detection submodule, when detecting that the processing concurrent task amount of the dialect understanding model is less than a preset threshold, uses the speech sample data with the dialect label to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model.

[0179] The instruction detection submodule, when receiving the fine-tuning instruction, uses the speech sample data to train the pre-trained model in response to the fine-tuning instruction, and uses the speech sample data with the dialect label to perform dialect adaptation training on the pre-trained model to obtain a dialect understanding model.

[0180] The convolution layer submodule increases the number of convolution layers of the pre-trained model to obtain a convolution layer for extracting dialect features.

[0181] The circulation layer submodule increases the number of circulation layers of the circulation layer of the pre-trained model to obtain a bidirectional circulation layer.

[0182] The fully connected layer submodule increases the number of fully connected layers of the fully connected layer of the pre-trained model to obtain a fully connected layer that maps dialect features.

[0183] The multimodal data submodule combines the text sample data with the dialect label with the voice sample data with the dialect label to obtain multimodal dialect data containing the voice sample data and the corresponding text sample data.

[0184] The multimodal training submodule uses the multimodal dialect data to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model.

[0185] Optionally, the fusion module 404 may specifically include:

[0186] The first predicted text submodule inputs the speech sample data into the pre-trained model and outputs a first predicted text recognition result of the speech sample data.

[0187] The second predicted text submodule inputs the speech sample data into the dialect understanding model and outputs a second predicted text recognition result.

[0188] The target predicted text submodule fuses the first predicted text recognition result and the second predicted text recognition result to form a target predicted text recognition result of the speech sample data.

[0189] The third predicted text submodule inputs the speech sample data with the Mandarin label, the speech sample data with the dialect label, and the speech sample data associated with the target predicted text recognition result into the initial model to obtain the third predicted text recognition result output by the initial model.

[0190] The first loss value submodule determines a first loss value according to a difference between the third predicted text recognition result and the target predicted text recognition result.

[0191] The second loss value submodule determines a second loss value according to the difference between the third predicted text recognition result, the Mandarin label, and the dialect label.

[0192] The loss function submodule trains the initial model according to the first loss value, the second loss value and a preset loss function to obtain the language understanding model.

[0193] In an example of the present invention, speech sample data is obtained, and a label is generated for each of the speech sample data, and a pre-trained model is trained using the speech sample data to obtain a trained pre-trained model. The speech sample data with the dialect label is used to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model; the pre-trained model and the dialect understanding model are fused to obtain a language understanding model. The present invention generates labels for speech sample data, classifies speech sample data to be trained, and after generating labels for the speech sample data, uses the speech sample data to train the pre-trained model, so that the trained pre-trained model has a basis for speech recognition. Since the speech sample data contains a large number of audio data with Mandarin labels, the recognition accuracy of Mandarin will continue to increase during training, so the pre-trained model has a good understanding ability for Mandarin. After the pre-training model is trained, all the voice sample data with dialect labels are used to perform dialect adaptation training on the pre-training model. Since the voice sample data used are all voice sample data with dialect labels, the training results are close to the dialect training results, so that the pre-trained model has the ability to recognize dialects and obtains a dialect understanding model. Finally, the dialect understanding model and the pre-training model are fused so that the fused language understanding model has the ability to accurately recognize both Mandarin and dialects. By accurately recognizing dialects and Mandarin, it can more accurately match the answers to user questions, improve communication efficiency, and bring better service experience to users.

[0194] Figure 6 A voice processing device for intelligent customer service provided by an embodiment of the present invention is provided. Figure 6 As shown, the device may include:

[0195] The second acquisition module 501 acquires the audio data to be processed input by the user;

[0196] The recognition module 502 is used to input the audio data to be processed into the trained language understanding model to obtain a text recognition result corresponding to the audio data to be processed.

[0197] The question-answering module 503 is used to determine a reply sentence matching the text recognition result based on the text recognition result and a preset knowledge base.

[0198] The reply module 504 is used to send the reply statement to the user's device so that the reply statement serves as a reply to the audio data to be processed.

[0199] In an example of the present invention, speech sample data is obtained, and a label is generated for each of the speech sample data, and a pre-trained model is trained using the speech sample data to obtain a trained pre-trained model. The speech sample data with the dialect label is used to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model; the pre-trained model and the dialect understanding model are fused to obtain a language understanding model. The present invention generates labels for speech sample data, classifies speech sample data to be trained, and after generating labels for the speech sample data, uses the speech sample data to train the pre-trained model, so that the trained pre-trained model has a basis for speech recognition. Since the speech sample data contains a large number of audio data with Mandarin labels, the recognition accuracy of Mandarin will continue to increase during training, so the pre-trained model has a good understanding ability for Mandarin. After the pre-training model is trained, all the voice sample data with dialect labels are used to perform dialect adaptation training on the pre-training model. Since the voice sample data used are all voice sample data with dialect labels, the training results are close to the dialect training results, so that the pre-trained model has the ability to recognize dialects and obtains a dialect understanding model. Finally, the dialect understanding model and the pre-training model are fused so that the fused language understanding model has the ability to accurately recognize both Mandarin and dialects. By accurately recognizing dialects and Mandarin, it can more accurately match the answers to user questions, improve communication efficiency, and bring better service experience to users.

[0200] The present invention also provides an electronic device, see Figure 7 , including: a processor 601, a memory 602, and a computer program 6021 stored in the memory and executable on the processor, and when the processor executes the program, the speech recognition model training method of the aforementioned embodiment is implemented.

[0201] The present invention also provides a readable storage medium, when the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can execute the speech recognition model training method of the aforementioned embodiment.

[0202] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0203] It should be noted that the various information and data obtained in the embodiments of the present invention are all obtained with the authorization of the information / data holder.

[0204] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0205] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0206] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0207] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0208] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the present invention. The present invention may also be implemented as a device or apparatus program for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0209] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0210] The user information (including but not limited to the user's device information, user personal information, etc.) and related data involved in the present invention are all information authorized by the user or by all parties.

[0211] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0212] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0213] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A speech recognition model training method, characterized in that: The method comprises: Acquire speech sample data, and generate a label for each of the speech sample data; the speech sample data includes Mandarin and various dialects, the labels include Mandarin labels and dialect labels, and the dialect labels are used to indicate the type of the dialect; Using the speech sample data to train the pre-trained model to obtain a trained pre-trained model; Using the speech sample data with the dialect label, the pre-trained model is trained for dialect adaptability to obtain a dialect understanding model; the dialect understanding model is used to identify the text recognition result of the dialect speech data, and the pre-trained model is used to identify the text recognition result of the Mandarin speech data; The pre-trained model and the dialect understanding model are subjected to model fusion to obtain a language understanding model; the language understanding model is used to identify text recognition results of dialect speech data and Mandarin speech data.

2. The method according to claim 1, characterized in that Before using the speech sample data to train the pre-trained model to obtain the trained pre-trained model, the method further includes: Performing noise filtering on the speech sample data to obtain noise-filtered speech sample data; Performing audio segmentation on the noise-filtered speech sample data word by word, and obtaining single-word speech data after segmentation; The method of using the speech sample data to train the pre-trained model to obtain a trained pre-trained model includes: After detecting that the audio segmentation is completed, the pre-training model is trained according to the single-word voice data and the label of the voice sample data to which the single-word voice data belongs to obtain a trained pre-training model.

3. The method according to claim 1, characterized in that The method uses the speech sample data with the dialect label to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model, including: When it is detected that the speech sample data with the dialect label is updated, the pre-trained model is trained for dialect adaptability using the speech sample data with the dialect label to obtain a dialect understanding model; When it is detected that the processing concurrent task amount of the dialect understanding model is less than a preset threshold, the pre-trained model is trained for dialect adaptability using speech sample data with the dialect label to obtain a dialect understanding model; When a fine-tuning instruction is received, the pre-trained model is trained using the speech sample data in response to the fine-tuning instruction, and the pre-trained model is dialect-adaptively trained using the speech sample data with the dialect label to obtain a dialect understanding model.

4. The method according to claim 1, characterized in that The pre-trained model includes a convolutional layer, a recurrent layer, and a fully connected layer. In the process of performing dialect adaptation training on the pre-trained model, the method further includes: The number of convolutional layers of the pre-trained model is increased to obtain a convolutional layer for extracting dialect features; when extracting features of speech sample data with dialect labels, the convolutional layer except the convolutional layer with dialect features is frozen to retain common features in the extracted features; Increasing the number of the recurrent layers of the pre-trained model to obtain a bidirectional recurrent layer; The number of fully connected layers of the pre-trained model is increased to obtain a fully connected layer that maps dialect features; the fully connected layer is used to map the dialect category of the speech sample data with the dialect label.

5. The method according to claim 1, characterized in that The step of fusing the pre-trained model and the dialect understanding model to obtain a language understanding model includes: Inputting the speech sample data into the pre-trained model, and outputting a first predicted text recognition result of the speech sample data; Inputting the speech sample data into the dialect understanding model and outputting a second predicted text recognition result; fusing the first predicted text recognition result and the second predicted text recognition result to form a target predicted text recognition result of the speech sample data; The initial model is trained according to the speech sample data with the Mandarin label, the speech sample data with the dialect label, and the speech sample data associated with the target predicted text recognition result to obtain the language understanding model.

6. The method according to claim 5, characterized in that The training of the initial model according to the speech sample data with the Mandarin label, the speech sample data with the dialect label, and the speech sample data associated with the target predicted text recognition result to obtain the language understanding model includes: Inputting the speech sample data with the Mandarin label, the speech sample data with the dialect label, and the speech sample data associated with the target predicted text recognition result into the initial model to obtain a third predicted text recognition result output by the initial model; Determine a first loss value according to a difference between the third predicted text recognition result and the target predicted text recognition result; Determining a second loss value according to a difference between the third predicted text recognition result, the Mandarin label, and the dialect label; The initial model is trained according to the first loss value, the second loss value and a preset loss function to obtain the language understanding model.

7. The method according to claim 1, characterized in that The acquiring of speech sample data and generating a label for each of the speech sample data includes: Performing text transcription according to the acquired voice sample data to obtain a text transcription result; the text transcription result includes text data corresponding to the voice sample data; The text transcription result is combined with the finals and pinyin of the text to obtain text sample data; Generate labels for the text sample data and the speech sample data respectively; the text sample data includes Mandarin and various dialects, the labels include Mandarin labels and dialect labels, and the dialect labels are used to indicate the type of dialect; The method uses the speech sample data with the dialect label to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model, including: Combining the text sample data with the dialect label with the speech sample data with the dialect label to obtain multimodal dialect data containing the speech sample data and the corresponding text sample data; The multimodal dialect data is used to perform dialect adaptability training on the pre-trained model to obtain a dialect understanding model; the dialect understanding model is used to recognize dialect speech data to obtain a text recognition result.

8. A voice processing method for intelligent customer service, characterized in that: include: Get the audio data to be processed input by the user; Inputting the audio data to be processed into the trained language understanding model according to any one of claims 1 to 6 to obtain a text recognition result corresponding to the audio data to be processed; According to the text recognition result and the preset knowledge base, a reply sentence matching the text recognition result is determined; The reply sentence is sent to the device of the user so that the reply sentence serves as a reply to the audio data to be processed.

9. A speech recognition model training device, characterized in that: include: A first acquisition module, used to acquire speech sample data and generate a label for each speech sample data; A pre-training module, used to train a pre-training model using the speech sample data to obtain a trained pre-training model; A dialect training module, used to perform dialect adaptability training on the pre-trained model using speech sample data with the dialect label to obtain a dialect understanding model; The fusion module performs model fusion on the pre-trained model and the dialect understanding model to obtain a language understanding model.

10. A voice processing device for intelligent customer service, characterized in that: include: The second acquisition module acquires the audio data to be processed input by the user; A recognition module, used for inputting the audio data to be processed into the trained language understanding model according to any one of claims 1 to 6 to obtain a text recognition result corresponding to the audio data to be processed; The question-answering module is used to determine a reply sentence matching the text recognition result based on the text recognition result and a preset knowledge base; The reply module is used to send the reply statement to the user's device so that the reply statement serves as a reply to the audio data to be processed.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A readable storage medium, characterized in that: When the instructions in the readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of method claims 1 to 8.

Images