Method and device for generating an intonation word recognition model, method for analyzing audio data

Abstract

Embodiments of the present disclosure disclose a generation method and device of an emotive word recognition model, and an audio data analysis method. A specific implementation of the generation method comprises: inputting sample audio in training sample data into a speech recognition sub-model in the emotive word recognition model to obtain a predicted recognition result of the sample audio, wherein the sample audio is audio data containing emotive words; inputting intermediate layer features output by a target network layer in the speech recognition sub-model into an emotive word intention recognition sub-model in the emotive word recognition model to obtain a predicted intention; and adjusting and training the emotive word recognition model based on the predicted recognition result, the predicted intention, a text label and an intention label corresponding to the sample audio in the training sample data until the training is completed. The implementation is related to speech recognition technology, and the emotive word intention recognition sub-model is constructed by taking the intermediate layer features of the speech recognition sub-model as input, which helps to improve the accuracy of emotive word intention recognition.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of speech recognition technology, specifically to a method and apparatus for generating a modal particle recognition model and an audio data analysis method. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence (AI) technology, using AI to replace human intervention in telephone communication with customers has been widely applied in various business scenarios. During the conversation, the system's ability to correctly recognize and understand the customer's voice is a prerequisite for ensuring a good user experience. Typically, the AI's process of understanding the customer's voice involves two cascading steps: First, Automatic Speech Recognition (ASR) technology identifies the text content within the customer's speech; second, the text recognition result is passed to a downstream Natural Language Processing (NLP) system to identify the customer's intended intent.

[0003] However, the inventors discovered that in real-world voice conversations, users frequently use colloquial interjections to express different intentions. While seemingly simple in Chinese expression, these interjections can convey vastly different intentions due to subtle differences in context and pronunciation. Therefore, the system's performance in recognizing and understanding colloquial interjections is particularly crucial.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide a method for generating a modal particle recognition model, an apparatus for generating a modal particle recognition model, an audio data analysis method, an audio data analysis apparatus, an electronic device, a computer-readable medium, and a computer program product to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a method for generating a modal particle recognition model, comprising: inputting sample audio from training sample data into a speech recognition sub-model of the modal particle recognition model to obtain a predicted recognition result of the sample audio, wherein the sample audio is audio data containing modal particles; inputting intermediate layer features output by the target network layer of the speech recognition sub-model into a modal particle intent recognition sub-model of the modal particle recognition model to obtain a predicted intent, wherein the target network layer is a network layer other than the input layer and the output layer; and adjusting and training the modal particle recognition model based on the predicted recognition result, the predicted intent, and the text labels and intent labels corresponding to the sample audio in the training sample data until training is completed.

[0008] In some embodiments, the speech recognition sub-model employs a pre-trained speech recognition ASR model; and the mood word intent recognition sub-model is further used to perform mood word recognition on the input intermediate layer features and output predicted mood words.

[0009] In some embodiments, the modal particle recognition model is adjusted and trained based on the predicted recognition results, predicted intent, and text labels and intent labels corresponding to the sample audio in the training sample data. This includes: fine-tuning the modal particle intent recognition sub-model according to the predicted modal particle, predicted intent, and text labels and intent labels corresponding to the sample audio.

[0010] In some embodiments, fine-tuning the training of the modal particle intent recognition sub-model includes: using a cross-entropy loss function to determine whether the loss value is less than a threshold; and in response to determining that the loss value is not less than the threshold, using an adaptive gradient optimization algorithm to adjust the relevant parameters in the modal particle intent recognition sub-model to continue model training.

[0011] In some embodiments, the interjection intent recognition sub-model employs a decoding network structure that includes at least one decoding layer.

[0012] In some embodiments, training sample data is obtained by: acquiring audio data representing interjections; performing text annotation based on the interjections represented by the audio data to obtain interjection text labels; performing intent category annotation based on the intent represented by the audio data to obtain intent labels; and generating training sample data based on the audio data, the corresponding interjection text labels, and the intent labels.

[0013] In some embodiments, training sample data is generated based on audio data, corresponding text labels of interjections, and intent labels, including: processing the audio data to obtain audio spectrum features with set attributes; determining the audio spectrum features, corresponding text labels of interjections, and intent labels as labeled data; and dividing the labeled data into training sample data and test sample data according to a set ratio.

[0014] Secondly, some embodiments of this disclosure provide a device for generating a modal particle recognition model, comprising: an audio recognition unit configured to input sample audio from training sample data into a speech recognition sub-model of the modal particle recognition model to obtain a predicted recognition result of the sample audio, wherein the sample audio is audio data containing modal particles; an intent prediction unit configured to input intermediate layer features output by the target network layer in the speech recognition sub-model into a modal particle intent recognition sub-model of the modal particle recognition model to obtain a predicted intent, wherein the target network layer is a network layer other than the input layer and the output layer; and a model adjustment unit configured to adjust and train the modal particle recognition model based on the predicted recognition result, the predicted intent, and the text labels and intent labels corresponding to the sample audio in the training sample data, until training is completed.

[0015] In some embodiments, the speech recognition sub-model employs a pre-trained speech recognition ASR model; and the mood word intent recognition sub-model is further used to perform mood word recognition on the input intermediate layer features and output predicted mood words.

[0016] In some embodiments, the model adjustment unit is further configured to fine-tune the tone word intent recognition sub-model based on the predicted tone word, the predicted intent, and the text labels and intent labels corresponding to the sample audio.

[0017] In some embodiments, the model adjustment unit is further configured to use the cross-entropy loss function to determine whether the loss value is less than a threshold; in response to determining that the loss value is not less than the threshold, to use an adaptive gradient optimization algorithm to adjust the relevant parameters in the interjection intent recognition sub-model to continue model training.

[0018] In some embodiments, the interjection intent recognition sub-model employs a decoding network structure that includes at least one decoding layer.

[0019] In some embodiments, the generating apparatus further includes a training sample data generating unit configured to acquire audio data representing interjections; perform text annotation based on the interjections represented by the audio data to obtain interjection text labels; perform intent category annotation based on the intent represented by the audio data to obtain intent labels; and generate training sample data based on the audio data, the corresponding interjection text labels, and the intent labels.

[0020] In some embodiments, the training sample data generation unit is further configured to process the audio data to obtain audio spectrum features with set attributes; determine the audio spectrum features, corresponding interjection text labels and intent labels as labeled data; and divide the labeled data into training sample data and test sample data according to a set ratio.

[0021] Thirdly, some embodiments of this disclosure provide an audio data analysis method, including: inputting the audio data to be analyzed into a pre-trained speech recognition sub-model of a tone word recognition model to predict the recognition result of the audio data, wherein the tone word recognition model is obtained by the generation method of the tone word recognition model described in any implementation of the first aspect above; in response to determining that the recognition result is represented as a tone word, inputting the intermediate layer features output by the target network layer in the speech recognition sub-model into the tone word intent recognition sub-model of the tone word recognition model to predict the intent of the tone word; and processing the audio data according to the intent of the tone word.

[0022] In some embodiments, processing audio data according to the intent of the interjection includes: generating response information corresponding to the audio data according to the intent of the interjection.

[0023] Fourthly, some embodiments of this disclosure provide an audio data analysis apparatus, comprising: a recognition unit configured to input audio data to be analyzed into a pre-trained speech recognition sub-model of a tone word recognition model, and predict the recognition result of the audio data, wherein the tone word recognition model is obtained by the generation method of the tone word recognition model described in any implementation of the first aspect; an intent recognition unit configured to, in response to determining that the recognition result is represented as a tone word, input the intermediate layer features output by the target network layer in the speech recognition sub-model into the tone word intent recognition sub-model of the tone word recognition model, and predict the intent of the tone word; and a processing unit configured to process the audio data according to the intent of the tone word.

[0024] In some embodiments, the processing unit is further configured to generate response information corresponding to the audio data based on the intent of the interjection.

[0025] Fifthly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the methods described in any of the implementations of the first or third aspects above.

[0026] Sixthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the implementations of the first or third aspect above.

[0027] In a seventh aspect, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the implementations of the first or third aspect above.

[0028] The various embodiments disclosed above have the following beneficial effects: the generation method of the modal particle recognition model in some embodiments of this disclosure can improve the accuracy of the model in recognizing and understanding the intent of modal particles. Specifically, in related modal particle recognition and understanding technologies, text data recognized by an ASR model is usually input into an NLP model to recognize and understand the user's intent. However, for the same modal particle, different pronunciations can lead to completely different meanings. Therefore, even if the user pronounces "um" with different pronunciations and it is correctly recognized as "um," the output text often loses the intonation or semantic information of the modal particle, which causes the downstream NLP model to be unable to accurately determine the user's true intent.

[0029] Based on this, the generation method of the interjection recognition model in some embodiments of this disclosure can use the intermediate layer features output by the target network layer as input data when the speech recognition sub-model performs recognition and analysis on sample audio. This input data is then fed into the interjection intent recognition sub-model to identify the intent expressed by interjections in the sample audio. Compared with traditional text input data, the intermediate layer features not only contain the textual information expressed in the audio but also often include tone or semantic information. This can improve the accuracy of intent prediction. Furthermore, it can facilitate smoother dialogue and communication between artificial intelligence and users, improve user experience, and thus increase the number of users using artificial intelligence in dialogue scenarios. Attached Figure Description

[0030] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0031] Figure 1 This is a flowchart of some embodiments of the generation method of the modal particle recognition model disclosed herein;

[0032] Figure 2A This is a schematic diagram of the structure of some embodiments of the modal particle recognition model disclosed herein;

[0033] Figure 2B These are schematic diagrams illustrating the structure of some embodiments of the generation apparatus for the modal particle recognition model disclosed herein;

[0034] Figure 3 This is a flowchart of some embodiments of the audio data analysis method disclosed herein;

[0035] Figure 4A Here is a schematic diagram illustrating some of the operational logic of this publicly available modal particle recognition model:

[0036] Figure 4BThis is a schematic diagram of the structure of some embodiments of the audio data analysis device disclosed herein;

[0037] Figure 5 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0038] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0039] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0040] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0041] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0042] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0043] Figure 1 A flow 100 of some embodiments of a generation method based on the modal particle recognition model of this disclosure is shown. The generation method includes the following steps:

[0044] Step 101: Input the sample audio from the training sample data into the speech recognition sub-model of the interjection recognition model to obtain the predicted recognition result of the sample audio.

[0045] In some embodiments, the execution entity (e.g., a server) of the method for generating the interjection recognition model can acquire or receive training sample data via a wired or wireless connection. The training sample data may include sample audio, text labels for the sample audio, and intent labels. The format of the sample audio is not limited, as long as it can be read and recognized by the execution entity. The sample audio is typically audio data containing interjections, such as a recording of a sentence containing interjections, or audio containing only or representing interjections. The interjections can be set according to the actual situation, and are usually colloquial interjections, such as "um," "ah," "oh," etc. The text labels can be text data representing the content of the sample audio, such as the entire text content of the sample audio, or the text content of a specific audio portion of the sample audio. For example, since this application is mainly for interjection recognition, the text labels here can also be text data representing interjections. Additionally, the intent labels can represent the intent information expressed by the interjections in the sample audio.

[0046] Here, after receiving the training sample data, the executing entity can input the sample audio into the speech recognition sub-model within the interjection recognition model to obtain the predicted recognition result of the sample audio. The speech recognition sub-model is used to recognize the input audio data to output the text recognition result. This speech recognition sub-model typically employs Automatic Speech Recognition (ASR) technology. ASR is generally a technique that converts human speech into text.

[0047] Understandably, with the rapid development of artificial intelligence technology, using AI to replace human intervention in telephone communication with customers has become widely adopted across various industries. The Automatic Speech Recognition (ASR) model is commonly used in intelligent call scenarios, indicating that this model technology is relatively mature and well-developed. Therefore, to improve the model's generation efficiency, the speech recognition sub-model here can employ an ASR pre-trained model. That is, the parameters from a pre-trained ASR model are used to construct the speech recognition sub-model in this embodiment. This ensures the model's recognition accuracy while reducing the number of training iterations, thereby improving the model's generation efficiency. Furthermore, using a pre-trained model also helps reduce the amount of training data required.

[0048] Step 102: Input the intermediate layer features output by the target network layer in the speech recognition sub-model into the tone word intent recognition sub-model in the tone word recognition model to obtain the predicted intent.

[0049] In some embodiments, during the processing of sample audio by the speech recognition sub-model, the executing entity can also cache the intermediate layer features output by the target network layer in the speech recognition sub-model. The target network layer is typically a network layer other than the input and output layers. Furthermore, the executing entity can input these intermediate layer features into the intent recognition sub-model within the intent recognition model to obtain the predicted intent. The intent recognition sub-model can be used to recognize the intent of intent words in the input data.

[0050] Here, the specific target network layer can be set according to the actual situation. As an example, the output of the K / 2th layer can be used as the intermediate layer feature. K is the total number of layers in the speech recognition sub-model. It should be noted that using K / 2 layers (using the floor function) can balance efficiency and performance. If this value is too large, there will be too many network layers, which will slow down the speed. If this value is too small, fewer network layers are used, which can improve speed, but because too few input parameters are used, the ASR model cannot be fully utilized, which generally leads to a performance degradation.

[0051] In some embodiments, the above-described modal particle intent recognition sub-model can employ various machine learning model structures. As an example, the modal particle intent recognition sub-model can adopt a Natural Language Processing (NLP) model structure. Since the input to the NLP model (i.e., the modal particle intent recognition sub-model) is the intermediate layer features of the ASR model, the NLP model can remove the traditional input layer or reduce the encoding layer of the input data, and directly analyze and process the intermediate layer features. This simplifies the structure and processing of the modal particle intent recognition sub-model, and is beneficial for further improving the model's processing efficiency.

[0052] Optionally, the intent recognition sub-model for interjections can also employ a decoding network structure that includes at least one decoding layer. For example, to balance performance and inference efficiency, the intent recognition sub-model for interjections can use a decoding network structure with only one Transformer layer to perform text and intent recognition of interjections. Alternatively, a network structure based on attention mechanism + Long Short-Term Memory (LSTM) network can also be used.

[0053] It should be noted that in related interjection recognition and understanding technologies, usually, the text data obtained by the ASR model through recognition is input into the NLP model to recognize and understand the user's intention. Although NLP models have achieved great improvements in the accuracy of text understanding, in a dialogue scenario, the same interjection may express very different customer intentions due to differences in pronunciation and context. For example, the utterance of the pronunciation "en" by a customer may express affirmation, hesitation, or a non-intentional response. Since ASR models usually use Chinese characters as modeling units, they cannot identify richer information such as intonation. Therefore, when only the interjection text correctly recognized by the ASR model is used as the input of the NLP model, it is usually still difficult to accurately determine the user's real intention.

[0054] In the embodiments of the present disclosure, intermediate layer features are used as the input data of the interjection intention recognition sub-model. Compared with traditional text input data, the intermediate layer features not only include the text information expressed by the audio, but often also include information such as intonation or semantics, which can improve the accuracy of intention prediction.

[0055] Step 103: Adjust and train the interjection recognition model based on the prediction recognition result, the predicted intention, and the text label and intention label corresponding to the sample audio in the training sample data until the training is completed.

[0056] In some embodiments, based on the above prediction recognition result, predicted intention, and the text label and intention label corresponding to the sample audio in the training sample data, the execution subject may adjust the relevant parameters in the interjection recognition model to continue the model training until the training is completed. As an example, the execution subject may perform adjustment and training of the speech recognition sub-model according to the comparison result between the prediction recognition result and the corresponding text label, and at the same time, perform adjustment and training of the interjection intention recognition sub-model according to the comparison result between the predicted intention and the corresponding intention label.

[0057] It should be noted that in the related art, in order to improve the recognition and understanding effect of the traditional ASR and NLP cascading system on interjections, a common direct method is to use a large amount of interjection annotation data to separately train the ASR and NLP models in the intelligent dialogue system. However, separately optimizing the two models of ASR and NLP leads to a relatively complicated training process. Moreover, since existing models are often unexplainable, overfitting may occur after optimization, resulting in unknown impacts on the recognition and understanding effects of non-interjections.

[0058] In some embodiments, if the speech recognition sub-model is a pre-trained model, it can be fixed during model optimization. Additionally, the aforementioned tone-of-voice intent recognition sub-model can also be used to recognize tone-of-voices from the input intermediate layer features, outputting predicted tone-of-voices. In this case, the tone-of-voice intent recognition sub-model can be fine-tuned based on the predicted tone-of-voices, predicted intent, and the text labels (tone-of-voice labels) and intent labels corresponding to the sample audio. This ensures the overall accuracy of the tone-of-voice recognition model while reducing overfitting in the speech recognition sub-model. Furthermore, fine-tuning only the tone-of-voice intent recognition sub-model simplifies the overall model training process and improves the overall optimization efficiency.

[0059] Specifically, the execution entity can use the cross-entropy loss function to determine whether the loss value is less than a threshold. If the loss value is determined to be not less than the threshold, the Adaptive Moment Estimation (ADAM) optimization algorithm can be used to adjust the relevant parameters in the interjection intent recognition sub-model to continue model training until the model converges. In other words, during the model parameter update process, gradient backpropagation can employ the ADAM optimization algorithm. ADAM can use a warm-up strategy, with a peak learning rate of 0.001 and a warm-up strategy step size of 1000 steps.

[0060] Furthermore, optimizing traditional models often requires a large amount of manually labeled data, leading to high optimization costs. To address this issue, this embodiment of the present disclosure uses intermediate layer features of the model network as input to construct an attention-based network for interjection recognition and intent classification tasks, building upon the ASR pre-trained model. This reduces the model's dependence on the amount of labeled data, thereby lowering data labeling costs.

[0061] As described above, the generation method of the interjection recognition model in some embodiments of this disclosure can use the intermediate layer features output by the target network layer as input data when the speech recognition sub-model performs recognition and analysis on sample audio. This input data is then fed into the interjection intent recognition sub-model to identify the intent expressed by interjections in the sample audio. Compared to traditional text input data, the intermediate layer features not only contain the textual information expressed in the audio but also often include tone or semantic information. This can improve the accuracy of intent prediction.

[0062] As an example, such as Figure 2A The diagram shows a speech recognition model for interjections. The shaded area represents the ASR pre-trained model, a sub-model for speech recognition. It can be divided into a shallow feature extraction network and a deep network. Here, for a given sample audio input data x... i On the one hand, through the shallow ASR network, the intermediate layer features e can be calculated.i Then, through the ASR deep network, the ASR recognition result can be calculated. On the other hand, the intermediate layer features e i As input, the modal particle decoding network (i.e., the modal particle intent recognition sub-model) can output the recognition results of the text and intent, that is...

[0063] p(y i |x i )=∏ u f(y i,u |e i ,y i,1:u-1 );

[0064] Among them, y i,u y is the label i The u-th element (including the start character) <sos>and end character <eos>);y i,1:u-1 the first u-1 elements of the labeled y; f represents the filler decoding network; C in CI represents the text label, and I represents the intent label. i

[0065] In some embodiments, the training sample data of the embodiments of the present disclosure can be obtained through the following method: first, obtain audio data representing a filler; then, perform text labeling according to the filler represented by the audio data to obtain a filler text label; next, perform intent category labeling according to the intent represented by the audio data to obtain an intent label; after that, generate training sample data based on the audio data, the corresponding filler text label and the intent label.

[0066] It can be understood that the embodiments of the present disclosure are mainly optimized for filler recognition. Therefore, in order to improve the collection efficiency of filler audio data, audios whose recognition results by the online ASR model are fillers can be collected and manually labeled to obtain text and intent category labels. In addition, by means of sampling, filler audio samples that are missed in recognition, for example, a user utters a filler but the ASR model recognizes it as a non-filler, can be used as supplementary data.

[0067] Further, the execution subject may also process the audio data to obtain audio spectrum features with set attributes. Next, the audio spectrum features, the corresponding filler text label and the intent label are determined as labeled data. After that, the labeled data is divided into training sample data and test sample data according to a set ratio.

[0068] A labeled dataset can be obtained in the above manner, denoted as D={x i ,p i |i∈[1,N]}. Wherein N is the number of samples in the training sample dataset; x i is the audio feature matrix of the i-th training sample data in the corresponding dataset; y i is the text-intent sequence labeling result corresponding to the audio feature x i . The audio spectrum features with the above set attributes can be set according to actual situations. For example, it can be an 80-dimensional Mel-spectrum feature, where each frame has a duration of 25 milliseconds and a step size of 10 milliseconds. The set ratio is also not limited, for example, it can be 9:1, that is, 90% of the data is training sample data and 10% is test sample data. The test sample data can be used to test the trained model. Wherein, the text label C can be divided into a total of 8 categories: 7 types of fillers including "hmm", "ah", "oh", "uh", "hello", "um", "ang", and 1 blank label. The intent label I can be divided into a total of 3 categories: "affirmative", "hesitant", and "no intent".

[0069] Reference is made below to​ Figure 2B As a response to the above Figure 1 The present disclosure provides some embodiments of a device for generating a modal particle recognition model, which is used to implement the generation method of the modal particle recognition model. These device embodiments are similar to... Figure 1 The examples of the generation methods shown correspond to those in the illustration. This generation and analysis apparatus can be specifically applied to various electronic devices.

[0070] like Figure 2B As shown, the generation device 200 for the tone word recognition model in some embodiments may include: an audio recognition unit 201, configured to input sample audio from the training sample data into the speech recognition sub-model of the tone word recognition model to obtain a predicted recognition result of the sample audio, wherein the sample audio is audio data containing tone words; an intent prediction unit 202, configured to input the intermediate layer features output by the target network layer in the speech recognition sub-model into the tone word intent recognition sub-model of the tone word recognition model to obtain a predicted intent, wherein the target network layer is a network layer other than the input layer and the output layer; and a model adjustment unit 203, configured to adjust and train the tone word recognition model based on the predicted recognition result, the predicted intent, and the text labels and intent labels corresponding to the sample audio in the training sample data until training is completed.

[0071] In some embodiments, the speech recognition sub-model employs a pre-trained speech recognition ASR model; and the mood word intent recognition sub-model is further used to perform mood word recognition on the input intermediate layer features and output predicted mood words.

[0072] In some embodiments, the model adjustment unit 203 may be further configured to fine-tune the tone word intent recognition sub-model based on the predicted tone word, the predicted intent, and the text labels and intent labels corresponding to the sample audio.

[0073] In some embodiments, the model adjustment unit 203 may be further configured to use the cross-entropy loss function to determine whether the loss value is less than a threshold; in response to determining that the loss value is not less than the threshold, to use an adaptive gradient optimization algorithm to adjust the relevant parameters in the interjection intent recognition sub-model to continue model training.

[0074] In some embodiments, the interjection intent recognition sub-model employs a decoding network structure that includes at least one decoding layer.

[0075] In some embodiments, the generation apparatus 200 may further include a training sample data generation unit (not shown in the figure), configured to acquire audio data representing interjections; perform text annotation based on the interjections represented by the audio data to obtain interjection text labels; perform intent category annotation based on the intent represented by the audio data to obtain intent labels; and generate training sample data based on the audio data, the corresponding interjection text labels, and the intent labels.

[0076] In some embodiments, the training sample data generation unit may be further configured to process the audio data to obtain audio spectrum features with set attributes; determine the audio spectrum features, corresponding interjection text labels and intent labels as labeled data; and divide the labeled data into training sample data and test sample data according to a set ratio.

[0077] It is understandable that the units described in the generating apparatus 200 are related to the reference. Figure 1 The steps in the described generation method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the generation device 200 of the modal particle recognition model and the units contained therein, and will not be repeated here.

[0078] Continue to refer to Figure 3 The diagram illustrates a flow 300 of some embodiments of the audio data analysis method according to this disclosure. The method includes the following steps:

[0079] Step 301: Input the audio data to be analyzed into the speech recognition sub-model in the pre-trained interjection recognition model to predict the recognition result of the audio data.

[0080] In some embodiments, the entity executing the audio data analysis method (may be related to...) Figure 1 (Whether the executing entities in the embodiments are the same or different) can receive the audio data to be analyzed via wired or wireless connection. The audio data here can be the audio data from the aforementioned test sample data, or it can be audio data obtained in real time based on actual voice dialogue scenarios. For example, the executing entity can convert the received dialogue audio into audio spectrum features to serve as the audio data to be analyzed.

[0081] Here, the executing entity can input the audio data to be analyzed into the speech recognition sub-model of the pre-trained tone word recognition model, thereby predicting the recognition result of the audio data. The tone word recognition model can employ the methods described above. Figure 1 In the embodiments, the generation method of the modal particle recognition model described in any implementation is obtained.

[0082] Step 302, in response to determining that the recognition result indicates a modal particle, input the intermediate layer features output by the target network layer in the speech recognition sub-model into the modal particle intention recognition sub-model in the modal particle recognition model, to predict the intention of the modal particle.

[0083] In some embodiments, the recognition result of the audio data is obtained according to step 301. If the recognition result indicates a modal particle, the intermediate layer features output by the target network layer in the speech recognition sub-model may be input into the modal particle intention recognition sub-model in the modal particle recognition model, so as to predict the intention of the modal particle.

[0084] As an example, such as Figure 4A shown in the figure, the execution subject may input the audio spectrum features into the ASR model. Intermediate layer features can be calculated and obtained through the shallow network of the ASR model. Then, an ASR recognition result can be calculated and obtained through the deep network of the ASR model. At this time, it can be judged whether the output result of the ASR model is a modal particle. If and only if the ASR recognition result is a modal particle, the modal particle decoding network is triggered for calculation, and the modal particle recognition result and the intention recognition result are output as the final result of the model.

[0085] It can be seen from the above description that the method in the embodiments of the present disclosure will not affect the audio recognition of non-modal particles. At the same time, when the model recognizes a modal particle, it only uses the modal particle decoding network with a small parameter amount for calculation, which effectively limits the generation of additional inference delay.

[0086] Step 303, processing the audio data according to the intention of the modal particle.

[0087] In some embodiments, according to the intention of the modal particle obtained in step 302, the execution subject may process the audio data. For example, the recognized text of the audio is supplemented or corrected according to the intention represented by the modal particle, so as to output the complete and real text data of the audio data.

[0088] Optionally, the execution subject may also generate reply information corresponding to the audio data according to the intention of the modal particle. For example, in an actual dialogue scenario, the subsequent dialogue content may be determined according to the intention expressed by the modal particle spoken by the user, so as to generate reply information and send it to the user.

[0089] It should be noted that the experimental test shown in the following table, that is, using test sample data to compare the performance of different methods, can demonstrate that the modal particle recognition model in the embodiments of the present disclosure can significantly improve the recognition accuracy of modal particles, especially the recognition precision rate of the most commonly used modal particle "hmm", and can directly output the intention recognition result.

[0090]

[0091] It is understood that the method for producing the modal particle recognition model in this disclosure improves the accuracy of the model's recognition and understanding of modal particles by utilizing a pre-trained model with a small amount of manually labeled data. Compared with existing technologies, the method proposed in this invention has three main advantages:

[0092] (1) This invention optimizes the shallow network of the pre-trained ASR model, which can improve the final recognition performance by utilizing the representational ability of the pre-trained model, while reducing the cost of manual data annotation for model optimization.

[0093] (2) Based on the ASR pre-trained model, a new modal word classification and understanding module is inserted, which can optimize the ASR model independently and avoids affecting the online non-modal word recognition performance.

[0094] (3) It can output both interjection and intent recognition results simultaneously to assist downstream tasks in making dialogue process decisions.

[0095] Further reference Figure 4B As a response to the above Figure 3 The present disclosure provides some embodiments of an audio data analysis apparatus to implement the audio data analysis method shown. These audio data analysis apparatus embodiments are similar to... Figure 3 The methods and embodiments shown correspond to those described. This audio data analysis device can be specifically applied to various electronic devices.

[0096] like Figure 4B As shown, the audio data analysis apparatus 400 in some embodiments may include: a recognition unit 401, configured to input the audio data to be analyzed into a pre-trained speech recognition sub-model of a tone word recognition model, and predict the recognition result of the audio data, wherein the tone word recognition model is obtained by the generation method of the tone word recognition model described in any implementation of the first aspect above; an intent recognition unit 402, configured to, in response to determining that the recognition result is represented as a tone word, input the intermediate layer features output by the target network layer in the speech recognition sub-model into the tone word intent recognition sub-model of the tone word recognition model, and predict the intent of the tone word; and a processing unit 403, configured to process the audio data according to the intent of the tone word.

[0097] In some embodiments, the processing unit 403 may be further configured to generate response information corresponding to the audio data based on the intent of the interjection.

[0098] It is understandable that the units described in the audio data analysis device 400 are related to the reference. Figure 3 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the audio data analysis device 400 and the units contained therein, and will not be repeated here.

[0099] The following is for reference. Figure 5 It shows a schematic diagram of the structure of an electronic device 500 suitable for implementing some embodiments of the present disclosure. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0100] like Figure 5 As shown, the electronic device 500 may include a processing unit 501 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0101] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, speakers, vibrators, etc.; storage devices 508 including, for example, disks, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 5 Each box shown can represent a device or multiple devices as needed.

[0102] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 509, or installed from storage device 508, or installed from ROM 502. When the computer program is executed by processing device 501, it performs the functions defined in the methods of some embodiments of this disclosure.

[0103] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0104] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0105] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: input sample audio from the training sample data into the speech recognition sub-model of the interjection recognition model to obtain a predicted recognition result for the sample audio, wherein the sample audio is audio data containing interjections; input the intermediate layer features output from the target network layer of the speech recognition sub-model into the interjection intent recognition sub-model of the interjection recognition model to obtain a predicted intent, wherein the target network layer is a network layer other than the input and output layers; and adjust and train the interjection recognition model based on the predicted recognition result, the predicted intent, and the text labels and intent labels corresponding to the sample audio in the training sample data until training is complete.

[0106] Alternatively, the electronic device may: input the audio data to be analyzed into a pre-trained speech recognition sub-model of a tone word recognition model to predict the recognition result of the audio data, wherein the tone word recognition model is obtained by the generation method of the tone word recognition model described in any of the implementations of the first aspect above; in response to determining that the recognition result is represented as a tone word, input the intermediate layer features output by the target network layer in the speech recognition sub-model into the tone word intent recognition sub-model of the tone word recognition model to predict the intent of the tone word; and process the audio data according to the intent of the tone word.

[0107] Furthermore, computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0109] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an audio recognition unit, an intent prediction unit, and a model adjustment unit; or a recognition unit, an intent recognition unit, and a processing unit. The names of these units do not necessarily limit the unit itself; for example, an audio recognition unit may also be described as "a unit that obtains the predicted recognition result of sample audio."

[0110] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0111] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the above-described methods for generating a modal particle recognition model or an audio data analysis method.

[0112] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.< / eos> < / sos>

Claims

1. A method for generating a modal particle recognition model, comprising: The sample audio from the training sample data is input into the speech recognition sub-model of the modal particle recognition model to obtain the predicted recognition result of the sample audio, wherein the sample audio is audio data containing modal particles. The intermediate layer features output by the target network layer in the speech recognition sub-model are input into the tone word intent recognition sub-model in the tone word recognition model to obtain the predicted intent. The target network layer is a network layer other than the input layer and the output layer. Based on the predicted recognition results, the predicted intent, and the text labels and intent labels in the training sample data corresponding to the sample audio, the tone word recognition model is adjusted and trained until training is complete.

2. The generation method according to claim 1, wherein, The speech recognition sub-model employs an ASR (Automatic Speech Recognition) pre-trained model; and The modal particle intent recognition sub-model is also used to perform modal particle recognition on the input intermediate layer features and output predicted modal particles.

3. The generation method according to claim 2, wherein, The step of adjusting and training the interjection recognition model based on the predicted recognition result, the predicted intent, and the text labels and intent labels in the training sample data corresponding to the sample audio includes: The tone word intent recognition sub-model is fine-tuned based on the predicted tone word, the predicted intent, and the text and intent labels corresponding to the sample audio.

4. The generation method according to claim 3, wherein, The fine-tuning training of the modal particle intent recognition sub-model includes: Use the cross-entropy loss function to determine whether the loss value is less than the threshold; In response to the determination that the loss value is not less than the threshold, the relevant parameters in the mood word intent recognition sub-model are adjusted using an adaptive gradient optimization algorithm to continue model training.

5. The generation method according to claim 1, wherein, The modal intent recognition sub-model adopts a decoding network structure that includes at least one decoding layer.

6. The generation method according to any one of claims 1-5, wherein, The training sample data was obtained through the following method: Obtain audio data representing modal particles; Based on the modal particles represented by the audio data, text annotation is performed to obtain modal particle text tags; Based on the intent represented by the audio data, intent category labeling is performed to obtain intent tags; Training sample data is generated based on the audio data, the corresponding text labels of interjections, and the intent labels.

7. The generation method according to claim 6, wherein, The step of generating training sample data based on the audio data, corresponding interjection text tags, and intent tags includes: The audio data is processed to obtain audio spectral features with defined attributes; The audio spectral features, corresponding interjection text labels, and intent labels are determined as annotation data; According to a set ratio, the labeled data is divided into training sample data and test sample data.

8. A device for generating a modal particle recognition model, comprising: The audio recognition unit is configured to input sample audio from the training sample data into the speech recognition sub-model of the modal particle recognition model to obtain the predicted recognition result of the sample audio, wherein the sample audio is audio data containing modal particles. The intent prediction unit is configured to input the intermediate layer features output by the target network layer in the speech recognition sub-model into the intent recognition sub-model of the tone word recognition model to obtain the predicted intent, wherein the target network layer is a network layer other than the input layer and the output layer. The model adjustment unit is configured to adjust and train the interjection recognition model based on the predicted recognition result, the predicted intent, and the text labels and intent labels in the training sample data corresponding to the sample audio, until the training is completed.

9. An audio data analysis method, comprising: The audio data to be analyzed is input into the speech recognition sub-model of the pre-trained tone word recognition model to predict the recognition result of the audio data. The tone word recognition model is obtained by the tone word recognition model generation method as described in any one of claims 1-7. In response to determining that the recognition result is represented as an interjection, the intermediate layer features output by the target network layer in the speech recognition sub-model are input into the interjection intent recognition sub-model in the interjection recognition model to predict the intent of the interjection; The audio data is processed according to the intent of the interjection.

10. The audio data analysis method according to claim 9, wherein, The step of processing the audio data according to the intent of the interjection includes: Based on the intent of the interjection, a response message corresponding to the audio data is generated.

11. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7 or 9-10.

12. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7 or 9-10.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7 or 9-10.

Images