Conversation intention recognition method, device and electronic device based on prompt learning

Abstract

The present invention discloses a method, device, electronic device and computer-readable medium for conversation intention recognition based on prompt learning. The method performs speech recognition and audio latent feature extraction on conversation speech to obtain conversation text and audio latent features corresponding to the conversation speech; and establishes prompts based on the correlation between the conversation text and the audio latent features; and trains a machine learning model using the conversation text, audio latent features, prompts and corresponding intentions of historical conversation speech, and uses the machine learning model to process the conversation text, audio latent features and prompts of new conversation speech to identify the conversation intention of the new conversation speech. Since the present invention adopts prompt learning and self-supervised training based on a pre-trained model, and introduces the audio latent features of conversation speech into intent recognition, the present invention can improve the accuracy of intent recognition.

Description

Technical Field

[0001] The present invention relates to the field of natural language understanding, and in particular to a conversation intention recognition method, device, electronic device and computer-readable medium based on prompt learning. Background Art

[0002] With the emergence of models such as word2vec, GPT, and BERT, the research and application of NLP (natural language processing) technology are increasingly centered on pre-trained language models. The prior art has proposed to implement conversational intent recognition by establishing a pre-trained model. When using a pre-trained model for intent recognition, the conversation text is usually input into the pre-trained model to generate text latent features, and then the text latent features are input into the intent recognition model to obtain the conversation intent.

[0003] Figure 1 is a flow chart of an existing intent recognition method. Figure 1 As shown in the figure, the audio is recognized into text by automatic speech recognition (ASR) technology, and the text is extracted through a pre-trained model to extract latent features. At this time, the latent features contain complex abstract features of the text, which are called text latent features. The text latent features enter the intent recognition model and output the intent prediction results. It can be seen that the existing methods only use text information.

[0004] However, since the pre-trained model is trained with a large amount of text data and has never used other types of features, when the text is wrong or inaccurate, the text hidden features output by the pre-trained model cannot accurately reflect the true intention of the speaker, which reduces the recognition accuracy of the conversation intention. How to obtain an audio decoder so that the audio hidden features it outputs can be understood and accepted by the pre-trained model is a problem that needs to be solved urgently. Summary of the invention

[0005] The present invention aims to solve the problem that the accuracy of intent recognition is low due to the existing conversation recognition method which is simply based on a pre-trained model of conversation text.

[0006] In order to solve the above technical problems, the first aspect of the present invention proposes a conversation intention recognition method based on prompt learning, including: performing speech recognition and audio latent feature extraction on conversational speech to obtain conversation text and audio latent features corresponding to the conversational speech; establishing prompts according to the correlation between the conversation text and the audio latent features; training a machine learning model using the conversation text, audio latent features, prompts and corresponding intentions of historical conversational speech, and using the machine learning model to process the conversation text, audio latent features and prompts of new conversational speech to identify the conversation intention of the new conversational speech.

[0007] According to a preferred embodiment of the present invention, the audio latent features of the conversational speech are extracted by using an audio encoder based on a neural network algorithm.

[0008] According to a preferred embodiment of the present invention, the machine learning model for intent recognition includes the audio encoder, and also includes a pre-trained model and an intent recognition model; the pre-trained model is used to generate text latent features for intent recognition based on conversation text, audio latent features, and prompts; the intent recognition model is used to generate conversation intent based on the text latent features.

[0009] According to a preferred embodiment of the present invention, the method of training a machine learning model using conversation text, audio latent features, prompts and corresponding intentions of historical conversation speech includes: establishing a training data set including conversation text, audio latent features, prompts and intentions; keeping the pre-trained model fixed, and using the conversation text, audio latent features and prompts in the training data set to perform self-supervised prompt learning to train the audio encoder.

[0010] According to a preferred embodiment of the present invention, the method of training a machine learning model using conversation text, audio latent features, prompts and corresponding intents of historical conversation speech includes: keeping the audio encoder and pre-trained model fixed, and training the intent recognition model using a training data set.

[0011] According to a preferred embodiment of the present invention, the step of establishing a training data set includes: performing speech recognition and audio latent feature extraction on historical conversation speech to obtain conversation text and audio latent features corresponding to the historical conversation speech; and establishing prompts based on the correlation between the text and audio latent features of the historical conversation.

[0012] According to a preferred embodiment of the present invention, the prompt includes any one of "equivalent to", "meaning is", "expressing means", "equivalent to" or a synonym thereof.

[0013] In order to solve the above technical problems, the second aspect of the present invention proposes a conversation intention recognition device, which includes: an extraction module, which is used to perform speech recognition and audio latent feature extraction on conversational speech to obtain conversational text and audio latent features corresponding to the conversational speech; a prompt generation module, which is used to establish prompts according to the correlation between the conversational text and the audio latent features; an intention recognition module, which is used to train a machine learning model using the conversational text, audio latent features, prompts and corresponding intentions of historical conversational speech, and use the machine learning model to process the conversational text, audio latent features and prompts of new conversational speech to identify the conversational intention of the new conversational speech.

[0014] According to a preferred embodiment of the present invention, the machine learning model for intent recognition includes the audio encoder, and also includes a pre-trained model and an intent recognition model; the pre-trained model is used to generate text latent features for intent recognition based on conversation text, audio latent features, and prompts; the intent recognition model is used to generate conversation intent based on the text latent features.

[0015] According to a preferred embodiment of the present invention, a data set establishment submodule is used to establish a training data set including conversation text, audio latent features, prompts and intentions; an audio encoder training submodule; the pre-trained model is kept fixed, and the conversation text, audio latent features and prompts in the training data set are used to perform self-supervised prompt learning to train the audio encoder.

[0016] According to a preferred embodiment of the present invention, the intention recognition module model training submodule is used to keep the audio encoder and the pre-trained model fixed, and use the training data set to train the intention recognition model.

[0017] In order to solve the above technical problem, the third aspect of the present invention proposes an electronic device, which includes a processor and a memory storing computer-executable instructions, and the computer-executable instructions enable the processor to execute the above method when executed.

[0018] In order to solve the above technical problem, the fourth aspect of the present invention proposes a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by a processor, the above method is implemented.

[0019] Since the present invention adopts prompt learning and self-supervised training based on a pre-training model and introduces the audio latent features of conversational speech into intent recognition, the present invention can improve the accuracy of intent recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to make the technical problems solved by the present invention, the technical means adopted and the technical effects achieved more clearly, the specific embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, it should be noted that the drawings described below are only drawings of exemplary embodiments of the present invention, and those skilled in the art can obtain drawings of other embodiments based on these drawings without creative work.

[0021] Figure 1 is a flow chart of an existing intent recognition method;

[0022] Figure 2 It is a flow chart of the prompt-based learning intention recognition method of the present invention;

[0023] Figure 3is a training flow chart of an audio encoder of the present invention;

[0024] Figure 4 is a flowchart of a specific embodiment of the prompt-based learning intention recognition method of the present invention;

[0025] Figure 5 is a flowchart of a method for training a machine learning model according to an embodiment of the present invention;

[0026] Figure 6 It is a module schematic diagram of a conversation intention recognition device based on prompt learning according to an embodiment of the present invention;

[0027] Figure 7 is a submodule composition diagram of an intention recognition module of a specific embodiment of the present invention;

[0028] Figure 8 is a structural block diagram of an exemplary embodiment of an electronic device according to the present invention;

[0029] Fig. 9 is a schematic diagram of one embodiment of a computer readable medium of the present invention. DETAILED DESCRIPTION

[0030] The exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. Although each exemplary embodiment can be implemented in a variety of specific ways, it should not be understood that the present invention is limited to the embodiments set forth herein. On the contrary, these exemplary embodiments are provided to make the content of the present invention more complete and to facilitate the comprehensive communication of the inventive concept to those skilled in the art.

[0031] Under the premise of conforming to the technical concept of the present invention, the structure, performance, effect or other characteristics described in a specific embodiment may be combined with one or more other embodiments in any appropriate manner.

[0032] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0033] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to the actual situation without departing from the main purpose of the present invention.

[0034] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0035] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0036] As mentioned above, the conversation recognition method used in traditional human-computer interaction is based on pure text. The deviation between the conversation speech and the recognized text will greatly affect the accuracy of intent recognition. In order to solve this problem, the present invention proposes to further use the audio information of the conversation speech to make up for the deficiency of pure text information and enhance the accuracy of intent recognition.

[0037] In order to introduce the audio information of the conversational speech, the present invention uses an audio encoder to process the conversational speech to obtain the latent features of the conversational speech. The so-called "hidden features" refer to the latent features that cannot be directly obtained, which are relative to the explicit features that can be directly obtained. For example, pauses are the explicit features of the prosodic word boundaries. It can be directly obtained that there are pauses at the prosodic word boundaries, but the relationship between the prosodic word boundaries and the acoustic features cannot be directly obtained, which is the latent feature of the prosodic word boundaries. Other audio latent features include logarithmic Mel-spectrograms, local binary patterns, gradient histograms and other features.

[0038] The present invention is not limited to a specific audio encoder for generating audio latent features, but preferably uses an audio encoder based on a neural network. For example, an audio encoder based on a convolutional neural network can be used. When implemented in this way, the sound signal can first be preprocessed and discrete Fourier transformed, the amplitude spectrum of the sound signal can be calculated, and a two-dimensional spectrogram signal can be formed. The output obtained by the trained convolutional neural network calculation is the audio latent feature to be generated.

[0039] During the intention recognition, after the conversational speech is obtained, it is necessary to convert the conversational speech into text. This conversion process is often referred to as ASR technology, i.e., Automatic Speech Recognition. There are various mature ASR technologies in the prior art, which can be directly adopted by the present invention, so the ASR technology will not be described in detail here.

[0040] As mentioned above, after obtaining the text of the conversational speech through ASR, the text is not always very accurate and may have certain errors, thereby causing deviations in the intent recognition based on pure text. To this end, the present invention proposes a method of combining audio latent features with text, using a pre-trained model and combining self-supervised prompt learning.

[0041] The so-called pre-trained model in the present invention refers to a pre-trained language model (PLMs, Pre-trained Language Models), which is a model that obtains complex and rich text information by collecting a large amount of text data for self-supervised learning. The present invention is not limited to the specific pre-training models adopted, such as BERT (Bidirectional Encoder Representations from Transformers), GPT-2 (Gererate Pre-Training Model), etc. The so-called self-supervised learning mainly uses auxiliary tasks (pretext) to mine its own supervisory information from large-scale unsupervised data, and trains the network through this constructed supervisory information, so that valuable representations for downstream tasks can be learned. In other words, the supervisory information of self-supervised learning is not manually annotated, but the algorithm automatically constructs supervisory information in large-scale unsupervised data to carry out supervised learning or training.

[0042] The pre-trained model used in the present invention is a trained NLP model, whose training data comes from text data in various fields and has strong information extraction capabilities. However, since the pre-trained model is trained by text data and no audio data has ever been involved in it, how to obtain an audio decoder so that the audio hidden features output by it can be understood and accepted by the pre-trained model is a key point for the present invention to solve the technical problem. In order to solve this problem, the present invention adopts a method called "prompt learning".

[0043] The "prompt learning" referred to in the present invention is a specific way of using a pre-trained model, which can be used for small sample learning or multimodal learning. "Prompt" refers to a description used to prompt the model what you want it to do. For example, in a pre-trained model of NLP (natural language processing), a sentence is input, and a "prompt" such as "translation" or "question and answer" is added, then the model will generate the corresponding result. The basic idea of ​​prompt learning is to find a suitable "prompt" so that the pre-trained model can correctly realize the relationship between the two ends of the prompt.

[0044] Figure 2 is a flow chart of the method for identifying learning intention based on prompts of the present invention. Figure 1 Compared with the method of the prior art shown in the figure, the method of the present invention has an additional audio encoder, which extracts the latent features of the conversation speech audio and transmits them to the pre-trained model, so that the pre-trained model will receive dual information of the conversation text and the audio latent features. After the pre-trained model has learned through self-supervised prompts, it can well understand and accept the audio latent feature information, thereby outputting the textual latent feature information as the input of the intent recognition model, and finally obtaining the recognized intent.

[0045] Figure 3 : is a training flow chart of the audio encoder of the present invention. Figure 3 As shown, first, the training data in the training data set includes the conversation text and audio of the conversation speech, that is, an (audio, text) pair. The training data in this step can use the training data commonly used in ASR.

[0046] Among them, the "audio encoder" is a neural network-based model, and the audio of the conversational speech passes through the audio encoder to obtain audio latent features.

[0047] A traditional "prompt" is a question-like phrase or sentence to a pre-trained model similar to a cloze test. For example, for the input sentence "What a nice weather today!", in order for the pre-trained model to understand the positive or negative emotions, the pre-trained model can be given a prompt like this: "It means the mood of ___", and the pre-trained model can thus obtain a "positive" output. The research of the inventors of the present invention shows that this "prompt" can be further generalized, and it can not only establish associations between texts, but also between texts and non-texts. Specifically, the present invention uses "prompts" to establish associations between conversation texts and their audio implicit features, which is an innovative use of existing prompt learning. Figure 3In the present invention, the "prompt" is a short paragraph of descriptive text, such as "equivalent to", "expresses the meaning of", or synonyms and antonyms of the above phrases. Thus, the audio latent features, prompts and text are spliced ​​and input into the pre-trained model for self-supervised learning. During the whole process, the pre-trained model remains fixed and the audio decoder is trainable. That is to say, during the training process of self-supervised learning, the parameters that are updated are the audio encoder itself. Thus, after the training is completed, the audio decoder will be able to extract audio features that the pre-trained model can understand.

[0048] The following is a more detailed description of the method of applying prompt learning to intent recognition according to the present invention. Figure 4 FIG. 1 is a flowchart of a specific embodiment of the method for identifying learning intention based on prompts of the present invention. Figure 4 As shown, the method of the present invention comprises the following steps:

[0049] S1. Perform speech recognition and audio latent feature extraction on the conversation speech to obtain conversation text and audio latent features corresponding to the conversation speech.

[0050] In this embodiment, the conversational voice can come from real-time voice or voice recording in user telephone consultation or telephone complaint. In order to complete the real-time response or prompt the user to perform the next operation, it is necessary to identify the intention expressed by the user in the conversation. In the present invention, speech recognition, namely automatic speech recognition (Automatic Speech Recognition), is required first. There are four common methods of speech recognition technology: 1. The square and acoustic method based on linguistics, 2. The random model method, 3. The method using artificial neural network, 4. Probabilistic grammar analysis. Among them, the most mainstream method is the random model method. The random model method is currently more mature in application. The method mainly uses the steps of extracting features, training templates, classifying templates and judging templates to recognize speech. There are generally three technologies involved in this method: dynamic time warping (DTW), hidden Markov model (HMM) theory and vector quantization (VQ) technology. Among them, the advantages of the HMM algorithm compared with the other two are simplicity and quality, and it is more excellent in speech recognition performance. Because of this, most speech recognition systems are using the HMM algorithm now. However, it should be noted that the present invention is not limited to a specific ASR method.

[0051] At the same time, in order to promote the accuracy of the final intention recognition by citing the conversational speech, this embodiment needs to extract audio latent features from the conversational speech. As mentioned above, audio latent features include features such as logarithmic Mel spectrogram, local binary pattern, gradient histogram, etc. In this embodiment, the audio latent features of the conversational speech are extracted by using an audio encoder based on a neural network algorithm.

[0052] It should be noted that, although the present invention preferably adopts an audio encoder based on a neural network, for example, an audio encoder based on a convolutional neural network can achieve better results, in fact, the present invention is not limited to the type of specific audio encoder for generating audio latent features. In specific implementation, the sound signal can be preprocessed and discrete Fourier transformed first, the amplitude spectrum of the sound signal can be calculated, and a two-dimensional spectrogram signal can be formed. The output obtained by the trained convolutional neural network calculation is the audio latent feature to be generated.

[0053] S2. Establish a prompt based on the correlation between the conversation text and the audio latent features.

[0054] After obtaining the conversation text and the audio latent features, in this embodiment of the present invention, a prompt is established according to the correlation between the conversation text and the audio latent features. The prompt can be generated by a prompt generation template, and the prompt template can be obtained by an automated search, that is, searching for some possible designs to find the best prompt. The prompt template can also be obtained by training, that is, the prompt is trained as a trainable parameter to obtain an optimized prompt. In the present invention, considering the particularity of the correlation between the conversation text and the audio latent features, the present invention preferably uses an artificially designed prompt template to generate prompts. For example, the prompt generation template of the present invention includes the following prompts: "equivalent to", "expressed means", or synonyms and near synonyms of the above phrases. Thus, the present invention can generate corresponding prompts based on the conversation text and the prompt generation template.

[0055] However, it should be noted that the present invention is not limited to a specific "prompt" generation method and the generated "prompt" content. As long as phrases and sentences can establish a certain correlation between the conversation text and the audio hidden features, they should be regarded as different implementation methods under the concept of the present invention and are all regarded as technical solutions within the scope of protection of the present invention.

[0056] S3. Use the conversation text, audio latent features, prompts and corresponding intentions of historical conversation speech to train a machine learning model, and use the machine learning model to process the conversation text, audio latent features and prompts of the new conversation speech to identify the conversation intention of the new conversation speech.

[0057] This step is the step of generating conversation intent. After obtaining the new conversation speech, the conversation text and audio latent features are generated as described in the aforementioned steps S1 and S2, and corresponding prompts are generated, which can be input into the trained machine learning model to obtain the conversation intent. The machine learning model referred to here is a general term, and does not refer to a single specific model or several specific models, but refers to a machine learning algorithm or algorithm combination that supports prompt learning. In theory, any machine learning training method that can associate the audio latent features and conversation text in the present invention through "prompts" can implement the present invention and achieve better results than intent recognition only through conversation text.

[0058] In this specific embodiment, Figure 3 As shown, the machine learning model for performing intent recognition includes the audio encoder, a pre-trained model and an intent recognition model. The audio encoder is the audio encoder mentioned in step S1, preferably an audio encoder based on a neural network algorithm.

[0059] The pre-trained model is used to generate textual latent features for intent recognition based on conversation text, audio latent features, and prompts. The pre-trained model is a trained NLP model whose training data comes from text data in various fields and has strong information extraction capabilities. The embodiments are not limited to the specific pre-trained models used, such as BERT (Bidirectional Encoder Representations from Transformers), GPT-2 (Gererate Pre-Training Model), etc.

[0060] The intent recognition model is a machine learning model used to recognize intent based on text features or text implicit features. Intent recognition is mainly to infer what the user wants to do, so it can generally be implemented using a classification model, because intent recognition is actually a classification problem, such as rule-based algorithms, traditional machine learning algorithms (SVM), and deep learning algorithms (CNN, LSTM, RCNN, C-LSTM, FastText), etc.

[0061] The key to the present invention is how to train the above-mentioned machine learning model so that the pre-trained model can understand the audio latent features well, thereby giving full play to the contribution of the audio of the conversational speech itself to the intent recognition.

[0062] Figure 5It is a flowchart of a method for training a machine learning model according to an embodiment of the present invention. This embodiment, as a preferred implementation, uses conversation text, audio latent features, prompts and corresponding intents of historical conversation speech to train a machine learning model. The specific steps include: establishing a training data set including conversation text, audio latent features, prompts and intents; keeping the pre-trained model fixed, and using the conversation text, audio latent features and prompts in the training data set to perform self-supervised prompt learning to train the audio encoder; and keeping the audio encoder and the pre-trained model fixed, and using the training data set to train the intent recognition model.

[0063] Step S31: Establish a training data set including conversation text, audio latent features, prompts and intentions.

[0064] Specifically, the step of establishing a training data set includes: performing speech recognition and audio latent feature extraction on historical conversation speech to obtain conversation text and audio latent features corresponding to the historical conversation speech; and establishing prompts based on the correlation between the text and audio latent features of the historical conversation. This step is similar to the process in steps S1 and S2, except that this step is performed on historical conversation speech, and will not be repeated here. It should be noted that the selected historical conversation speech is a conversation speech with corresponding conversation intention information.

[0065] Step S32: keep the pre-trained model fixed, and use the conversation text, audio latent features and prompts in the training data set to perform self-supervised prompt learning to train the audio encoder.

[0066] This step is a key step in the training step. Different from the general training steps in the prior art, this step splices the audio hidden features, prompts and text, and then inputs them into the pre-training model for self-supervised learning. In the whole process, only the audio encoder is trainable. That is to say, in the training process of self-supervised learning, the updated parameter is the audio encoder itself. Therefore, after the training is completed, the audio decoder will be able to extract audio features that the pre-training model can understand. The self-supervision here is included in the training process. For example, the conversation text is "It is Friday afternoon now", then during training, we partially mask the input conversation text, for example, it becomes "It is Friday afternoon now", and the output of the model is "It is Friday afternoon now". The "#" sign represents being masked, which is an example of self-supervised learning. The self-supervised learning based on prompt learning of the present invention is not limited to masking specific parts, it can be text or hidden features.

[0067] Step S33: keep the audio encoder and the pre-trained model fixed, and use the training data set to train the intent recognition model.

[0068] This step is to train the intent recognition model using conventional model training methods. After the audio encoder and pre-training model are fixed, the training data includes conversation speech (or audio latent features), conversation text, and conversation intent. Figure 3 As shown, the conversation speech, conversation text and conversation intent are input into the audio encoder and pre-trained model in sequence, and then the obtained text hidden data and conversation intent are input into the intent recognition model for training. The final intent recognition model is obtained through training with a large amount of data.

[0069] The above describes a specific embodiment of the intent recognition method of the present invention. Those skilled in the art will appreciate that all or part of the steps to implement the above embodiments are implemented as a program executed by a data processing device (including a computer), i.e., a computer program. When the computer program is executed, the above method provided by the present invention can be implemented. Moreover, the computer program can be stored in a computer-readable storage medium, which can be a readable storage medium such as a disk, an optical disk, a ROM, a RAM, or a storage array composed of multiple storage media, such as a disk or a tape storage array. The storage medium is not limited to centralized storage, it can also be distributed storage, such as cloud storage based on cloud computing.

[0070] The following describes an apparatus embodiment of the present invention, which can be used to execute a method embodiment of the present invention. The details described in the apparatus embodiment of the present invention should be regarded as supplementary to the above method embodiment; details not disclosed in the apparatus embodiment of the present invention can be implemented with reference to the above method embodiment.

[0071] Figure 6 FIG. 1 is a module diagram of a conversation intention recognition device based on prompt learning according to an embodiment of the present invention. Figure 6 As shown, the conversation intention recognition device includes an extraction module, a prompt generation module and an intention recognition module.

[0072] The extraction module is used to perform speech recognition and audio latent feature extraction on the conversational speech to obtain the conversational text and audio latent features corresponding to the conversational speech. The extraction module needs to perform speech recognition and audio latent feature extraction, which are two different processes and can be performed by two submodules respectively, namely, a speech recognition submodule for automatic speech recognition (Automatic Speech Recognition). As mentioned above, the extraction module of the present invention is not limited to a specific ASR method.

[0073] At the same time, another audio latent feature extraction submodule needs to extract audio latent features from the conversational speech. As mentioned above, audio latent features include features such as logarithmic Mel spectrogram, local binary pattern, gradient histogram, etc. In this embodiment, the audio latent feature extraction of the conversational speech is to extract the audio latent features of the conversational speech using an audio encoder based on a neural network algorithm. Similarly, the audio latent feature extraction submodule is not limited to the type of specific audio encoder that generates audio latent features. In specific implementation, the sound signal can first be preprocessed and discrete Fourier transformed, and the amplitude spectrum of the sound signal can be calculated to form a two-dimensional spectrogram signal. The output is calculated through a trained convolutional neural network, and the audio latent feature to be generated is obtained.

[0074] A prompt generation module is used to establish prompts according to the correlation between the conversation text and the audio hidden features. The prompt generation module can generate prompts through a prompt generation template, and the prompt template can be obtained by an automated search, that is, searching for some possible designs to find the best prompt. The prompt template can also be obtained by training, that is, the prompt is treated as a trainable parameter to be trained to obtain an optimized prompt. In the present invention, considering the particularity of the correlation between the conversation text and the audio hidden features, the present invention preferably uses an artificially designed prompt template to generate prompts. For example, the prompt generation template of the present invention includes the following prompts: "equivalent to", "expressed means", or synonyms and near synonyms of the above phrases. Thus, the present invention can generate corresponding prompts according to the conversation text and the prompt generation template. However, it should be noted that the prompt generation module is not limited to the specific "prompt" generation method and the generated "prompt" content. As long as it is a phrase, sentence, etc. that can establish a certain correlation between the conversation text and the audio hidden features, it should be regarded as a different implementation method under the concept of the present invention, and it is regarded as a technical solution within the scope of protection of the present invention.

[0075] The intent recognition module is used to train a machine learning model using the conversation text, audio latent features, prompts and corresponding intents of historical conversation speech, and use the machine learning model to process the conversation text, audio latent features and prompts of new conversation speech to identify the conversation intent of the new conversation speech.

[0076] The intention recognition module is the main module for generating conversation intentions. After obtaining the new conversation speech, the intention recognition module generates conversation text and audio latent features like the extraction module and the prompt generation module, and generates corresponding prompts, which can be input into the trained machine learning model to obtain the conversation intention. The machine learning model for intention recognition includes the audio encoder, and also includes a pre-trained model and an intention recognition model. The audio encoder is the audio encoder mentioned in the extraction module, and preferably uses an audio encoder based on a neural network algorithm.

[0077] The machine learning model for performing intent recognition includes the audio encoder, a pre-trained model and an intent recognition model, wherein the pre-trained model is used to generate textual latent features for intent recognition based on conversation text, audio latent features and prompts. The intent recognition model is used to generate conversation intent based on the textual latent features.

[0078] Figure 7 FIG. 1 is a diagram showing the submodule structure of an intention recognition module according to a specific embodiment of the present invention. Figure 7 As shown, the intent recognition module includes a data set establishment submodule, an audio encoder training submodule, and an intent recognition module model training submodule. Among them, the data set establishment submodule is used to establish a training data set including conversation text, audio latent features, prompts and intent; the audio encoder training submodule; keeps the pre-trained model fixed, and uses the conversation text, audio latent features and prompts in the training data set to perform self-supervised prompt learning to train the audio encoder. The intent recognition module model training submodule is used to keep the audio encoder and pre-trained model fixed, and use the training data set to train the intent recognition model. The specific steps of the above submodules correspond to steps S31 to S32 of the intent recognition method of the present invention, so they will not be repeated here.

[0079] Those skilled in the art will appreciate that the modules in the above device embodiments may be distributed in the device as described, or may be changed accordingly and distributed in one or more devices different from the above embodiments. The modules in the above embodiments may be combined into one module, or may be further split into multiple submodules.

[0080] The electronic device embodiment of the present invention is described below, and the electronic device can be regarded as a physical implementation of the method and device embodiments of the present invention described above. The details described in the electronic device embodiment of the present invention should be regarded as a supplement to the above method or device embodiments; details not disclosed in the electronic device embodiment of the present invention can be implemented with reference to the above method or device embodiments.

[0081] Figure 8 is a structural block diagram of an exemplary embodiment of an electronic device according to the present invention. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0082] like Figure 8 As shown, the electronic device 810 of this exemplary embodiment is in the form of a general data processing device. The components of the electronic device 810 may include, but are not limited to: at least one processing unit 811, at least one storage unit 812, a bus 816 connecting different system components (including the storage unit 812 and the processing unit 311), a display unit 813, etc.

[0083] The storage unit 812 stores a computer-readable program, which may be a source program or a code of a read-only program. The program may be executed by the processing unit 811, so that the processing unit 210 performs the steps of various embodiments of the present invention. For example, the processing unit 811 may execute the following steps: Figure 8 Steps shown.

[0084] The storage unit 812 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 8121 and / or a cache storage unit 8122, and may further include a read-only storage unit (ROM) 8123. The storage unit 812 may also include a program / utility 8124 having a set (at least one) of program modules 8125, such program modules 8125 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include the implementation of a network environment.

[0085] Bus 816 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0086] The electronic device 810 may also communicate with one or more external devices 820 (e.g., keyboards, displays, network devices, Bluetooth devices, etc.) so that a user can interact with the electronic device 820 via these external devices 820, and / or the electronic device 810 can communicate with one or more other data processing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 814, and may also be performed via a network adapter 815 with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet). The network adapter 815 may communicate with other modules of the electronic device 820 via a bus 816. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in the electronic device 810, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0087] Fig. 9 Schematic diagram of a computer readable medium embodiment of the present invention. Fig. 9As shown, the computer program can be stored on one or more computer readable media. The computer readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. When the computer program is executed by one or more data processing devices, the computer-readable medium is able to implement the above-mentioned method of the present invention, namely: performing speech recognition and audio latent feature extraction on the conversational speech to obtain conversational text and audio latent features corresponding to the conversational speech; establishing prompts according to the correlation between the conversational text and the audio latent features; training a machine learning model using the conversational text, audio latent features, prompts and corresponding intentions of historical conversational speech, and using the machine learning model to process the conversational text, audio latent features and prompts of new conversational speech to identify the conversational intention of the new conversational speech.

[0088] Through the description of the above implementation modes, it is easy for those skilled in the art to understand that the exemplary embodiments described in the present invention can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation mode of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a data processing device (which can be a personal computer, a server, or a network device, etc.) to perform the above method according to the present invention.

[0089] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0090] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0091] In summary, the present invention can be implemented by a method, apparatus, electronic device or computer-readable medium that executes a computer program. In practice, a general data processing device such as a microprocessor or a digital signal processor (DSP) can be used to implement some or all functions of the present invention.

[0092] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A conversation intention recognition method based on prompt learning, characterized in that: include: Perform speech recognition and audio latent feature extraction on the conversational speech to obtain the conversational text and audio latent features corresponding to the conversational speech; The audio latent feature extraction includes: preprocessing and discrete Fourier transforming the sound signal of the conversation language, calculating the amplitude spectrum of the sound signal to form a two-dimensional spectrogram signal, and calculating it through a trained convolutional neural network to obtain an output that is the audio latent feature to be generated; the audio latent feature includes: at least one of a logarithmic Mel spectrogram, a local binary pattern, and a gradient histogram; Establishing a prompt based on the correlation between the conversation text and the audio latent feature; After splicing the conversation text, audio latent features and prompts of the historical conversation speech, a machine learning model is trained with the corresponding intent, and the conversation text, audio latent features and prompts of the new conversation speech are processed using the machine learning model to identify the conversation intent of the new conversation speech.

2. The method according to claim 1, characterized in that: The audio latent features of conversational speech are extracted using an audio encoder based on a neural network algorithm.

3. The method according to claim 2, characterized in that: The machine learning model includes the audio encoder, the pre-trained model and the intent recognition model; The pre-trained model is used to generate text latent features for intent recognition based on conversation text, audio latent features, and prompts; The intent recognition model is used to generate a conversation intent according to the text latent features.

4. The method according to claim 3, characterized in that: The concatenation of the conversation text, audio latent features and prompts of the historical conversation speech and the corresponding intention training machine learning model include: Create a training dataset that includes at least conversation text, audio latent features, prompts, and intents; The pre-trained model is kept fixed, and the audio encoder is trained by self-supervised prompt learning using the conversation text, audio latent features and prompts in the training dataset.

5. The method according to claim 4, characterized in that: The concatenation of the conversation text, audio latent features and prompts of the historical conversation speech and the corresponding intention training machine learning model include: The audio encoder and the pre-trained model are kept fixed, and the intent recognition model is trained using a training dataset.

6. The method according to claim 4, characterized in that: The steps of building a training dataset including at least conversation text, audio latent features, prompts, and intents include: Perform speech recognition and audio latent feature extraction on the historical conversation speech to obtain the conversation text and audio latent features corresponding to the historical conversation speech; Prompts are created based on the correlation between text and audio latent features of historical conversations.

7. A conversation intention recognition device based on prompt learning, characterized in that: include: An extraction module is used to perform speech recognition and audio latent feature extraction on the conversation speech to obtain the conversation text and audio latent features corresponding to the conversation speech; The audio latent feature extraction includes: preprocessing and discrete Fourier transforming the sound signal of the conversation language, calculating the amplitude spectrum of the sound signal to form a two-dimensional spectrogram signal, and calculating it through a trained convolutional neural network to obtain an output that is the audio latent feature to be generated; the audio latent feature includes: at least one of a logarithmic Mel spectrogram, a local binary pattern, and a gradient histogram; A prompt generation module, used for establishing a prompt according to the correlation between the conversation text and the audio latent feature; The intent recognition module is used to concatenate the conversation text, audio latent features and prompts of historical conversation speech, train a machine learning model with the corresponding intent, and use the machine learning model to process the conversation text, audio latent features and prompts of new conversation speech to identify the conversation intent of the new conversation speech.

8. The device according to claim 7, characterized in that: The machine learning model includes an audio encoder, a pre-trained model and an intent recognition model; The pre-trained model is used to generate text latent features for intent recognition based on conversation text, audio latent features, and prompts; The intent recognition model is used to generate a conversation intent according to the text latent features.

9. The device according to claim 8, characterized in that: The intention recognition module includes: The dataset building submodule is used to build a training dataset including conversation text, audio latent features, prompts, and intents; Audio encoder training submodule: keeping the pre-trained model fixed, using the conversation text, audio latent features and prompts in the training data set to perform self-supervised prompt learning to train the audio encoder.

10. The device according to claim 9, characterized in that: The intention recognition module also includes: The intention recognition module model training submodule is used to keep the audio encoder and the pre-trained model fixed and use the training data set to train the intention recognition model.

11. An electronic device, comprising: processor; as well as A memory storing computer executable instructions which, when executed, cause the processor to perform the method according to any one of claims 1-6.

12. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images