Voice intention recognition method, electronic device and storage medium

Abstract

The present application provides a method, electronic device and storage medium for speech intention recognition, which belongs to the field of data processing technology. The method includes: receiving speech information, and determining the target task type corresponding to the speech information based on the trigger information of the speech information; setting a corresponding target input prompt for the speech information based on the target task type, the target input prompt including a text transcription prompt and / or an intent recognition prompt; inputting the speech information with the target input prompt into a pre-trained Whisper model, calling the pre-trained Whisper model to perform an operation corresponding to the target input prompt on the speech information, and outputting an execution result, the execution result including a text transcription result and / or an intent recognition result; obtaining the execution result output by the pre-trained Whisper model, and executing the execution result. The present application can solve the problems of resource waste and cumulative accuracy loss caused by the independent deployment of dual models in the prior art, while optimizing the end-side deployment cost.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for recognizing speech intentions, an electronic device, and a storage medium. Background Art

[0002] In recent years, demand for voice interaction on mobile terminals, IoT devices, and other edge devices has grown rapidly. Its implementation relies heavily on the collaborative processing of Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) models. Existing technologies generally employ a dual-model architecture with independent deployment: the ASR model first converts speech signals into text sequences, and the NLU model then parses the text semantics and extracts user intent, ultimately driving device actions.

[0003] However, ASR and NLU models are typically deployed as independent inference units, each requiring significant computational resources and storage space. To reduce the cost of dual-model deployment, existing technologies have proposed model lightweighting techniques (such as knowledge distillation and parameter quantization). However, these methods require separate compression for both ASR and NLU, resulting in cumulative accuracy loss.

[0004] Therefore, how to build a lightweight end-side voice interaction architecture while ensuring the accuracy of voice interaction is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The purpose of this application is to provide a speech intention recognition method, electronic device and storage medium to solve the above problems.

[0006] To achieve the above objectives, in a first aspect, the present application proposes a method for speech intent recognition, the method comprising:

[0007] Receive voice information, and determine the target task type corresponding to the voice information based on trigger information of the voice information;

[0008] According to the target task type, a corresponding target input prompt is set for the voice information, wherein the target input prompt includes a text transcription prompt and / or an intent recognition prompt;

[0009] Inputting a voice message with a target input prompt into a pre-trained Whisper model, calling the pre-trained Whisper model to perform an operation corresponding to the target input prompt on the voice message, and outputting an execution result, the execution result including a text transcription result and / or an intent recognition result;

[0010] Obtain an execution result output by the pre-trained Whisper model, and execute the execution result.

[0011] In some embodiments, the pre-trained Whisper model includes an input layer and an output layer, the input layer includes an intent adaptation layer, the output layer includes a dual-stream decoder, the dual-stream decoder includes a text transcription stream and an intent recognition stream, and the dual-stream decoder is further used to select and execute the text transcription stream and / or the intent recognition stream according to an input prompt.

[0012] In some embodiments, when the target input prompt includes an intent recognition prompt, calling the pre-trained Whisper model to perform an operation corresponding to the target input prompt on the voice information and outputting an execution result includes:

[0013] Performing feature extraction on the speech information to generate an original speech embedding vector;

[0014] Based on the intent recognition prompt, calling the intent adaptation layer to map the original speech embedding vector to the feature space of the intent recognition stream to generate a new embedding vector;

[0015] Concatenating the new embedding vector with the original speech embedding vector to generate a reconstructed embedding vector;

[0016] The reconstructed embedding vector is input into the intent recognition flow, and the intent recognition flow is called to generate an intent recognition result based on the reconstructed embedding vector.

[0017] In some embodiments, the intent recognition flow includes a multi-layer perceptron and a hierarchical intent classifier, and calling the intent recognition flow to generate an intent recognition result based on the reconstructed embedding vector includes:

[0018] Performing feature extraction on the reconstructed embedding vector to generate an intent feature vector;

[0019] Performing feature enhancement on the intention feature vector by the multi-layer perceptron to generate an enhanced intention feature vector;

[0020] Inputting the enhanced intent feature vector into the hierarchical intent classifier, and forming several levels of intent prediction results through the hierarchical intent classifier;

[0021] The intention prediction results of the several levels are structured to generate intention recognition results.

[0022] In some embodiments, the intent recognition flow further includes a termination judge, which inputs the enhanced intent feature vector into the hierarchical intent classifier, and generates several levels of intent prediction results through the hierarchical intent classifier, including:

[0023] Inputting the enhanced intent feature vector into the first level of the hierarchical intent classifier to obtain the intent prediction result of the first level;

[0024] Inputting the intention prediction result of the current level and the enhanced intention feature vector into the termination judge to determine whether to proceed to the intention prediction of the next level;

[0025] If so, the intention prediction result of the current level and the enhanced intention feature vector are input to the next level of the hierarchical intention classifier, and the step of inputting the intention prediction result of the current level and the enhanced intention feature vector to the termination judge to determine whether to perform intention prediction for the next level is repeated;

[0026] If not, the prediction is terminated.

[0027] In some embodiments, when the target input prompt includes a text transcription prompt, calling the pre-trained Whisper model to perform an operation corresponding to the target input prompt on the voice information and outputting an execution result, the execution result including a text transcription result and / or an intent recognition result, including:

[0028] Extracting features from the speech information using a multi-head attention mechanism and a feedforward neural network to generate an initial speech feature vector, wherein the initial speech feature vector includes long-term dependencies and context information in the speech information;

[0029] Based on the text transcription prompt, the text transcription flow is called to convert the initial speech feature vector into a text sequence, and the text sequence is output as a text transcription result.

[0030] In some embodiments, before inputting the speech information provided with the target input prompt into the pre-trained Whisper model, the method further includes:

[0031] Obtaining an initial Whisper model, and replacing the translation prompt of the initial Whisper model with the intent recognition prompt;

[0032] Adding an intent recognition layer to the input layer of the initial Whisper model; and constructing a dual-stream decoder at the output layer of the initial Whisper model to form a modified Whisper model;

[0033] The dual-stream decoder includes a text transcription stream and an intent recognition stream, and the dual-stream decoder is used to select and execute the text transcription stream and / or the intent recognition stream according to an input prompt;

[0034] The modified Whisper model is trained using a training sample set containing a number of quadruple data, a text transcription sample set, and an intent recognition sample set to obtain a pre-trained Whisper model.

[0035] In some embodiments, the training of the initial Whisper model using a training sample set comprising a plurality of quadruple data, a text transcription sample set, and an intent recognition sample set to obtain a pre-trained Whisper model includes:

[0036] Obtaining a training sample set comprising a plurality of quadruple data, wherein the quadruple data comprises speech, text, input prompt, and intent recognition result;

[0037] Obtaining a text transcription sample set and an intent recognition sample set, and setting a corresponding input prompt for each sample in the text transcription sample set and the intent recognition sample set;

[0038] Inputting the text transcription sample set and the intent recognition sample set into the initial Whisper model to obtain a text transcription result set and an intent recognition result set respectively output by the initial Whisper model;

[0039] Calculating a text transcription loss based on the text transcription result set and the training sample set, and calculating an intent classification loss based on the intent recognition result set and the training sample set;

[0040] Feedback updates are performed on the initial Whisper model based on the text transcription loss and the intent classification loss until training is completed.

[0041] In a second aspect, the present application provides an electronic device, comprising:

[0042] one or more processors;

[0043] a memory for storing one or more programs,

[0044] When the one or more programs are executed by the one or more processors, the one or more processors execute the voice intention recognition method as described above.

[0045] In a third aspect, the present application proposes a storage medium storing executable instructions, which, when executed by a processor, causes the processor to execute the voice intent recognition method as described above.

[0046] Compared with the prior art, the advantages of this application include:

[0047] By modifying and retraining the Whisper model, the pre-trained Whisper model can handle both text transcription and intent recognition tasks simultaneously, resolving the resource waste and cumulative accuracy loss issues caused by the independent deployment of two models in existing technologies. Specifically: First, by modifying the prompt mechanism of the Whisper model, the translation task prompt is replaced with the intent recognition prompt, allowing the Whisper model, which does not have intent recognition capabilities, to perform the intent recognition operation corresponding to the intent recognition prompt. Second, an intent adaptation layer is added to the input layer of the Whisper model. The original speech embedding vector is mapped to the feature space of intent recognition through linear projection, and is concatenated with the original embedding vector to generate a reconstructed embedding vector, allowing the model to capture the intent features in the speech without relying on an independent intent recognition model. Third, by constructing a dual-stream decoder at the output layer, including a text transcription stream and an intent recognition stream, the model can dynamically select the corresponding stream to execute based on the input prompt, thereby flexibly responding to different task requirements and improving processing efficiency. The improvements described above not only retain the high accuracy of the Whisper model in text transcription, but also enhance its performance in intent recognition, avoid the accuracy loss caused by model compression, and optimize the end-side deployment costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope of the present application.

[0049] Figure 1 1 is a flow chart of a method for speech intent recognition according to an embodiment;

[0050] Figure 2 A schematic diagram of a process for performing intent recognition in one embodiment;

[0051] Figure 3 A schematic diagram of a detailed process for performing intent recognition in one embodiment;

[0052] Figure 4 A schematic diagram of a detailed process for hierarchical intent recognition in one embodiment;

[0053] Figure 5 A schematic diagram of a process for performing text transcription in one embodiment;

[0054] Figure 6 Schematic diagram of a process for modifying and training an initial Whisper model in one embodiment;

[0055] Figure 7 This is a structural diagram of the electronic device involved in the voice intention recognition method in the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0057] All terms (including technical and scientific terms) used in this application have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0058] For example, the terms "first," "second," etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element.

[0059] For example, the terms "include", "comprising", etc. used in this application indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0060] As mentioned above, the demand for voice interaction on mobile terminals, IoT devices, and other edge devices has grown rapidly in recent years. Its implementation relies heavily on the collaborative processing of Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) models. Existing technologies generally employ a dual-model independent deployment architecture: the ASR model converts speech signals into text sequences, while the NLU model parses the text semantics and extracts user intent, ultimately driving device execution. However, the ASR and NLU models are typically deployed as independent inference units, each requiring significant computing resources and storage space. To reduce the cost of dual-model deployment, existing technologies have proposed model lightweighting techniques (such as knowledge distillation and parameter quantization). However, these methods require separate compression of the ASR and NLU models, resulting in cumulative accuracy loss. Therefore, building a lightweight edge-to-edge voice interaction architecture while maintaining voice interaction accuracy is a pressing technical challenge. To this end, this application proposes a speech intent recognition method, electronic device and storage medium. By modifying and retraining the Whisper model (Whisper is a general speech recognition model developed by OpenAI, focusing on converting speech into text with high precision, especially in multilingual and complex environments), the pre-trained Whisper model not only retains its high precision in text transcription, but also enhances its performance in intent recognition. It can handle text transcription and intent recognition tasks at the same time, solving the problems of resource waste and cumulative accuracy loss caused by the independent deployment of dual models in the existing technology, avoiding the accuracy loss caused by model compression, and optimizing the end-side deployment cost.

[0061] In one embodiment, Figure 1 As shown, the embodiment of the present application proposes a method for speech intention recognition, which includes the following steps:

[0062] Step S10: receiving voice information, and determining the target task type corresponding to the voice information according to triggering information of the voice information.

[0063] In this embodiment, voice information refers to the acoustic signal captured by a user through a voice input device (such as a microphone), typically represented as a digital audio waveform or mel-spectrogram. The target task type refers to the type of task the model needs to perform, including text transcription and / or intent recognition. Trigger information refers to specific features in the voice information used to identify the task type. This can be the trigger method and keywords used when the user triggers voice input. For example, if the software used to trigger the voice input is an input method, the trigger method can be used to determine the target task type as text transcription. If the trigger information also includes keywords such as "send message," the target task type is determined to be text transcription plus intent recognition. If the voice input is triggered by a command call, the trigger method of the command call can be used to determine the target task type as intent recognition. Trigger information can also be keywords in the voice information, contextual features, or multimodal features combined with other sensor data (such as visual information). For example, in the MR glasses scenario, trigger information can include keywords in the voice and visual information collected by the glasses (such as information about the object the user is looking at or gestures) to assist in determining the task type.

[0064] Step S20: Setting a corresponding target input prompt for the voice information according to the target task type.

[0065] In this embodiment, an input prompt refers to an identifier used to instruct the model to perform the target task, including a text transcription prompt and / or an intent recognition prompt. Specifically, a text transcription prompt refers to an identifier used to instruct the model to perform the text transcription task, and an intent recognition prompt refers to an identifier used to instruct the model to perform the intent recognition task.

[0066] In some implementations, the input prompt can be an additional dimension or a marker for a specific position in the model input, or it can serve as an input prefix for the speech information. For example, when the input prompt serves as the input prefix for the speech information, and the input prefix includes [decoder_start_token_id, language_id, task_id, notimestamps], then "task_id" can be set to "transcribe".

[0067] In some embodiments, a language prompt may be set for the voice information based on the language information of the voice information. For example, when the language prompt is used as the input prefix of the voice information, and the input prefix includes [decoder_start_token_id, language_id, task_id, notimestamps], then "language_id" may be set to "CN" (Chinese) or "EN" (English), etc.

[0068] Step S30: inputting the voice information provided with the target input prompt into the pre-trained Whisper model, calling the pre-trained Whisper model to perform an operation corresponding to the target input prompt on the voice information, and outputting the execution result.

[0069] In this embodiment, the pre-trained Whisper model refers to a model that has been modified and retrained based on the original Whisper model (the original Whisper model is a general-purpose speech recognition model developed by OpenAI, which focuses on high-precision speech-to-text conversion, particularly outstanding performance in multilingual and complex environments). This allows the pre-trained Whisper model to not only retain its high accuracy in text transcription but also enhance its performance in intent recognition, enabling it to handle both text transcription and intent recognition tasks simultaneously. The execution results include text transcription results and / or intent recognition results.

[0070] In some embodiments, the pre-trained Whisper model includes an input layer and an output layer. The input layer includes an intent adaptation layer, which is a sublayer of the input layer and is used to map the embedded vector of the speech information to the feature space of the intent recognition task. The output layer includes a two-stream decoder, which includes a text transcription stream and an intent recognition stream. The two-stream decoder is further used to select and execute the text transcription stream and / or the intent recognition stream based on the input prompt. The text transcription stream is used to generate text transcription results based on a multi-head attention mechanism and a feedforward neural network. The intent recognition stream is suitable for generating intent recognition results based on a multi-layer perceptron and a hierarchical intent classifier.

[0071] In some embodiments, a pre-trained Whisper model can be called to perform an operation corresponding to the text transcription prompt on the voice information and output a text transcription result, and / or a pre-trained Whisper model can be called to perform an operation corresponding to the intent recognition prompt on the voice information and output an intent recognition result.

[0072] In some embodiments, when the target input prompt includes an intent recognition prompt, such as Figure 2 As shown, step S20 includes:

[0073] Step A10: extract features from the speech information to generate an original speech embedding vector.

[0074] In some embodiments, the speech information can be converted into a Mel spectrum, and the Mel spectrum can be feature extracted to generate an original speech embedding vector. The original speech embedding vector is a preliminary feature representation of the speech information by the pre-trained Whisper model, which can be used to carry the acoustic features of the speech information (such as pitch, phonemes, speaking speed, etc.) and is expressed as a vector of fixed dimension.

[0075] In some embodiments, the feature extraction process can directly use the encoder of the pre-trained Whisper model, or use algorithms such as short-time Fourier transform (STFT) or wavelet transform, or embed a pre-trained feature extraction model (such as ResNet) to generate more advanced feature representations.

[0076] Step A20: Based on the intent recognition prompt, call the intent adaptation layer to map the original speech embedding vector to the feature space of the intent recognition stream to generate a new embedding vector.

[0077] In some embodiments, based on the intent recognition prompt, the intent adaptation layer is called to map the original speech embedding vector to the feature space of the intent recognition stream through linear projection to generate a new embedding vector. The new embedding vector has stronger feature expression capabilities and is more suitable for intent recognition tasks.

[0078] Step A30: concatenate the new embedding vector with the original speech embedding vector to generate a reconstructed embedding vector.

[0079] In this embodiment, the reconstructed embedding vector refers to a new concatenated vector that contains both the acoustic and intent features of the speech information. By concatenating the new embedding vector with the original speech embedding vector, we can introduce intent features while preserving the acoustic features of the speech information, further enhancing the model's expressive power.

[0080] In some embodiments, the new embedding vector may be weightedly concatenated with the original speech embedding vector to generate a reconstructed embedding vector, wherein the weights of the new embedding vector and the original speech embedding vector may be fixed or obtained through pre-training learning.

[0081] The new embedding vector can also be concatenated with the original speech embedding vector using an attention mechanism to generate a reconstructed embedding vector. Specifically, the new embedding vector can be identified as the query vector, the key vector can be identified as the original speech embedding vector, the similarity between the query vector (new embedding vector) and the key vector (original speech embedding vector) can be calculated using a dot product or a scaled dot product, and an attention weight can be generated. The attention weight is then applied to the concatenation of the new embedding vector and the original speech embedding to generate the reconstructed embedding vector.

[0082] Step A40: input the reconstructed embedding vector into the intent recognition flow, and call the intent recognition flow to generate an intent recognition result based on the reconstructed embedding vector.

[0083] In some embodiments, the intent recognition flow includes a multi-layer perceptron and a hierarchical intent classifier, such as Figure 3 As shown, the step A40 includes:

[0084] Step A41: perform feature extraction on the reconstructed embedding vector to generate an intent feature vector.

[0085] The feature vector related to intent recognition is extracted from the reconstructed embedding vector as the intent feature vector.

[0086] Step A42: Perform feature enhancement on the intention feature vector through the multi-layer perceptron to generate an enhanced intention feature vector.

[0087] In this embodiment, a multi-layer perceptron (MLP) is a feed-forward neural network composed of multiple fully connected layers for feature transformation and enhancement.

[0088] In some embodiments, the reconstructed embedding vector is input into a multi-layer perceptron, and the intent feature vector is nonlinearly transformed through the fully connected layer and activation function (such as ReLU, Swish, or Mish) in the multi-layer perceptron to generate an enhanced intent feature vector.

[0089] Step A43: input the enhanced intent feature vector into the hierarchical intent classifier, and form several levels of intent prediction results through the hierarchical intent classifier.

[0090] In this embodiment, a hierarchical intent classifier is a classifier used to handle multi-level intent recognition tasks and is composed of multiple levels of classifiers. When the enhanced intent feature vector is input into the hierarchical intent classifier, each level of the classifier predicts the intent at that level in turn and passes the results to the next level, thus forming a multi-level intent prediction result.

[0091] In some embodiments, the intent recognition flow further includes a termination judgement, such as Figure 4 As shown, step A43 includes:

[0092] Step A431: input the enhanced intent feature vector into the first level of the hierarchical intent classifier to obtain the intent prediction result of the first level.

[0093] In this embodiment, the enhanced intent feature vector is input into the first level of the hierarchical intent classifier to obtain the probability distribution corresponding to various possible first-level intents, and the first-level intent with the highest probability is used as the first-level intention prediction result.

[0094] For example, if the voice message is "Take a selfie for me," the enhanced intent feature vector includes the embedded representations of the keywords "selfie" and "photo" in the voice message and contextual information. The first level of the hierarchical intent classifier, based on the embedded representations in the enhanced intent feature vector, outputs various first-level intents, such as "Open the camera app," "Open the gallery," "Open the front camera," and "Find selfies." It selects the first-level intent with the highest probability, such as "Open the camera app," as the first-level intent prediction result.

[0095] Step A432: input the intention prediction result of the current level and the enhanced intention feature vector into the termination judge.

[0096] Step A433, determine whether to perform intention prediction at the next level.

[0097] In this embodiment, the termination determinator is a module used to determine whether to continue predicting the intent at the next level. It can be a small multi-layer perceptron (MLP). The intent prediction result of the current level and the enhanced intent feature vector are concatenated into a high-dimensional fusion vector. This high-dimensional fusion vector is input to the termination determinator. The termination determinator outputs a probability value indicating the likelihood of continuing the prediction. By determining whether the probability value reaches a preset threshold, it determines whether to proceed to the next level of intent prediction.

[0098] In some implementations, the termination determinator is pre-trained to enable it to determine the number of levels of intent prediction. Specifically, a training sample set containing multi-level intent labels is obtained, where each sample includes speech, text, and multi-level intent labels, as well as a flag indicating whether further prediction is required. The termination determinator is trained using this training sample set to obtain a pre-trained termination determinator.

[0099] Step A434: If yes, the intent prediction result of the current level and the enhanced intent feature vector are input to the next level of the hierarchical intent classifier, and steps A432 and A433 are repeated.

[0100] In this embodiment, if the probability value output by the termination judge is greater than a preset threshold, the intention prediction result and enhanced intention feature vector of the current level are input into the classifier of the next level, and the judgment step is repeated and executed.

[0101] Exemplarily, the enhanced intent feature vector is input into the first level of the hierarchical intent classifier to obtain the probability distribution corresponding to various possible first-level intents, and the first-level intent with the highest probability is used as the intention prediction result of the first level.

[0102] Exemplarily, the enhanced intent feature vector including the embedded representation of the keywords "selfie" and "photo" in the voice information and the contextual information, as well as the first-level intention prediction result (opening the photo-taking application) are input into the second level of the hierarchical intent classifier to obtain the probability distribution of various secondary intentions, such as "opening the front camera", "taking a photo", etc., and the secondary intention with the highest probability, such as "opening the front camera", is used as. Then, the second-level intention prediction result "opening the front camera" and the enhanced intent feature vector are input into the termination judge to determine whether it is necessary to continue predicting the third-level intent. If so, the above process is repeated to obtain the third-level intention prediction result "taking a photo".

[0103] Step A435: If not, terminate the prediction.

[0104] In this embodiment, if the probability value output by the termination judge is less than or equal to a preset threshold, further intention prediction is stopped.

[0105] Step A44: Structural processing is performed on the intention prediction results of the multiple levels to generate intention recognition results.

[0106] In this embodiment, multiple levels of intent prediction results are combined into a structured output format. For example, if the first-level intent prediction result is "open the camera app," the second-level intent prediction result is "open the front camera," and the third-level intent prediction result is "take a photo," the generated intent recognition result might be "open the camera app\nopen the front camera\ntake a photo."

[0107] In some embodiments, when the target input prompt includes a text transcription prompt, such as Figure 5 As shown, step S20 includes:

[0108] Step B10: extract features from the speech information through a multi-head attention mechanism and a feedforward neural network to generate an initial speech feature vector.

[0109] In this embodiment, the multi-head attention mechanism is used to capture long-term dependencies in sequential data. It uses multiple attention heads to perform parallel computations, capturing correlations between different positions. A feedforward neural network is a simple neural network structure consisting of multiple fully connected layers for feature transformation. The initial speech feature vector extracted using the multi-head attention mechanism and feedforward neural network can include long-term dependencies and contextual information in the speech information.

[0110] Step B20: Based on the text transcription prompt, call the text transcription flow to convert the initial speech feature vector into a text sequence, and output the text sequence as a text transcription result.

[0111] In this example, the text transcription flow is actually the end-to-end speech recognition decoder natively implemented in the Whisper model. This decoder, composed of multiple Transformer layers, is responsible for gradually generating a text sequence from the initial speech feature vectors generated by the encoder. At each step in the text sequence generation process, the decoder predicts the next most likely character or word based on the current context and previously generated text. The resulting text sequence interacts with the encoder output through an attention mechanism to obtain relevant contextual information.

[0112] Step S40: Obtain the execution result output by the pre-trained Whisper model and execute the execution result.

[0113] In this embodiment, the electronic device executing the speech intent recognition method obtains the execution result from the pre-trained Whisper model and performs the corresponding operation based on the result type. If the execution result is a text transcription result, the text transcription result is displayed to the user. If the execution result is an intent recognition result, the corresponding application operation is triggered based on the intent recognition result.

[0114] In the speech intent recognition method proposed in the embodiment of the present application, by modifying and retraining the Whisper model (Whisper is a general speech recognition model developed by OpenAI, focusing on converting speech into text with high precision, especially outstanding performance in multi-language and complex environments), the pre-trained Whisper model not only retains the high precision in text transcription, but also enhances its performance in intent recognition. It can handle text transcription and intent recognition tasks at the same time, solving the problems of resource waste and cumulative accuracy loss caused by the independent deployment of dual models in the existing technology, avoiding the accuracy loss caused by model compression, and optimizing the end-side deployment cost.

[0115] In one embodiment, Figure 6 As shown, before step S30, the following steps are also included:

[0116] Step S00: obtaining an initial Whisper model, and replacing the translation prompt of the initial Whisper model with an intent recognition prompt.

[0117] In this example, the initial Whisper model refers to a general-purpose speech recognition model developed by OpenAI, which focuses on high-precision speech-to-text conversion, particularly excelling in multilingual and complex environments. By modifying the initial Whisper model's prompt mechanism, replacing translation prompts with intent recognition prompts, the model's core structure can be adapted to new tasks while maintaining its original structure.

[0118] Step S01: adding an intent recognition layer to the input layer of the initial Whisper model.

[0119] In this embodiment, an intent adaptation layer is added to the input layer of the initial Whisper model to map the original speech embedding vector to the feature space of the intent recognition task.

[0120] Step S02: constructing a dual-stream decoder at the output layer of the initial Whisper model to form a modified Whisper model.

[0121] In this embodiment, the dual-stream decoder includes a text transcription stream and an intent recognition stream, and the dual-stream decoder is used to select and execute the text transcription stream and / or the intent recognition stream according to the input prompt. The initial Whisper model itself has a text transcription function. By expanding the initial Whisper model and adding an intent recognition stream, it can simultaneously process text transcription and intent recognition tasks. The multi-layer perceptron and hierarchical intent classifier in the intent recognition stream are components specially designed for intent recognition tasks. They work in conjunction with the encoder and decoder of the initial Whisper model to achieve multi-tasking processing.

[0122] Step S03: training the modified Whisper model using a training sample set containing a plurality of quadruple data, a text transcription sample set, and an intent recognition sample set to obtain a pre-trained Whisper model.

[0123] In some embodiments, a training sample set comprising several quadruple data is obtained, wherein the quadruple data includes speech, text, input prompts, and intent recognition results; a text transcription sample set and an intent recognition sample set are obtained, and corresponding input prompts are set for each sample in the text transcription sample set and the intent recognition sample set, respectively; the text transcription sample set and the intent recognition sample set are input into the initial Whisper model to obtain a text transcription result set and an intent recognition result set output by the initial Whisper model, respectively; a text transcription loss is calculated based on the text transcription result set and the training sample set, and an intent classification loss is calculated based on the intent recognition result set and the training sample set; and the initial Whisper model is feedback updated based on the text transcription loss and the intent classification loss until the training is completed.

[0124] In the speech intention recognition method proposed in the embodiment of the present application, by modifying and retraining the Whisper model, the pre-trained Whisper model can simultaneously process text transcription and intent recognition tasks, solving the problem of resource waste and cumulative accuracy loss caused by the independent deployment of dual models in the prior art. Specifically as follows: First, by modifying the prompt mechanism of the Whisper model, the translation task prompt is replaced with the intent recognition prompt, so that the Whisper model that does not have the intent recognition function can perform the intent recognition operation corresponding to the intent recognition prompt. Secondly, an intent adaptation layer is added to the input layer of the Whisper model, and the original speech embedding vector is mapped to the feature space of intent recognition through linear projection, and spliced with the original embedding vector to generate a reconstructed embedding vector, so that the model can capture the intent features in the speech without relying on an independent intent recognition model. Thirdly, by constructing a dual-stream decoder at the output layer, including a text transcription stream and an intent recognition stream, the model can dynamically select the corresponding stream to execute according to the input prompt, thereby flexibly responding to different task requirements and improving processing efficiency. The improvements described above not only retain the high accuracy of the Whisper model in text transcription, but also enhance its performance in intent recognition, avoid the accuracy loss caused by model compression, and optimize the end-side deployment costs.

[0125] In one embodiment, a computer-readable storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the steps in the above-mentioned method embodiments.

[0126] In one embodiment, an electronic device is also provided, comprising one or more processors; a memory, wherein one or more programs are stored in the memory, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the steps in the above-mentioned method embodiments.

[0127] In one embodiment, Figure 7 , which shows a schematic diagram of the structure of an electronic device for implementing an embodiment of the present application. Electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 702 or programs loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 also stores various programs and data required for the operation of electronic device 700. CPU 701, ROM 702, and RAM 703 are connected to each other via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0128] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, and the like; an output section 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 708 including devices such as a hard disk; and a communication section 709 including a network interface card such as a LAN card or a modem. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. Removable media 711, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 710 as needed, so that computer programs read from the media can be installed in the storage section 708 as needed.

[0129] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer-readable medium carrying instructions. In such embodiments, the instructions can be downloaded and installed from a network via communication component 709 and / or installed from removable media 711. When the instructions are executed by central processing unit (CPU) 701, the various method steps described in this application are performed.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

[0131] Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and to form different embodiments. For example, any of the above embodiments may be used in any combination. The information disclosed in this background section is intended solely to enhance understanding of the overall background of this application and should not be construed as an admission or any implication that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for speech intention recognition, characterized in that: The method comprises: Receive voice information, and determine the target task type corresponding to the voice information based on trigger information of the voice information; According to the target task type, a corresponding target input prompt is set for the voice information, wherein the target input prompt includes a text transcription prompt and / or an intent recognition prompt; Inputting speech information provided with a target input prompt into a pre-trained Whisper model, calling the pre-trained Whisper model to perform an operation corresponding to the target input prompt on the speech information, and outputting an execution result, wherein the execution result includes a text transcription result and / or an intent recognition result, wherein the pre-trained Whisper model includes an input layer and an output layer, the input layer includes an intent adaptation layer, the output layer includes a dual-stream decoder, the dual-stream decoder includes a text transcription stream and an intent recognition stream, and the dual-stream decoder is further used to select and execute the text transcription stream and / or the intent recognition stream according to the input prompt; Obtaining an execution result output by the pre-trained Whisper model, and executing the execution result; Wherein, when the target input prompt includes an intent recognition prompt, calling the pre-trained Whisper model to perform an operation corresponding to the target input prompt on the voice information and outputting an execution result includes: Perform feature extraction on the speech information to generate an original speech embedding vector; based on the intent recognition prompt, call the intent adaptation layer to map the original speech embedding vector to the feature space of the intent recognition stream to generate a new embedding vector; concatenate the new embedding vector with the original speech embedding vector to generate a reconstructed embedding vector; input the reconstructed embedding vector into the intent recognition stream; and call the intent recognition stream to generate an intent recognition result based on the reconstructed embedding vector.

2. The method for speech intention recognition according to claim 1, wherein: The intent recognition flow includes a multi-layer perceptron and a hierarchical intent classifier, and calling the intent recognition flow to generate an intent recognition result based on the reconstructed embedding vector includes: Performing feature extraction on the reconstructed embedding vector to generate an intent feature vector; Performing feature enhancement on the intention feature vector by the multi-layer perceptron to generate an enhanced intention feature vector; Inputting the enhanced intent feature vector into the hierarchical intent classifier, and forming several levels of intent prediction results through the hierarchical intent classifier; The intention prediction results of the several levels are structured to generate intention recognition results.

3. The method for speech intention recognition according to claim 2, wherein: The intent recognition flow also includes a termination judge, which inputs the enhanced intent feature vector into the hierarchical intent classifier, and forms several levels of intent prediction results through the hierarchical intent classifier, including: Inputting the enhanced intent feature vector into the first level of the hierarchical intent classifier to obtain the intent prediction result of the first level; Inputting the intention prediction result of the current level and the enhanced intention feature vector into the termination judge to determine whether to proceed to the intention prediction of the next level; If so, the intention prediction result of the current level and the enhanced intention feature vector are input to the next level of the hierarchical intention classifier, and the step of inputting the intention prediction result of the current level and the enhanced intention feature vector to the termination judge to determine whether to perform intention prediction for the next level is repeated; If not, the prediction is terminated.

4. The method for speech intention recognition according to claim 1, wherein: When the target input prompt includes a text transcription prompt, calling the pre-trained Whisper model to perform an operation corresponding to the target input prompt on the voice information and outputting an execution result, the execution result including a text transcription result and / or an intent recognition result, including: Extracting features from the speech information using a multi-head attention mechanism and a feedforward neural network to generate an initial speech feature vector, wherein the initial speech feature vector includes long-term dependencies and context information in the speech information; Based on the text transcription prompt, the text transcription flow is called to convert the initial speech feature vector into a text sequence, and the text sequence is output as a text transcription result.

5. The method for speech intention recognition according to claim 1, wherein: Before inputting the voice information provided with the target input prompt into the pre-trained Whisper model, the method further includes: Obtaining an initial Whisper model, and replacing the translation prompt of the initial Whisper model with the intent recognition prompt; Adding an intent recognition layer to the input layer of the initial Whisper model; and Constructing a two-stream decoder at the output layer of the initial Whisper model to form a modified Whisper model; The dual-stream decoder includes a text transcription stream and an intent recognition stream, and the dual-stream decoder is used to select and execute the text transcription stream and / or the intent recognition stream according to an input prompt; The modified Whisper model is trained using a training sample set containing a number of quadruple data, a text transcription sample set, and an intent recognition sample set to obtain a pre-trained Whisper model.

6. The method for speech intention recognition according to claim 5, wherein: The initial Whisper model is trained using a training sample set containing a plurality of quadruple data, a text transcription sample set, and an intent recognition sample set to obtain a pre-trained Whisper model, including: Obtaining a training sample set comprising a plurality of quadruple data, wherein the quadruple data comprises speech, text, input prompt, and intent recognition result; Obtaining a text transcription sample set and an intent recognition sample set, and setting a corresponding input prompt for each sample in the text transcription sample set and the intent recognition sample set; Inputting the text transcription sample set and the intent recognition sample set into the initial Whisper model to obtain a text transcription result set and an intent recognition result set respectively output by the initial Whisper model; Calculating a text transcription loss based on the text transcription result set and the training sample set, and calculating an intent classification loss based on the intent recognition result set and the training sample set; Feedback updates are performed on the initial Whisper model based on the text transcription loss and the intent classification loss until training is completed.

7. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the speech intention recognition method according to any one of claims 1 to 6.

8. A storage medium, characterized in that: The storage medium stores executable instructions, which, when executed by a processor, enable the processor to execute the speech intention recognition method according to any one of claims 1 to 6.

Images