Multitask voice understanding method and device

Through multi-task learning and multi-head attention mechanism, the speech vector weight is dynamically adjusted, combined with the Transformer network, the nuances and deep meaning recognition difficulties and model in the existing speech comprehension system when dealing with natural language are solved, and the problem of model inability to interpret is achieved. Accurate and reliable voice comprehension enhances the interactive experience.

CN118982980APending Publication Date: 2024-11-19DMAI (GUANGZHOU) CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411181282.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing speech comprehension systems are difficult to capture the nuances and deep meanings of language when dealing with natural language, and the increase in model parameters leads to uninterpretation and misjudgment risks, lack of in-depth modeling of human thinking patterns, and it is difficult to deal with multitasking and complexity. Scene.

Method used

Using multi-task learning method, the feature extraction and encoding is obtained by obtaining speech signals, the multi-head attention mechanism is used to dynamically adjust the weight of the speech vector, and multi-task recognition is combined with the Transformer network, including gender, age, language, speech speed, emotion, and speakers. and design a comprehensive loss function optimization model.

Benefits of technology

It realizes a deep understanding and processing of human language, improves the comprehensive performance and application scope of the speech comprehension system, provides a more natural and friendly interactive experience, and improves the accuracy and reliability of speech recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982980A_ABST
    Figure CN118982980A_ABST
Patent Text Reader

Abstract

The invention relates to a multi-task voice understanding method and device. The method comprises the steps of obtaining a voice signal for feature extraction to obtain voice features; encoding the voice features to obtain a hidden state of a voice vector, performing multi-task recognition according to the hidden state of the voice vector, and obtaining a voice understanding result according to a recognition result; wherein the task identification comprises the steps of dynamically adjusting the weight of the hidden state of the voice vector by adopting multi-head attention, and performing task identification according to the adjusted voice vector. According to the invention, thinking and behavior modes of human can be better simulated and learned, accuracy and reliability of voice understanding can be improved, and more natural and friendly interaction experience can be provided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of natural language understanding, and more particularly to a multi-task speech understanding method and device. Background Art

[0002] In today's information society, speech understanding systems, as one of the key technologies for human-computer interaction, have made significant progress. With the continuous advancement of deep learning technology and the improvement of computing power, these systems have evolved from simple command recognition to being able to process complex natural language instructions. Modern speech understanding systems can not only recognize keywords in speech, but also deeply understand semantic, emotional and contextual information, thereby providing a more personalized and rich interactive experience.

[0003] At present, speech understanding technologies include traditional speech recognition systems, context-aware speech systems, and emotion recognition speech systems. These systems have their own focuses, but they all have the following shortcomings:

[0004] Existing technologies often focus on a single task or a specific type of speech recognition and lack the ability to handle multi-tasks and complex scenarios;

[0005] When processing natural language, speech understanding systems often have difficulty capturing the nuances and deeper meanings of the language, leading to deviations in understanding.

[0006] Although some systems attempt to introduce contextual awareness and emotion recognition, they generally lack in-depth modeling of human thinking patterns and cannot achieve true natural language understanding. That is, although existing speech understanding technologies perform well in specific fields, they still show obvious deficiencies in simulating and learning human thinking patterns, which limits their performance in handling multi-tasks and complex scenarios.

[0007] At the same time, as the system functions continue to expand and deepen, the number of model parameters has gradually increased. Although the generalization ability of the speech understanding system has been significantly improved, it has also caused the problem of model unexplainability. This makes it difficult to see through the decision-making process of the model in some cases, increasing the risk of misjudgment.

[0008] In order to overcome the above shortcomings, the present invention proposes a multi-task oriented speech understanding system. Summary of the invention

[0009] In view of this, the present invention provides a multi-task speech understanding method and device, aiming to achieve in-depth understanding and processing of human language through a multi-task learning method, thereby improving the comprehensive performance and application scope of the speech understanding system.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] A multi-task speech understanding method, comprising:

[0012] S1, obtain the speech signal and perform feature extraction to obtain speech features;

[0013] S2, encode the speech features to obtain the hidden state of the speech vector;

[0014] S3. Perform multi-task recognition according to the hidden state of the speech vector, and obtain the speech understanding result according to the recognition result; wherein the task recognition includes: dynamically adjusting the weight of the hidden state of the speech vector using multi-head attention; and performing task recognition according to the adjusted speech vector.

[0015] Preferably, in S1, after acquiring the voice signal, preprocessing is performed first, and the preprocessing includes noise reduction, echo cancellation and volume normalization.

[0016] Preferably, in S1, the feature extraction step includes:

[0017] Convert speech signals from time domain to frequency domain through fast Fourier transform;

[0018] The frequency domain signal is filtered using a Mel filter bank to obtain a Mel spectrum graph;

[0019] The mel-frequency cepstral coefficients of the mel-spectrogram are used as speech features.

[0020] Preferably, the speech feature is position-encoded by sine and cosine functions, and the encoding result is added to the corresponding speech feature.

[0021] Preferably, in S3, multi-head attention is used to dynamically adjust the weight of the hidden state of the speech vector according to task requirements, and the steps include:

[0022] Get the attention weight of each head in the multi-head attention, dynamically adjust it using the control factor, and then merge the weights to the hidden state of the corresponding speech vector;

[0023] The regulatory factor is obtained by the following formula:

[0024] a i =σ(GH i )

[0025] In the formula, G represents the learnable weight matrix, σ represents the sigmoid function, and H i represents the hidden state of the ith vector.

[0026] Preferably, the multi-task recognition includes:

[0027] The adjusted speech vectors are respectively subjected to gender recognition, age recognition, language recognition, speech rate recognition, emotion recognition, speaker recognition, voice recognition, speech recognition and / or semantic understanding.

[0028] Preferably, the dynamic adjustment process and the multi-task speech understanding process are trained according to the following loss function;

[0029]

[0030] Where, L j represents the loss function of the jth speech understanding task, N represents the total number of speech understanding tasks, and b j represents the assigned weight of the loss function of the j-th speech understanding task;

[0031] in,

[0032]

[0033] Where, T j represents the difficulty coefficient of the j-th speech understanding task, and ∈ represents a constant.

[0034] A multi-task speech understanding device, using the multi-task speech understanding method as described above, comprises:

[0035] A feature extraction unit, used to acquire a speech signal and perform feature extraction to obtain speech features;

[0036] Transformer network, which is used to encode speech features and dynamically adjust the weights of the encoded speech vectors using multi-head attention;

[0037] The multi-task recognition unit is used to perform multi-task recognition according to the adjusted speech vector and obtain a speech understanding result according to the recognition result.

[0038] Preferably, the multi-task recognition unit includes: a gender recognition decoder, an age recognition decoder, a language recognition decoder, a speaking rate recognition decoder, an emotion recognition decoder, a speaker recognition decoder, a sound recognition decoder, a speech recognition decoder and / or a semantic understanding decoder.

[0039] It can be seen from the above technical solutions that the present invention discloses a multi-task speech understanding method and device, the purpose of which is to provide a comprehensive, efficient and intelligent speech understanding system to meet the needs of modern speech interaction technology; compared with the prior art, this application is based on the concept of value alignment and adopts a multi-task learning method, which can better simulate and learn human thinking and behavior patterns, while improving the accuracy and reliability of speech recognition, and providing a more natural and friendly interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0041] Figure 1 A flow chart of the multi-task speech understanding method provided by the present invention;

[0042] Figure 2 A flowchart of a multi-task speech understanding method provided by the present invention;

[0043] Figure 3 This is a schematic diagram of the structure of the multi-task speech understanding device provided by the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In order to solve the above-mentioned problems existing in the prior art, the present application provides a highly integrated artificial intelligence solution to achieve comprehensive analysis and understanding of voice input through a series of carefully designed modules.

[0046] In one embodiment,

[0047] like Figure 1 , the present application discloses a multi-task speech understanding method, the steps comprising:

[0048] S1, obtain the speech signal and perform feature extraction to obtain speech features;

[0049] S2, encode the speech features to obtain the hidden state of the speech vector;

[0050] S3. Perform multi-task recognition according to the hidden state of the speech vector, and obtain the speech understanding result according to the recognition result; wherein the task recognition includes: dynamically adjusting the weight of the hidden state of the speech vector using multi-head attention; and performing task recognition according to the adjusted speech vector.

[0051] Specifically, Figure 2As shown, in step S1, after the speech signal is acquired, noise reduction, echo cancellation, volume normalization and other operations are first performed to ensure the clarity and quality of the speech signal.

[0052] Subsequently, feature extraction is performed on the preprocessed speech signal. Specifically, the time domain signal can be converted into the frequency domain using Fast Fourier Transform (FFT), and a Mel-spectrogram can be generated using a Mel filter bank as the acoustic representation of the input data.

[0053] The feature extraction step in this embodiment specifically includes:

[0054] Convert speech signals from time domain to frequency domain through fast Fourier transform;

[0055] The frequency domain signal is filtered using a Mel filter bank to obtain a Mel spectrum graph;

[0056] The mel-frequency cepstral coefficients of the mel-spectrogram are used as speech features to capture information such as the pitch, volume, and timbre of the speech.

[0057] Furthermore, in one embodiment, position information is added to each frame feature to ensure that the model can understand the time series characteristics in the speech signal; in this embodiment, different frequencies of sine and cosine functions are used to adapt to feature representations of different time steps and to ensure that the model can utilize the time order information in the sequence.

[0058] For each time step t and each dimension d, the positional encoding is computed as follows:

[0059]

[0060] where d model is the dimension of the hidden layer of the model, t is the current position, and d is the current dimension index;

[0061] Then the sine and cosine values ​​of all dimensions are added to get the final position code PE(t,d).

[0062] This application ensures that the encoding of each position is unique and contains different signals from low frequency to high frequency by assigning even dimensions to sine functions and odd dimensions to cosine functions.

[0063] For each time step in the sequence, the position encoding vector PE(t) of the corresponding position is added to the feature vector X(t) of the time step to obtain the feature representation X'(t) with position information; the formula is as follows:

[0064] X′(t)=X(t)+PE(t)

[0065] Step S2, encode the speech features to obtain the hidden state of the speech vector.

[0066] This embodiment uses a self-attention mechanism to process the input sequence, allowing the model to consider the information of all positions in the sequence when encoding. In this embodiment, a Transformer encoder module is used to encode the speech features, that is, the encoded hidden state of each element in the sequence is calculated through the self-attention mechanism and position encoding.

[0067] In step S3, multi-task recognition is performed according to the hidden state of the speech vector, and the task recognition includes: dynamically adjusting the weight of the hidden state of the speech vector using multi-head attention; and performing task recognition according to the adjusted speech vector.

[0068] According to the needs of different tasks, this application dynamically adjusts the weights of attention heads based on the multi-head attention mechanism, which can simultaneously focus on information in different subspaces and enhance the understanding of input data, so that the recognition task can more flexibly focus on features that are more important to the current task.

[0069] The specific steps for dynamic weight adjustment are as follows:

[0070] 1) Get the attention weight of each head in the multi-head attention,

[0071] The dynamic multi-head attention module is based on the multi-head attention mechanism, where the input sequence is split into multiple "heads" and each head learns a different representation of the input. For each head, the representation of the query (Q), key (K), and value (V) is calculated:

[0072] Q i =W Q H,K i =W K H,V i =W V H

[0073] Among them, H represents the hidden state, W Q , W K and W V is the learnable weight matrix and i is the index of the head.

[0074] Attention weight calculation: In this embodiment, the attention weight of each head is calculated by the following formula:

[0075]

[0076] 2) Use regulatory factors for dynamic adjustment,

[0077] To achieve dynamics, the attention weights of each head can be controlled by a learnable weight vector α, which is adjusted through a gating mechanism such as the sigmoid function:

[0078] a i =σ(GH)

[0079] Where G represents the learnable weight matrix and σ represents the sigmoid function.

[0080] Attention′ i =α i ⊙Attention i

[0081] The present invention can adjust the importance of each head by applying different transformations to the weight vector according to the requirements of different tasks.

[0082] 3) The adjusted attention weights are combined and weighted to each feature vector; that is, the weighted attention outputs of all heads are combined to form the final attention representation:

[0083] MultiHead(X)=Concat(Attention'1,...,Attention' h )W O

[0084] Where h is the number of attention heads, W O is another learnable weight matrix.

[0085] In this embodiment, during training, all learnable parameters, including the weight matrix W, are updated by the back propagation algorithm. Q , W K , W V , G and W O and parameters in the gating mechanism.

[0086] In one embodiment, multiple recognition tasks are designed, including but not limited to gender recognition, age recognition, language recognition, speech rate recognition, emotion recognition, speaker recognition, voice recognition, speech recognition and / or semantic understanding;

[0087] The above recognition task uses an autoregressive mechanism to predict the result of the current time step based on the output of the previous time step, and allows the recognition model to be optimized specifically for different tasks. In this embodiment, the corresponding task recognition network structure is designed according to the characteristics of each recognition task; the details are as follows:

[0088] (1) A gender recognition network, which is used to provide gender-related context information. In this embodiment, the network is a small convolutional neural network (CNN), and the structure includes:

[0089] Input layer: receives the encoded hidden state from the Transformer Encoder;

[0090] Convolutional layer 1: 32 filters of size 3x3, ReLU activation function;

[0091] Pooling layer: maximum pooling, window size is 2x2;

[0092] Convolutional layer 2: 64 filters of size 3x3, ReLU activation function;

[0093] Fully connected layer: has 128 neurons and ReLU activation function;

[0094] Output layer: has 2 neurons, softmax activation function, representing male and female.

[0095] (2) Age recognition network, used to provide contextual information related to age. In this embodiment, a recurrent neural network (RNN) model, especially a long short-term memory network (LSTM) is used for age classification:

[0096] Input layer: receives the encoded hidden state from the Transformer Encoder;

[0097] LSTM layer: has 128 units;

[0098] Fully connected layer: has 64 neurons and ReLU activation function;

[0099] Output layer: has 1 neuron, linear activation function, and outputs continuous value of age.

[0100] (3) Language recognition network: This embodiment uses a fully connected neural network (FCNN) as a language recognition decoder. The network structure is as follows:

[0101] Input layer: receives the encoded hidden state from the Transformer Encoder;

[0102] Hidden layer 1: has 128 neurons and ReLU activation function;

[0103] Hidden layer 2: has 64 neurons and ReLU activation function;

[0104] Output layer: has N neurons and softmax activation function, where N is the number of languages.

[0105] (4) Speech rate recognition network, using a single-layer perceptron model for speech rate classification:

[0106] Input layer: receives the encoded hidden state from the Transformer Encoder;

[0107] Hidden layer: 32 neurons, sigmoid activation function;

[0108] Output layer: has 1 neuron, linear activation function, and outputs the continuous value of speech rate.

[0109] (5) Emotion recognition network, which infers the speaker's emotional state by analyzing the pitch, rhythm, intensity and other characteristics of the speech and combining them with contextual information. This embodiment uses a deep CNN for emotion recognition, and its structure includes:

[0110] Input layer: receives the encoded hidden state from the Transformer Encoder;

[0111] Convolutional layer 1: 64 filters of size (3,3), ReLU activation function;

[0112] Convolutional layer 2: 128 filters of size (3,3), ReLU activation function;

[0113] Pooling layer: maximum pooling, window size is (2,2);

[0114] Fully connected layer 1: has 256 neurons and ReLU activation function;

[0115] Fully connected layer 2: has 128 neurons and ReLU activation function;

[0116] Output layer: has 8 neurons and softmax activation function, representing different emotion categories.

[0117] (6) Speaker recognition network, which uses unique acoustic features to distinguish different speakers and enhance the system's personalized service capabilities. This application uses deep CNN combined with feature pooling strategy;

[0118] Input layer: receives the encoded hidden state from the Transformer Encoder;

[0119] Convolutional layer 1: 128 filters of size (5,5), ReLU activation function;

[0120] Convolutional layer 2: 256 filters of size (5,5), ReLU activation function;

[0121] Pooling layer: average pooling, window size is (2,2);

[0122] Fully connected layer: has 512 neurons and ReLU activation function;

[0123] Output layer: has M neurons and softmax activation function, where M is the number of speakers.

[0124] (7) A sound recognition network that recognizes non-human environmental sounds, such as natural sounds or object collision sounds, to determine the scene and provide contextual information related to the scene. In this embodiment, the network is a blue-scale CNN with the following structure:

[0125] Input layer: receives the encoded hidden state from the Transformer Encoder;

[0126] Convolutional layer 1: 16 filters of size (3,3), ReLU activation function;

[0127] Convolutional layer 2: 32 filters of size (3,3), ReLU activation function;

[0128] Pooling layer: maximum pooling, window size is (2,2);

[0129] Fully connected layer: has 64 neurons and ReLU activation function;

[0130] Output layer: has K neurons and softmax activation function, where K is the number of sound categories.

[0131] (8) The speech recognition network uses a well-trained acoustic model to recognize phonemes and words and outputs a phoneme sequence. This embodiment is a sequence-to-sequence model that also incorporates an attention mechanism:

[0132] Input layer: receives the encoded hidden state from the Transformer Encoder;

[0133] Attention layer: used to weight the encoded hidden state;

[0134] GRU layer: has 256 units and is used to process attention-weighted input;

[0135] Fully connected layer: has 128 neurons and ReLU activation function;

[0136] Output layer: has V neurons and softmax activation function, where V is the size of the vocabulary.

[0137] (9) Semantic understanding network, which uses natural language processing technology to perform word sense disambiguation, syntactic analysis and semantic role labeling to deeply understand the meaning of the text. The present invention uses a Transformer-based encoder-decoder architecture:

[0138] Input layer: receives the encoded hidden state from the Transformer Encoder;

[0139] Decoder: consists of multiple self-attention layers and feed-forward network layers;

[0140] Attention layer: implements masked self-attention in the decoder to prevent future information leakage;

[0141] Output layer: Generates the probability distribution of vocabulary through a linear layer and a softmax activation function.

[0142] Furthermore, this embodiment provides a context manager and a decision engine for obtaining a speech understanding result according to the recognition result; specifically,

[0143] The context manager is used to track the context information of the conversation, including historical user input, system output, and conversation topic, to ensure a reasonable response;

[0144] A decision engine that combines semantic understanding, contextual information, and the results of gender, emotion, speaker, and voice recognition to generate appropriate responses.

[0145] It should be noted that the output layers of the above-mentioned task recognition networks are all designed according to specific needs, and can be generally divided into classification tasks, regression tasks and sequence generation tasks.

[0146] For classification tasks, the output layer uses a fully connected layer followed by a softmax activation function; it is expressed as follows:

[0147] O class =softmax(W class H+b class )

[0148] Among them, H represents the encoding hidden state, W class and b class are learnable weights and biases.

[0149] For regression tasks, the output layer can be a single-layer fully connected network, with either a linear activation function or no activation function:

[0150] O regress =W regress H+b regress

[0151] For sequence generation tasks, the output layer can be a sequence-to-sequence model, where the output of the network at each time step is mapped to the next hidden state through a linear layer:

[0152] O seq (t) = W seq H(t)+bseq

[0153] Among them, H d (t) represents the hidden state of the decoder at time step t.

[0154] In addition, for tasks that need to consider contextual information, this application sets the output layer to 5. An attention mechanism is integrated to focus on the key parts of the input sequence:

[0155] O attn =softmax(QH)V

[0156] where Q and V are the query and value matrices of the decoder.

[0157] To further optimize the above technical solution, this application designs a comprehensive loss function to optimize the performance of all tasks at the same time. The loss function includes classification loss, regression loss or other specific task losses, and balances the importance of different tasks by adjusting weights. First, define a loss function for each task, such as cross entropy loss, mean square error loss, etc., and then add the weighted sum of the loss functions of all tasks to form a comprehensive loss function. Finally, use the comprehensive loss function to train the model end-to-end.

[0158] Specifically, the classification task is defined as the cross entropy loss, that is:

[0159]

[0160] The regression task is defined as mean square error loss, that is:

[0161]

[0162] In the above formula, y ic is the true label of task i in category c, is the predicted probability or value and C is the number of categories or outputs.

[0163] Further, in this application, the comprehensive loss function is expressed as follows;

[0164]

[0165] Where, L j represents the loss function of the jth speech understanding task, N represents the total number of speech understanding tasks, and b j represents the assigned weight of the loss function of the j-th speech understanding task;

[0166] in,

[0167]

[0168] Where, T jrepresents the difficulty coefficient of the j-th speech understanding task, and ∈ represents a constant.

[0169] In another embodiment, the present application also discloses a multi-task speech understanding device, preferably using a Transformer model framework; Figure 3 , the device comprises:

[0170] A feature extraction module is used to obtain speech signals and perform feature extraction to obtain speech features;

[0171] A feature encoding module is used to encode speech features to obtain speech vectors;

[0172] The feature decoding module is used to perform multi-task recognition based on the speech vector and obtain the speech understanding result based on the recognition result;

[0173] Among them, the feature decoding module includes a weight dynamic adjustment unit and a multi-task recognition unit;

[0174] A weight dynamic adjustment unit, which is used to dynamically adjust the weight of the speech vector according to task requirements using multi-head attention;

[0175] The multi-task recognition unit is used to perform task recognition according to the adjusted speech vector.

[0176] Furthermore, the multi-task recognition unit includes: a gender recognition decoder, an age recognition decoder, a language recognition decoder, a speech rate recognition decoder, an emotion recognition decoder, a speaker recognition decoder, a sound recognition decoder, a speech recognition decoder and / or a semantic understanding decoder.

[0177] Preferably, the multi-task speech understanding device of the present application is provided with a signal preprocessing module and a position encoding module before feature extraction;

[0178] A post-processing and optimization module is provided after the decoding module for optimizing and adjusting the outputs of all modules. In one embodiment, it also includes confidence evaluation of the recognition results and adjustment of the response strategy.

[0179] The multi-task speech understanding method or system provided in this application can better simulate and learn human thinking and behavior patterns, improve the accuracy and reliability of speech recognition, and provide a more natural and friendly interactive experience, thereby meeting the needs of modern speech interaction technology. Specifically, it can be applied to the following scenarios:

[0180] Smart home control: Users can control smart devices in their homes, such as lighting, temperature, security systems, etc., through voice commands. The system needs to recognize the user's instructions, speech speed, emotional state, etc. to provide a more personalized and comfortable home experience.

[0181] Customer service and call center: In the field of customer service, speech understanding systems can automatically handle customer inquiries, complaints or service requests, and provide more humane services through gender and emotion recognition.

[0182] Medical and health consultation: In the medical field, the system can help patients with preliminary health consultation, provide medical advice or guide patients to seek professional medical help through speech recognition and semantic understanding.

[0183] Education and learning assistance: In educational scenarios, the speech understanding system can be used as a learning assistance tool to help students with language learning, knowledge Q&A, etc., while providing personalized learning suggestions based on students' speaking speed, emotions, etc.

[0184] Traffic navigation and safety monitoring: In the field of transportation, the system can provide real-time navigation services while monitoring the driver's voice status, such as fatigue or emotional fluctuations, to ensure driving safety.

[0185] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0186] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-task speech understanding method, characterized in that: include: S1, obtain the speech signal and perform feature extraction to obtain speech features; S2, encode the speech features to obtain the hidden state of the speech vector; S3. Perform multi-task recognition according to the hidden state of the speech vector, and obtain the speech understanding result according to the recognition result; wherein the task recognition includes: dynamically adjusting the weight of the hidden state of the speech vector using multi-head attention; and performing task recognition according to the adjusted speech vector.

2. A multi-task speech understanding method according to claim 1, characterized in that: In S1, after the voice signal is acquired, preprocessing is performed first, and the preprocessing includes noise reduction, echo cancellation and volume normalization.

3. A multi-task speech understanding method according to claim 1, characterized in that: In S1, the feature extraction step includes: Convert speech signals from time domain to frequency domain through fast Fourier transform; The frequency domain signal is filtered using a Mel filter bank to obtain a Mel spectrum graph; The mel-frequency cepstral coefficients of the mel-spectrogram are used as speech features.

4. A multi-task speech understanding method according to claim 3, characterized in that: The speech features are positionally encoded using sine and cosine functions, and the encoded results are added to the corresponding speech features.

5. A multi-task speech understanding method according to claim 1, characterized in that: In S3, multi-head attention is used to dynamically adjust the weight of the hidden state of the speech vector according to task requirements. The steps include: Get the attention weight of each head in the multi-head attention, dynamically adjust it using the control factor, and then merge the weights to the hidden state of the corresponding speech vector; The regulatory factor is obtained by the following formula: to i =σ(GH i ) In the formula, G represents the learnable weight matrix, σ represents the sigmoid function, and H i represents the hidden state of the ith vector.

6. A multi-task speech understanding method according to claim 1, characterized in that: Multi-task recognition includes: The adjusted speech vectors are respectively subjected to gender recognition, age recognition, language recognition, speech rate recognition, emotion recognition, speaker recognition, voice recognition, speech recognition and / or semantic understanding.

7. A multi-task speech understanding method according to claim 1, characterized in that: The dynamic adjustment process and the multi-task speech understanding process are trained according to the following loss function; Where, L j represents the loss function of the jth speech understanding task, N represents the total number of speech understanding tasks, and b j represents the assigned weight of the loss function of the j-th speech understanding task; in, Where T represents the difficulty coefficient of the j-th speech understanding task, and ∈ represents a constant.

8. A multi-task speech understanding device, characterized in that: A multi-task speech understanding method according to any one of claims 1 to 7 is applied, comprising: A feature extraction unit, used to acquire a speech signal and perform feature extraction to obtain speech features; Transformer network, which is used to encode speech features and dynamically adjust the weights of the encoded speech vectors using multi-head attention; The multi-task recognition unit is used to perform multi-task recognition according to the adjusted speech vector and obtain a speech understanding result according to the recognition result.

9. The multi-task speech understanding device according to claim 8, characterized in that: The multi-task recognition unit includes: a gender recognition decoder, an age recognition decoder, a language recognition decoder, a speech rate recognition decoder, an emotion recognition decoder, a speaker recognition decoder, a sound recognition decoder, a speech recognition decoder and / or a semantic understanding decoder.