An intention recognition method and system based on multimodal fusion of voice and sight

Abstract

The present invention relates to an intention recognition method and system based on multimodal fusion of speech and sight, belonging to the technical field of intention recognition. It includes: feature extraction: extracting text, speech and sight features from speech and face; multimodal representation: including: 1) modal shared representation; 2) modal specific representation; multimodal fusion: including: 3) intra-modal fusion; 4) cross-modal fusion; intention recognition: inputting the final fusion feature into a multi-layer perceptron, and connecting the softmax layer to output the classification result for intention recognition. The present invention randomly selects a subject in the training set in each training step, and applies a meta-learning method, that is, an algorithm based on a first-order gradient to update the model, alleviate the overfitting problem, and enable the model to optimize new parameters. The present invention realizes appearance-independent sight estimation and extraction of key facial features by applying a sight feature extraction method based on full-face appearance and a random identity adversarial network.

Description

Technical Field

[0001] The present invention relates to an intention recognition method and system based on multimodal fusion of speech and sight, belonging to the technical field of intention recognition. Background Art

[0002] In the interactive environment of the digital marketing metaverse, intent recognition helps improve the efficiency and quality of customer interactions, helps companies gain a deeper understanding of customer needs, provide personalized services, achieve precision marketing, and enhance customer satisfaction.

[0003] At present, in order to accurately understand customer intentions, multimodal fusion technology can more comprehensively and accurately understand user intentions by comprehensively utilizing information from different modalities. In the digital marketing metaverse environment, voice in human-computer interaction can extract customer text expression information and tone and emotional information, while facial sight information can reflect the user's attention and interests. Fusion of voice and sight information can provide richer intention information and improve the accuracy and robustness of the intention recognition algorithm.

[0004] The gaze of a face contains rich information about human intentions, and gaze estimation technology based on deep learning has received great attention. This technology can learn high-level gaze features from high-dimensional video images, thereby significantly improving the accuracy of face gaze estimation. However, due to differences in lighting, occlusion, and facial appearance in face images, using a limited dataset of specific people for model training often results in overfitting of the model. Therefore, it is still challenging to apply face gaze estimation to actual intent recognition applications.

[0005] In addition, although multimodal fusion technology has solved these problems to a certain extent, it still faces many challenges in how to effectively fuse information from different modalities to improve the accuracy of customer intent recognition. For example, how to deal with the heterogeneity and synchronization of data from different modalities, and how to design efficient feature fusion algorithms.

[0006] The main reasons for the above problems include: incomplete or improper selection of modal information; limited line of sight estimation training data; heterogeneity of modal information; information redundancy and noise; unreasonable design of feature fusion method; weight distribution problem during fusion; and insufficient interaction between modalities.

[0007] Problems such as improper selection of modal information, overfitting of the line of sight estimation model, and unreasonable multimodal fusion methods will affect the generalization ability and accuracy of the intent recognition model. To solve these problems, the performance of the model can be improved by carefully selecting modal information, meta-learning, optimizing regularization techniques, and designing reasonable feature fusion methods. Summary of the invention

[0008] In view of the defects and shortcomings in the prior art, the present invention proposes a multimodal fusion method based on the joint action of speech and sight for intention recognition, aiming to solve the following technical problems:

[0009] 1. Selection of modal information: In the task of human-computer interaction intention recognition, selecting appropriate modal information is the key. Among them, facial gaze information can reflect the user's attention and interests, and voice can extract the customer's text expression information and tone of voice emotional information. Both gaze and voice are key information of user intention.

[0010] 2. Overfitting of the gaze estimation model: In the gaze estimation task, overfitting may manifest itself as the model's prediction of a specific scene, individual, or gaze direction is very accurate, but when faced with different scenes or characters, the model's prediction accuracy drops significantly.

[0011] 3. The impact of multimodal feature fusion methods on intent recognition accuracy: Multimodal feature fusion refers to extracting features from multiple modalities (such as voice, text, images, gestures, etc.) and effectively combining them to improve the model's ability to understand user intent in complex scenarios. Selecting an appropriate fusion method can enhance the information complementarity between different modalities and improve the accuracy and robustness of intent recognition.

[0012] The technical solution of the present invention is:

[0013] An intention recognition method based on multimodal fusion of speech and sight, comprising:

[0014] Feature extraction: Extract text, voice, and gaze features from voice and face based on the pre-trained BERT model, Wav2vec 2.0 model, and self-trained FGEN model;

[0015] Multimodal representation: including: 1) Modality sharing representation: construct a modality sharing encoder to learn shared features across modalities; the modality sharing encoder converts text, speech and sight features into a unified feature space to obtain shared features, and uses the central moment difference to minimize the similarity loss between shared features of different modalities; 2) Modality specific representation: construct a modality specific encoder to learn specific features of each modality; the modality specific encoder converts text, speech and sight features into a specific feature space to obtain specific features; through differential loss, ensure that the distribution of shared features and specific features of the same modality is different, and the distribution of specific features of different modalities is also different;

[0016] Multimodal fusion: including: 3) Intra-modal fusion: The shared features and specific features of each modality are fused through the self-attention mechanism to obtain the single-modal fusion features; 4) Cross-modal fusion: Use the cross-attention mechanism to learn the relevant features of cross-modality, and fuse the features of different modalities through the gating mechanism to obtain the final fusion features;

[0017] Intent recognition: The final fusion features are input into the multi-layer perceptron and connected to the softmax layer to output the classification results for intent recognition.

[0018] Extract text features based on the pre-trained BERT model; including:

[0019] Active acquisition is used to capture the speech in the interactive environment and the acquired text are all input into the BERT model. is the word vector in the text, m is the total number of word vectors, and the output of the last hidden layer of the BERT model represents the text feature , is the sequence length of the text utterance, is the feature dimension of each token.

[0020] Extract speech features based on the pre-trained Wav2vec 2.0 model; including:

[0021] Active acquisition is used to capture speech in interactive environments , each speech segment are all input into the Wav2vec 2.0 model; the output of the last hidden layer of the Wav2vec 2.0 model represents the speech features , Indicates the sequence length of the speech segment, Represents the feature dimension of each token.

[0022] Preferably, according to the present invention, the line of sight features are extracted based on the pre-trained FGEN model; comprising:

[0023] Using SWCNN as the training baseline model, the network architecture is extended by connecting the face identity classifier (face recognition model) to achieve appearance-independent gaze estimation and facial key feature extraction; a meta-learning strategy is used to train the FGEN model; that is, each iterative training randomly selects subjects from the face gaze dataset to form a meta-training dataset and a meta-test dataset to obtain the gaze feature embedding is the sequence length of key frames, is the feature dimension of each frame; Indicates a A real number matrix, including the feature embeddings of all key frames, where each row corresponds to the feature of a key frame and each column corresponds to the value of a specific feature dimension; as shown below:

[0024] ;

[0025] in, represents the face identity classifier, represents the gaze prediction model and ML represents the meta-learning strategy.

[0026] Preferably, according to the present invention, a meta-learning strategy is adopted to train the FGEN model, and the FGEN model training utilizes an adversarial strategy to achieve the independence of the sight line feature and the appearance subject; including:

[0027] Face recognition parameters updated;

[0028] The parameters of the FGEN model are updated using line-of-sight loss and adversarial loss.

[0029] The gaze prediction model converts a face image into a gaze direction vector; as shown below:

[0030] ;

[0031] in, Represents the line of sight yaw and pitch vector labels, represents the facial gaze feature vector output by the FGEN model, represents the face gaze prediction model, Represents a face dataset;

[0032]

[0033] represents the face identity classifier, represents the recognition probability vector of each type of face in the training subjects;

[0034] Face identity classifier The parameters of are updated according to the cross entropy loss function of the recognition probability and the label. The cross entropy loss function As shown below:

[0035] ;

[0036] in, is the identification label of each subject’s face, is the predicted recognition probability of the nth subject’s face, and N is the total number of training data; represents the logarithmic function;

[0037] Next, the FGEN model continues adversarial training; face gaze prediction model Optimize the opposite goal of the face identity classifier; thus, the adversarial loss of appearance generalization is defined as follows:

[0038] ;

[0039] ;

[0040] in, represents the accuracy of the predicted probability of the subject’s face, adversarial loss representing appearance generalization; , is a uniform distribution, k is the number of subjects in the training set; Indicates the norm; Use cosine similarity to construct a loss function to make the predicted constant probability Use uniform distribution;

[0041] Gaze direction loss uses L1 loss , defined as:

[0042] ;

[0043] in, Represents the face gaze prediction model The predicted vector.

[0044] Using multi-objective loss functions Training is as follows:

[0045]

[0046] Among them, according to experience, .

[0047] Preferably, according to the present invention, in each training step, a subject in the training set is randomly selected, and a meta-learning method, i.e., an algorithm based on a first-order gradient, is applied to update the model; comprising:

[0048] First, k subjects are randomly selected from the training dataset and a new meta-training set is constructed using the face images of these k subjects , the meta-training set The sequence is input into the FGEN model, and the gradient vector is calculated during the optimization process as follows:

[0049] ;

[0050] in, is the initial weight of the line of sight prediction model, x is the number of meta-training iterations, represents the gradient vector calculated during the optimization of the line of sight prediction model, It is the calculation function of the gradient vector;

[0051] Then, update the initial weights using the following equation :

[0052]

[0053]

[0054] in, is the step size used by the stochastic gradient descent operation, are the weights of the gaze prediction model optimized by the meta-training process, refers to the weights updated by gradient descent;

[0055] Next, in the meta-training set Construct a meta-adaptation set with p non-overlapping topics in , and adapt the element to the iterative weights Updated to:

[0056] ;

[0057] After the meta-adaptation iteration, a new meta-adaptation set is created by selecting new topics , repeat the meta-adaptation process δ times and calculate the updated weights ;

[0058] Finally, the face gaze prediction model is based on the following formula To update:

[0059] ;

[0060] ;

[0061] in, y is the number of meta-adaptation iterations, For use The weights of the face gaze prediction model optimized by the meta-adaptation process;

[0062] Line of sight feature extraction: Input the processed image and trained parameters into the FGEN model to obtain line of sight features =FGEN( , ,ML).

[0063] According to the preferred embodiment of the present invention, in the text feature , voice features and sight characteristics Before multimodal representation, a multi-headed Transformer is used for preprocessing , and .

[0064] Preferably, according to the present invention, the modal sharing representation includes:

[0065] By constructing a modality-shared encoder To learn shared features across modalities, the modality-shared encoder , and Convert to a unified feature space to obtain shared features of text, speech, and sight respectively , and :

[0066] ;

[0067] ;

[0068] ;

[0069] Use the central moment difference CMD to minimize the similarity loss; let X and Y be bounded random samples in the interval The probability distributions on are p and q respectively, and the CMD regularization matrix is defined as:

[0070] ;

[0071] In the formula, X and Y will be cross-bred into three different modes respectively, A and B represent the value range of X and Y, , is the empirical expectation vector of random sample X, , is the vector of all k-order sample central moments in X, and calculates the shared features between each pair of patterns :

[0072] ;

[0073] Among them, t, and v are text, voice, and sight identifiers, respectively, and are the common features of mode m1 and mode m2; mode m1 and mode m2 are any two of the three modes; minimize the loss , will force the shared feature representation distributions of each pair of modalities to be similar.

[0074] Preferably, according to the present invention, the modality-specific representation comprises: constructing a modality-specific encoder, comprising: , ,and , corresponding to text, speech and sight respectively; the modality-specific encoder converts , and Transform to unique feature space to obtain specific features , and :

[0075] ;

[0076] ;

[0077] ;

[0078] The following differential loss is formed Calculation formula:

[0079] ;

[0080] in, is the square of the L2-norm, if and ; and are orthogonal, then the differential loss Minimum, t, and v are text, voice, and sight identifiers, respectively, and are the common features of mode m1 and mode m2 respectively; mode m1 and mode m2 are any two of the three modes respectively.

[0081] According to the preferred embodiment of the present invention, before the multimodal fusion operation, a decoder is constructed To input shared features and specific features, where represents shared features, Indicates specific characteristics, Represents the decoder parameters; reconstructs the original feature space:

[0082] ;

[0083] ;

[0084] ;

[0085] in, 、 、 They represent the essential information of three modal features respectively;

[0086] Use the mean square error to estimate the reconstruction error:

[0087] ;

[0088] in, is the square of the L2-norm, Regularization term to prevent overfitting, W is the decoder parameter; Representation reconstruction error refers to the error or loss incurred by the model when encoding the input data into a latent representation and reconstructing it through the decoder.

[0089] Preferably, the intra-modality fusion according to the present invention comprises:

[0090] The calculation formula of the self-attention mechanism is as follows:

[0091]

[0092] in, , and denote query, key, and value matrices respectively; , and are the parameters that need to be learned. is the dimension of K; for the self-attention mechanism, the three matrices Q, K, and V come from the same input; the shared features and specific features of text, speech, and sight are connected respectively and input into the self-attention mechanism to obtain the single-modal fusion features , and .

[0093] Preferably, cross-modal fusion according to the present invention includes:

[0094] After obtaining the unimodal fusion features, the cross-attention mechanism is used to learn text-to-view. , Eyes to Text , Text to Speech , Speech to Text , Voice to Sight , sight to voice The cross attention mechanism combines two sequences asymmetrically, one as the input of Q and the other as the input of K and V; the calculation formula of the cross attention mechanism is as follows:

[0095] ;

[0096] Will and Combine the inputs into the sight gate to get the sight feature fusion weight ;Will and The combination of is input to the auditory gate to obtain the speech feature fusion weight ;Will and Combine the inputs into the text gate to get the text feature fusion weights ; The calculation formula is as follows:

[0097] ;

[0098] ;

[0099] ;

[0100] in, , and is the linear layer parameter, , and is the bias term;

[0101] According to the fusion weight, the line of sight features and voice features With text features Fusion is performed to obtain the final fusion feature h:

[0102] .

[0103] Preferably, according to the present invention, intention identification includes:

[0104] After obtaining the final fusion feature h, h is input into the multi-layer perceptron and connected to the softmax layer to obtain the classification result , as shown below:

[0105] ;

[0106] Where W and b represent the linear layer parameters and the bias term.

[0107] According to the preferred embodiment of the present invention, the loss is optimized together , , and cross entropy loss ; The final optimization target L is as follows:

[0108] ;

[0109] ;

[0110] in, , and is the weight that determines the contribution to the overall loss L; N is the number of training samples; and Represent the actual label distribution and predicted label distribution of sample i respectively; is L2 regularization, and W is the model parameter.

[0111] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the processor implements the steps of an intention recognition method based on multimodal fusion of speech and sight.

[0112] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of an intention recognition method based on multimodal fusion of speech and sight.

[0113] An intention recognition system based on multimodal fusion of speech and sight, comprising:

[0114] The feature extraction module is configured to extract text, speech, and gaze features from speech and face based on the pre-trained BERT model, Wav2vec 2.0 model, and the self-trained FGEN model;

[0115] The multimodal representation module is configured to include: modality sharing representation: constructing a modality sharing encoder to learn cross-modal shared features; the modality sharing encoder converts text, speech and sight features into a unified feature space to obtain shared features, and uses the central moment difference to minimize the similarity loss between shared features of different modalities; modality specific representation: constructing a modality specific encoder to learn specific features of each modality; the modality specific encoder converts text, speech and sight features into a specific feature space to obtain specific features; through differential loss, ensure that the distribution of shared features and specific features of the same modality is different, and the distribution of specific features of different modalities is also different;

[0116] The multimodal fusion module is configured to include: intra-modal fusion: fusing the shared features and specific features of each modality through the self-attention mechanism to obtain the single-modal fusion features; cross-modal fusion: using the cross-attention mechanism to learn the relevant features of the cross-modality, and fusing the features of different modalities through the gating mechanism to obtain the final fusion features;

[0117] The intent recognition module is configured to: input the final fusion features into the multi-layer perceptron, and connect the softmax layer to output the classification results for intent recognition.

[0118] The beneficial effects of the present invention are:

[0119] 1. Solve the overfitting problem of the gaze estimation model: randomly select a subject in the training set in each training step and apply a meta-learning method, that is, a first-order gradient-based algorithm to update the model to alleviate the overfitting problem and enable the model to optimize new parameters.

[0120] 2. Efficient feature extraction: By applying the gaze feature extraction method based on full face appearance and random identity adversarial network, appearance-independent gaze estimation and extraction of key facial features are achieved.

[0121] 3. Optimize multimodal information fusion: Propose a fusion method of a gated neural network based on an attention mechanism, and improve the status of voice and sight features to help achieve a more comprehensive understanding of customer intent.

[0122] 4. Improve the accuracy and robustness of intent recognition: Improve the accuracy and robustness of intent recognition through a line of sight feature extraction method based on full face appearance and random identity adversarial network and a fusion method of a gated neural network based on an attention mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0123] Figure 1 It is a flowchart of the intention recognition method based on multimodal fusion of speech and sight;

[0124] Figure 2 A flowchart for training the FGEN model using a meta-learning strategy;

[0125] Figure 3 The overall block diagram of the gaze feature extraction method based on full face appearance and random identity adversarial network;

[0126] Figure 4 Schematic diagram of the self-attention calculation process;

[0127] Figure 5 Schematic diagram of the cross-attention calculation process. DETAILED DESCRIPTION

[0128] The present invention will be further defined below in conjunction with the accompanying drawings and embodiments, but is not limited thereto.

[0129] Example 1

[0130] An intention recognition method based on multimodal fusion of speech and sight, such as Figure 1 As shown, including:

[0131] Feature extraction: Extract text, speech, and gaze features from speech and face based on the pre-trained BERT model, Wav2vec 2.0 model, and self-trained FGEN (Face togaze encoder) model;

[0132] Multimodal representation: Considering the complementarity and consistency between different modalities, a modality-specific encoder and a modality-shared encoder are constructed to learn specific features and shared features respectively; including: 1) Modality-shared representation: A modality-shared encoder is constructed to learn cross-modal shared features; the modality-shared encoder transforms text, speech, and line-of-sight features into a unified feature space to obtain shared features, and uses the central moment difference (CMD) to minimize the similarity loss between shared features of different modalities; 2) Modality-specific representation: A modality-specific encoder is constructed to learn specific features of each modality; the modality-specific encoder transforms text, speech, and line-of-sight features into a specific feature space to obtain specific features; differential loss is used to ensure that the distribution of shared features and specific features of the same modality is different, and the distribution of specific features of different modalities is also different;

[0133] Multimodal fusion: An adaptive multimodal fusion method based on the attention mechanism of the gated neural network with speech and sight features is designed to dynamically adjust its model parameters or strategies to improve the performance and generalization ability of the model; including: 3) Intra-modal fusion: Through the self-attention mechanism (such as Figure 4 4) Cross-modal fusion: Use the cross-attention mechanism (as shown in Figure 5 As shown in Figure 2, it learns cross-modal related features and fuses the features of different modalities through a gating mechanism to obtain the final fused features;

[0134] Intent recognition: The final fusion features are input into the multi-layer perceptron (MLP) and connected to the softmax layer to output the classification results for intent recognition.

[0135] Example 2

[0136] An intention recognition method based on multimodal fusion of speech and sight, the difference is:

[0137] In this embodiment,

[0138] Modality shared encoder: is a structure used for multimodal learning, whose main function is to extract shared features across different modalities.

[0139] Modality-specific encoder: Unlike the modality-shared encoder, it is an encoder structure designed independently for each modality.

[0140] Decoder: It is a module used to map the features extracted by the encoder back to the required output space. The main task of the decoder is to restore the feature vector or feature map extracted by the encoder into a specific output format, such as generating a density map, a segmentation map, reconstructing an image, or performing a classification task.

[0141] Self-attention mechanism: It is a deep learning mechanism widely used in natural language processing, computer vision and other fields to capture the internal correlation of features. It learns the global dependencies within the data by letting each feature position pay attention to other positions in the input.

[0142] Cross-attention mechanism: It is a deep learning mechanism that is mainly used to handle feature interaction and fusion in multi-modal or multi-input scenarios. Unlike the self-attention mechanism, the cross-attention mechanism calculates attention weights between different modalities or different inputs in order to capture the association between different data sources to achieve feature complementarity.

[0143] Face Identity Classifier (Face Recognition Model): Receives a gaze representation vector as input and then predicts the face identity probability. The proposed architecture FGEN aims to extract a gaze representation that generalizes to appearance by predicting a uniform probability value for all faces.

[0144] Gaze prediction model: converts face images into gaze direction vectors (yaw and pitch), , where D is the normalized face image set, is the gaze vector, It is the face gaze feature vector extracted from the middle layer.

[0145] Extract text features based on the pre-trained BERT model; including:

[0146] The pre-trained BERT model can extract text semantic features and has become an indispensable module for natural language understanding tasks. Therefore, the pre-trained language model BERT is used to extract text features. Compared with Word2vec, the BERT model solves the polysemy problem by considering the context. At the same time, the different semantic features obtained through hierarchical learning provide rich feature selection for downstream tasks. Based on the pre-trained model BERT, the downstream tasks are fine-tuned and good performance can be achieved with only a small number of training samples. The speech in the interactive environment is captured by active acquisition, and the acquired text is converted into are all input into the BERT model. is the word vector in the text, m is the total number of word vectors, and the output of the last hidden layer of the BERT model represents the text feature , is the sequence length of the text utterance, is the feature dimension of each tag. The feature extraction process is as follows:

[0147] .

[0148] Extract speech features based on the pre-trained Wav2vec 2.0 model; including:

[0149] The Wav2vec 2.0 model is an unsupervised speech pre-training model released by Meta in 2020. The core idea is to build a self-supervised training target through vector quantization (VQ). The model performs better than standard acoustic features. Therefore, the pre-trained Wav2vec 2.0 model is used to extract speech features. Similarly, active acquisition is used to capture the speech of the interactive environment. , each speech segment are all input into the Wav2vec 2.0 model; the output of the last hidden layer of the Wav2vec 2.0 model represents the speech features , Indicates the sequence length of the speech segment, Represents the feature dimension of each tag. The feature extraction process is as follows:

[0150] ;

[0151] Among them, P represents the preprocessing operation, F represents the feature extractor, and Q represents the quantizer.

[0152] Extract line of sight features based on the pre-trained FGEN model; including:

[0153] People use their eyes to observe things of interest around them and receive important information. Therefore, the sight of the eyes looking at objects is an important non-verbal clue of human intention. Studies in recent years have shown that the information features of the entire face area can more accurately predict the position of the sight line. To this end, the present invention proposes a method for extracting sight line features (FGEN-Face to gaze encoder, FGEN) based on the full face appearance and random identity adversarial network, namely the FGEN model. This method does not require special infrared or depth equipment and only uses an ordinary camera. The present invention uses SWCNN (Switchable Convolutional Neural Network, a convolutional neural network model used in multimodal or multi-task scenarios) as a training baseline model, and expands the network architecture by connecting a face identity classifier (face recognition model) to achieve appearance-independent sight line estimation and extraction of key facial features; in addition, in order to solve the problem of model overfitting caused by insufficient training sets, a meta-learning strategy is used to train the FGEN model; that is, each iterative training randomly selects subjects from the face sight line dataset to form a meta-training dataset and a meta-test dataset to obtain the sight line feature embedding property. is the sequence length of key frames, is the feature dimension of each frame; Indicates a A real number matrix, including the feature embeddings of all key frames, where each row corresponds to the feature of a key frame and each column corresponds to the value of a specific feature dimension; as shown below:

[0154] ;

[0155] in, represents the face identity classifier, Denotes the gaze prediction model, and ML denotes the meta-learning strategy. It means that by embedding the gaze feature into the feature space, the unique information of the eye's gaze direction or gaze point is captured, thereby providing richer context and semantic understanding for visual tasks. FGEN() represents a function about FGEN, which is used to extract gaze features.

[0156] The specific architecture of the FGEN model is as follows Figure 2 shown.

[0157] The FGEN model is trained using a meta-learning strategy. The FGEN model training uses an adversarial strategy to achieve the independence of sight features and appearance subjects; Figure 3 As shown, including:

[0158] Face recognition parameters updated;

[0159] The parameters of the FGEN model are updated using line-of-sight loss and adversarial loss.

[0160] The gaze prediction model converts a face image into a gaze direction vector; as shown below:

[0161]

[0162] in, Represents the line of sight yaw and pitch vector labels, represents the facial gaze feature vector output by the FGEN model, represents the face gaze prediction model, Represents a face dataset;

[0163]

[0164] represents the face identity classifier, represents the recognition probability vector of each type of face in the training subjects;

[0165] Facial gaze invariant feature learning:

[0166] Face identity classifier The parameters of are updated according to the cross entropy loss function of the recognition probability and the label. The cross entropy loss function As shown below:

[0167]

[0168] in, is the identification label of each subject’s face, is the predicted recognition probability of the nth subject’s face, and N is the total number of training data; represents the logarithmic function;

[0169] Identity probability distribution equal feature learning:

[0170] Next, the FGEN model continues adversarial training; face gaze prediction model Optimize the opposite goal of the face identity classifier (i.e. optimize the parameters of the gaze prediction model so that the identity probability of all samples The distribution of is equal); thus, the adversarial loss of appearance generalization is defined as follows:

[0171] ;

[0172] ;

[0173] in, represents the accuracy of the predicted probability of the subject’s face, adversarial loss representing appearance generalization; , is a uniform distribution, k is the number of subjects in the training set; Indicates the norm; Use cosine similarity to construct a loss function to make the predicted constant probability Use uniform distribution;

[0174] Therefore, based on the appearance-related information of face images, a face gaze prediction model is generalized.

[0175] Gaze direction loss uses L1 loss , defined as:

[0176] ;

[0177] in, Represents the face gaze prediction model The predicted vector.

[0178] Using multi-objective loss functions Training is as follows:

[0179] ;

[0180] Among them, according to experience, When the parameters of the face gaze prediction model are updated, the face recognition model is frozen. Through the above adversarial learning, the limitations of appearance-based gaze estimation methods are alleviated by extracting the key information of appearance without the need for additional non-key labels.

[0181] In order to improve the generalization performance, a new learning strategy is introduced. This strategy is designed to compensate for the overfitting problem caused by person-specific appearance factors. Generally speaking, gaze estimation performs leave-one-out cross-validation, which only leaves one subject in the test dataset and uses the remaining subject for training. In this training setting, the model is biased by the appearance factors of the limited subjects used in the training dataset and is prone to overfitting. To avoid these problems, a new training strategy is constructed to randomly select a subject in the training set at each training step and apply a meta-learning method, i.e., a first-order gradient-based algorithm, to update the model; including:

[0182] First, k subjects are randomly selected from the training dataset and a new meta-training set is constructed using the face images of these k subjects , the meta-training set The sequence is input into the FGEN model, and the gradient vector is calculated during the optimization process as follows:

[0183]

[0184] in, is the initial weight of the line of sight prediction model, x is the number of meta-training iterations, represents the gradient vector calculated during the optimization of the line of sight prediction model, is the calculation function of the gradient vector;

[0185] Then, update the initial weights using the following equation :

[0186] ;

[0187] ;

[0188] in, is the step size used by the stochastic gradient descent (SGD) operation, are the weights of the gaze prediction model optimized by the meta-training process, refers to the weights updated by gradient descent;

[0189] Next, in the meta-training set Construct a meta-adaptation set with p non-overlapping topics in , and adapt the element to the iterative weights Updated to:

[0190] ;

[0191] After the meta-adaptation iteration, a new meta-adaptation set is created by selecting new topics , repeat the meta-adaptation process δ times and calculate the updated weights ;

[0192] Finally, the face gaze prediction model is based on the following formula To update:

[0193] ;

[0194] ;

[0195] in, y is the number of meta-adaptation iterations, For use The meta-adaptation process optimizes the weights of the face gaze prediction model; the present invention applies this process to alleviate the overfitting problem, allowing the model to optimize new parameters and prevent the model from updating in a dominant direction.

[0196] Line of sight feature extraction: Input the processed image and trained parameters into the FGEN model to obtain line of sight features .

[0197] There is complementarity and consistency between different modes. When people express their language, voice, and purpose, these factors contribute to a common motivation and goal, showing consistency between modes. At the same time, language, voice, and purpose have unique semantics, intonation, and expression, respectively, and show complementary characteristics. Therefore, modality-sharing and modality-specific encoders are designed to learn shared and specific features of text, voice, and sight. This representation provides a holistic view of the modality and lays the foundation for subsequent multimodal fusion. In the text feature , voice features and sight characteristics Before multimodal representation, a multi-headed Transformer is used for preprocessing , and .

[0198] Modal shared representation: including:

[0199] By constructing a modality-shared encoder To learn shared features across modalities, the modality-shared encoder , and Convert to a unified feature space to obtain shared features of text, speech, and sight respectively , and :

[0200] ;

[0201] ;

[0202] ;

[0203] In order to ensure that the shared features of different modalities of the same sample are similar, the central moment difference (CMD) is used to minimize the similarity loss; CMD contains higher-order moment information than KL divergence. Compared with the maximum mean difference (MMD), CMD reduces the amount of computation because there is no need to calculate the kernel matrix. CMD estimates the difference between two distributions by matching the order moment difference between them. Let X and Y be bounded random samples in the interval The probability distributions on are p and q respectively, and the CMD regularization matrix is defined as:

[0204] ;

[0205] In the formula, X and Y will be cross-bred into three different modes respectively, A and B represent the value range of X and Y, , is the empirical expectation vector of random sample X, , is the vector of all k-order sample central moments in X, and calculates the shared features between each pair of modes :

[0206] ;

[0207] Among them, t, and v are text, voice, and sight identifiers, respectively, and are the common features of mode m1 and mode m2 respectively; mode m1 and mode m2 are any two of the three modes respectively; intuitively speaking, minimizing the loss , will force the shared feature representation distributions of each pair of modalities to be similar.

[0208] Modality-specific representation: In order to understand the specific characteristics of different modes, a modality-specific encoder is constructed, including: , ,and , corresponding to text, speech and sight respectively; the modality-specific encoder converts , and Transform to unique feature space to obtain specific features , and :

[0209] ;

[0210] ;

[0211] ;

[0212] On the one hand, a good modality-specific feature must ensure that the distribution of shared features and specific features of the same modality is different. On the other hand, it is also necessary to ensure that the distribution of specific features is different between different modes. Therefore, the following differential loss is formed Calculation formula:

[0213] ;

[0214] in, is the square of the L2-norm. Intuitively, if and ; and are orthogonal, then the differential loss Minimum, t, and v are text, voice, and sight identifiers, respectively, and are the common features of mode m1 and mode m2 respectively; mode m1 and mode m2 are any two of the three modes respectively.

[0215] Before the multimodal fusion operation, a decoder is constructed to ensure that the shared features and specific features obtained by the encoder maintain the basic properties of the original feature space. To input shared features and specific features, where represents shared features, Indicates specific characteristics, Represents the decoder parameters; reconstructs the original feature space:

[0216] ;

[0217] ;

[0218] ;

[0219] in, 、 、 They respectively represent the essential information of the three modal features, and are new features that can ensure that their feature dimensions and distribution are consistent with the original input data.

[0220] The reconstruction error is estimated using the mean square error (MSE):

[0221] ;

[0222] in, is the square of the L2-norm, Regularization term to prevent overfitting, W is the decoder parameter; Representation reconstruction error refers to the error or loss incurred by the model when encoding the input data into a potential representation (such as feature embedding) and reconstructing it through the decoder.

[0223] Intra-modal fusion; including:

[0224] After obtaining the shared features and specific features of each modality through multimodal representation learning, they are connected to the self-attention mechanism model. The attention mechanism captures the correlation between shared features and specific features and obtains single-modal fusion features. Self-attention can well implement parallel computing and long-distance dependencies. Figure 4 As shown. The calculation formula of the self-attention mechanism is as follows:

[0225] ;

[0226] in, , and denote query, key, and value matrices respectively; , and are the parameters that need to be learned. is the dimension of K; for self-attention, the three matrices Q, K, and V come from the same input; the shared features and specific features of text, speech, and sight are connected separately and input into the self-attention mechanism to obtain the single-modal fusion features , and .

[0227] Cross-modal fusion; including:

[0228] After obtaining the unimodal fusion features, the cross-attention mechanism is used to learn text-to-view. , Eyes to Text , Text to Speech , Speech to Text , Voice to Sight , sight to voice The main difference between the cross-attention mechanism and the self-attention mechanism is that the input of the cross-attention mechanism comes from different sequences. The cross-attention mechanism asymmetrically combines two sequences, one as the input of Q and the other as the input of K and V; Figure 5 As shown, the calculation formula of the cross attention mechanism is as follows:

[0229] ;

[0230] Will and Combine the inputs into the sight gate to get the sight feature fusion weight ;Will and The combination of is input to the auditory gate to obtain the speech feature fusion weight ;Will and Combine the inputs into the text gate to get the text feature fusion weights ; The calculation formula is as follows:

[0231] ;

[0232] ;

[0233] ;

[0234] in, , and is the linear layer parameter, , and is the bias term;

[0235] According to the fusion weight, the line of sight features and voice features With text features Fusion is performed to obtain the final fusion feature h:

[0236] + .

[0237] Intent recognition; including:

[0238] After obtaining the final fusion feature h, h is input into the multi-layer perceptron (MLP) and connected to the softmax layer to obtain the classification result , as shown below:

[0239] ;

[0240] Where W and b represent the linear layer parameters and the bias term.

[0241] In order to achieve an end-to-end training process of multimodal representation, fusion and prediction, we jointly optimize the loss , , and cross entropy loss ; The final optimization target L is as follows:

[0242] ;

[0243] ;

[0244] in, , and is the weight that determines the contribution to the overall loss L; N is the number of training samples; and Represent the actual label distribution and predicted label distribution of sample i respectively; is L2 regularization, which reduces the overfitting of the model. W is the model parameter.

[0245] Compared with other models, the performance improvement of the proposed model is relatively stable, and in most cases the performance is higher than other models, with a higher macro-average score of accuracy and better domain adaptation and generalization capabilities.

[0246] Example 3

[0247] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of an intention recognition method based on multimodal fusion of speech and vision described in Example 1 or 2 are implemented.

[0248] Example 4

[0249] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of an intention recognition method based on multimodal fusion of speech and sight described in Example 1 or 2 are implemented.

[0250] Example 5

[0251] An intention recognition system based on multimodal fusion of speech and sight, comprising:

[0252] The feature extraction module is configured to extract text, speech, and gaze features from speech and face based on the pre-trained BERT model, Wav2vec 2.0 model, and the self-trained FGEN model;

[0253] The multimodal representation module is configured to include: modality sharing representation: constructing a modality sharing encoder to learn cross-modal shared features; the modality sharing encoder converts text, speech and sight features into a unified feature space to obtain shared features, and uses the central moment difference to minimize the similarity loss between shared features of different modalities; modality specific representation: constructing a modality specific encoder to learn specific features of each modality; the modality specific encoder converts text, speech and sight features into a specific feature space to obtain specific features; through differential loss, ensure that the distribution of shared features and specific features of the same modality is different, and the distribution of specific features of different modalities is also different;

[0254] The multimodal fusion module is configured to include: intra-modal fusion: fusing the shared features and specific features of each modality through the self-attention mechanism to obtain the single-modal fusion features; cross-modal fusion: using the cross-attention mechanism to learn the relevant features of the cross-modality, and fusing the features of different modalities through the gating mechanism to obtain the final fusion features;

[0255] The intent recognition module is configured to: input the final fusion features into the multi-layer perceptron, and connect the softmax layer to output the classification results for intent recognition.

Claims

1. An intention recognition method based on multimodal fusion of speech and sight, characterized in that: include: Feature extraction: Extract text, voice, and gaze features from voice and face based on the pre-trained BERT model, Wav2vec 2.0 model, and self-trained FGEN model; Multimodal representation: including: 1) Modality sharing representation: construct a modality sharing encoder to learn cross-modal shared features; the modality sharing encoder converts text, speech and sight features into a unified feature space to obtain shared features, and uses the central moment difference to minimize the similarity loss between shared features of different modalities; 2) Modality specific representation: construct a modality specific encoder to learn specific features of each modality; the modality specific encoder converts text, speech and sight features into a specific feature space to obtain specific features; through differential loss, ensure that the distribution of shared features and specific features of the same modality is different, and the distribution of specific features of different modalities is also different; Multimodal fusion: including: 3) Intra-modal fusion: The shared features and specific features of each modality are fused through the self-attention mechanism to obtain the single-modal fusion features; 4) Cross-modal fusion: Use the cross-attention mechanism to learn the relevant features of cross-modality, and fuse the features of different modalities through the gating mechanism to obtain the final fusion features; Intent recognition: The final fusion features are input into the multi-layer perceptron and connected to the softmax layer to output the classification results for intent recognition; Modal shared representation: including: In the text feature z t , voice features a and sight line feature z v Before performing multimodal representation, a multi-headed Transformer is used for preprocessing to obtain u t ,u a and u v ; By constructing a modality-shared encoder E c (u (t,a,v) θ c ) to learn cross-modal shared features, the modality shared encoder converts u t ,u a and u v Convert to a unified feature space to obtain shared features of text, speech, and sight respectively and Use the central moment difference CMD to minimize the similarity loss; let X and Y be bounded random samples in the interval [A, B] N The probability distributions on are p and q respectively, and the CMD regularization matrix CMD k The definition of (X,Y) is: In the formula, X and Y will be cross-bred into three different modes respectively, A and B represent the value range of X and Y. is the empirical expectation vector of random sample X, is the vector of central moments of all k-order samples in X, and the CMD between the shared features of each pair of patterns is calculated. k : Among them, t, a and v are text, voice and sight identifiers respectively. and are the common features of mode m1 and mode m2; mode m1 and mode m2 are any two of the three modes; minimize the loss L 2 / 3 , will force the shared feature representation distributions of each pair of modalities to be similar; Modality-specific representation: includes: building a modality-specific encoder, including: and Corresponding to text, speech and sight respectively; the modality-specific encoder converts u t 、u a and u v Transform to unique feature space to obtain specific features and The following differential loss L is formed : / ； Calculation formula: where ||·||" is the square of the L2-norm. If and and are orthogonal, then the differential loss L : / ； At minimum, t, a, and v are text, voice, and sight identifiers, respectively. and are the common features of mode m1 and mode m2 respectively; mode m1 and mode m2 are any two of the three modes respectively; Before the multimodal fusion operation, build a decoder To input shared features and specific features, where represents shared features, Represents a specific feature, θ: represents the decoder parameter; reconstructs the original feature space: in, They represent the essential information of three modal features respectively; Use the mean square error to estimate the reconstruction error: Among them, ||·||" is the square of the L2-norm, To prevent overfitting, W is the regularization term, and L is the decoder parameter. PQRST Representation reconstruction error refers to the error or loss incurred by the model when encoding the input data into a latent representation and reconstructing it through the decoder; Intra-modal fusion; including: The calculation formula of the self-attention mechanism is as follows: Where Q = W c X1, K = W k X1 and V = W v X1 represents query, key and value matrices respectively; W c , W k and W v is the parameter to be learned, d k is the dimension of K; for self-attention, the three matrices Q, K, and V come from the same input; the shared features and specific features of text, speech, and sight are connected respectively and input into the self-attention mechanism to obtain the single-modal fusion feature h t ,h a and h v ; Cross-modal fusion; including: After obtaining the unimodal fusion features, the cross-attention mechanism is used to learn text-to-view CA t-v , Line of sight to text CA v-t , Text-to-speech CA t-a , Speech to Text CA a-t , Voice to Line of Sight CA a-v , Line of sight to voice CA v-a The cross attention mechanism combines two sequences asymmetrically, one as the input of Q and the other as the input of K and V; the calculation formula of the cross attention mechanism is as follows: CA t-v and CA v-t Combine the inputs into the sight gate to get the sight feature fusion weight W ratB_v ; CA t-a and CA a-t The combination of is input to the auditory gate to obtain the speech feature fusion weight W ratB_a ; CA a-v and CA v-a Combine the inputs into the text gate to get the text feature fusion weight W ratB_t ; The calculation formula is as follows: W ratB_v =sigmoid(W v [CA t-v ;CA v-t ]+b v ); W ratB_a =sigmoid(W x [CA t-a ;CA a-t ]+b a ); W ratB_t =sigmoid(W t [CA a-v ;CA v-a ]+b y ); Among them, W v , W a and W y is the linear layer parameter, b v 、b a and b t is the bias term; According to the fusion weight, the line of sight feature h v and speech features h a With the text feature h t Fusion is performed to obtain the final fusion feature h: h=W ratB_v *h v +W ratB_a *h a +W ratB_t *h t ; Intent recognition; including: After obtaining the final fusion feature h, h is input into the multi-layer perceptron and connected to the softmax layer to obtain the classification result As shown below: Where W1 and b represent the linear layer parameters and bias terms; Intra-modal fusion; jointly optimizing the loss L 2 / 3 , L : / ；； , L ABcCD and the cross entropy loss L ta2k ; The final optimization target L is as follows: L=L ta2k +αL 2 / 3 +βL : / ；； +γL ABcCD ; Among them, α, β and γ are weights that determine the contribution to the overall loss L; N3 is the number of training samples; y l and Represent the actual label distribution and predicted label distribution of sample l respectively; is L2 regularization, and W is the decoder parameter.

2. According to claim 1, the method for intention recognition based on multimodal fusion of speech and sight is characterized in that: Extract line of sight features based on the self-trained FGEN model; including: Using SWCNN as the training baseline model, the network architecture is extended by connecting the face identity classifier to achieve appearance-independent gaze estimation and extraction of key facial features. The FGEN model is trained using a meta-learning strategy; that is, each iterative training randomly selects subjects from the face gaze dataset to form a meta-training dataset and a meta-test dataset to obtain the gaze feature embedding. l v is the sequence length of key frames, h v is the feature dimension of each frame; R represents a l v ×h v A real number matrix, including the feature embeddings of all key frames, where each row corresponds to the feature of a key frame and each column corresponds to the value of a specific feature dimension; as shown below: z v =FGEN(C / : ,E r ,ML) Among them, C / : represents the face identity classifier, E r represents the gaze prediction model and ML represents the meta-learning strategy.

3. The method for intention recognition based on multimodal fusion of speech and sight according to claim 1 is characterized in that: The FGEN model is trained using a meta-learning strategy. The FGEN model training uses an adversarial strategy to achieve the independence of sight features and appearance subjects; including: Face recognition parameters updated; Update the parameters of the FGEN model using line of sight loss and adversarial loss; The gaze prediction model converts a face image into a gaze direction vector; as shown below: (v r ,h r )=E r (D) Among them, v r Indicates the line of sight yaw and pitch vector labels, h r represents the facial gaze feature vector output by the FGEN model, E r represents the face gaze prediction model, D represents the face dataset; C / : represents the face identity classifier, Represents the recognition probability vector of each type of face in the training sample; Face Identity Classifier C / : The parameters of are updated according to the cross entropy loss function of the recognition probability and the label. The cross entropy loss function L / :c As shown below: Among them, p / : is the identification label of each face sample, is the predicted recognition probability of the nth face sample, N1 is the total number of training data; CE() represents the logarithmic function; Next, the FGEN model continues adversarial training; the face gaze prediction model E r Optimize the opposite goal of the face identity classifier; thus, the adversarial loss of appearance generalization is defined as follows: in, Indicates the accuracy of the subject's face prediction probability, L adv represents the adversarial loss of appearance generalization; e = [1 / k1,...1 / k1] d , is a uniform distribution, k1 is the number of subjects in the training set; ||e|| represents the norm; a loss function is constructed using cosine similarity to make the prediction of constant probability Use uniform distribution; The gaze direction loss uses L1 loss L r , defined as: in, Represents the face gaze prediction model E r The predicted vector; Using the multi-objective loss function L tCtal Training is as follows: L tCtal =the a:v L a:v +L r 。 4. The method for intention recognition based on multimodal fusion of speech and sight according to claim 3 is characterized in that: In each training step, a subject in the training set is randomly selected and a meta-learning method, i.e., a first-order gradient-based algorithm, is applied to update the model; including: First, k random samples are selected from the training dataset. " subjects, and using these k " subjects to construct a new meta-training set The meta-training set The sequence is input into the FGEN model, and the gradient vector is calculated during the optimization process as follows: in, is the initial weight of the line of sight prediction model, is the number of meta-training iterations, It represents the gradient vector calculated during the optimization of the sight prediction model, and U() is the calculation function of the gradient vector; Then, update the initial weights using the following equation where ε is the step size used by the stochastic gradient descent operation, φ is the weight of the gaze prediction model optimized by the meta-training process, refers to the weights updated by gradient descent; Next, in the meta-training set Construct a Meta-adaptation sets of non-overlapping topics And adapt the element to the iterative weights Updated to: After the meta-adaptation iteration, a new meta-adaptation set is created by selecting new topics Repeat the meta-adaptation process δ times and calculate the updated weights Finally, the face gaze prediction model E is calculated according to the following formula: r To update: Among them, y is the number of meta-adaptation iterations, For use The weights of the face gaze prediction model optimized by the meta-adaptation process; Line of sight feature extraction: The processed image and trained parameters are input into the FGEN model to obtain the line of sight feature z v =FGEN(C ·d ,E ± ,ML).

5. An intention recognition system based on multimodal fusion of speech and sight, characterized in that: include: The feature extraction module is configured to extract text, speech, and gaze features from speech and face based on the pre-trained BERT model, Wav2vec 2.0 model, and the self-trained FGEN model; The multimodal representation module is configured to include: modality sharing representation: constructing a modality sharing encoder to learn cross-modal shared features; the modality sharing encoder converts text, speech and sight features into a unified feature space to obtain shared features, and uses the central moment difference to minimize the similarity loss between shared features of different modalities; modality specific representation: constructing a modality specific encoder to learn specific features of each modality; the modality specific encoder converts text, speech and sight features into a specific feature space to obtain specific features; through differential loss, ensure that the distribution of shared features and specific features of the same modality is different, and the distribution of specific features of different modalities is also different; The multimodal fusion module is configured to include: intra-modal fusion: fusing the shared features and specific features of each modality through the self-attention mechanism to obtain the single-modal fusion features; cross-modal fusion: using the cross-attention mechanism to learn the relevant features of the cross-modality, and fusing the features of different modalities through the gating mechanism to obtain the final fusion features; The intent recognition module is configured to: input the final fusion features into the multi-layer perceptron and connect the softmax layer to output the classification results for intent recognition; Modal shared representation: including: In the text feature z t , voice features a and sight line feature z v Before performing multimodal representation, a multi-headed Transformer is used for preprocessing to obtain u t ,u a and u v ; By constructing a modality-shared encoder E c (u (t,a,v) θ c ) to learn cross-modal shared features, the modality shared encoder converts u t ,u a and u v Convert to a unified feature space to obtain shared features of text, speech, and sight respectively and Use the central moment difference CMD to minimize the similarity loss; let X and Y be bounded random samples in the interval [A, B] N The probability distributions on are p and q respectively, and the CMD regularization matrix CMD k The definition of (X,Y) is: In the formula, X and Y will be cross-bred into three different modes respectively, A and B represent the value range of X and Y. is the empirical expectation vector of random sample X, is the vector of central moments of all k-order samples in X, and the CMD between the shared features of each pair of patterns is calculated. k : Among them, t, a and v are text, voice and sight identifiers respectively. and are the common features of mode m1 and mode m2; mode m1 and mode m2 are any two of the three modes; minimize the loss L 2 / 3 , will force the shared feature representation distributions of each pair of modalities to be similar; Modality-specific representation: includes: building a modality-specific encoder, including: and Corresponding to text, speech and sight respectively; the modality-specific encoder converts u t 、u a and u v Transform to unique feature space to obtain specific features and The following differential loss L is formed : / ； Calculation formula: where ||·||" is the square of the L2-norm. If and and are orthogonal, then the differential loss L : / ； At minimum, t, a, and v are text, voice, and sight identifiers, respectively. and are the common features of mode m1 and mode m2 respectively; mode m1 and mode m2 are any two of the three modes respectively; Before the multimodal fusion operation, build a decoder To input shared features and specific features, where represents shared features, Represents a specific feature, θ: represents the decoder parameter; reconstructs the original feature space: in, They represent the essential information of three modal features respectively; Use the mean square error to estimate the reconstruction error: Among them, ||·||" is the square of the L2-norm, To prevent overfitting, W is the regularization term, and L is the decoder parameter. PQRST Representation reconstruction error refers to the error or loss incurred by the model when encoding the input data into a latent representation and reconstructing it through the decoder; Intra-modal fusion; including: The calculation formula of the self-attention mechanism is as follows: Where Q = W c X1, K = W k X1 and V = W v X1 represents query, key and value matrices respectively; W c , W k and W v is the parameter to be learned, d k is the dimension of K; for self-attention, the three matrices Q, K, and V come from the same input; the shared features and specific features of text, speech, and sight are connected respectively and input into the self-attention mechanism to obtain the single-modal fusion feature h t ,h a and h v ; Cross-modal fusion; including: After obtaining the unimodal fusion features, the cross-attention mechanism is used to learn text-to-view CA t-v , Line of sight to text CA v-t , Text-to-speech CA t-a , Speech to Text CA a-t , Voice to Line of Sight CA a-v , Line of sight to voice CA v-a The cross attention mechanism combines two sequences asymmetrically, one as the input of Q and the other as the input of K and V; the calculation formula of the cross attention mechanism is as follows: CA t-v and CA v-t Combine the inputs into the sight gate to get the sight feature fusion weight W ratB_v ; CA t-a and CA a-t The combination of is input to the auditory gate to obtain the speech feature fusion weight W ratB_a ; CA a-v and CA v-a Combine the inputs into the text gate to get the text feature fusion weight W ratB_t ; The calculation formula is as follows: W ratB_v =sigmoid(W v [CA t-v ;CA v-t ]+b v ); W ratB_a =sigmoid(W x [CA t-a ;CA a-t ]+b a ); W ratB_t =sigmoid(W t [CA a-v ;CA v-a ]+b y ); Among them, W v , W a and W y is the linear layer parameter, b v 、b a and b t is the bias term; According to the fusion weight, the line of sight feature h v and speech features h a With the text feature h t Fusion is performed to obtain the final fusion feature h: h=W ratB_v *h v +W ratB_a *h a +W ratB_t *h t ; Intent recognition; including: After obtaining the final fusion feature h, h is input into the multi-layer perceptron and connected to the softmax layer to obtain the classification result As shown below: Where W1 and b represent the linear layer parameters and bias terms; Intra-modal fusion; jointly optimizing the loss L 2 / 3 , L : / ；； , L ABcCD and the cross entropy loss L ta2k ; The final optimization target L is as follows: L=L ta2k +αL 2 / 3 +βL : / ；； +γL ABcCD ; Among them, α, β and γ are weights that determine the contribution to the overall loss L; N3 is the number of training samples; y l and Represent the actual label distribution and predicted label distribution of sample l respectively; is L2 regularization, and W is the decoder parameter.

Images