Multi-modal intention recognition method and system

Abstract

The invention relates to a multi-mode intention recognition method and system, and the method comprises the steps: carrying out the time domain and frequency domain enhancement of the features of text, video and audio modes, carrying out the splicing to obtain non-language mode fusion features, combining the features of an original text, modeling the time synchronization relation of audio-text and video-text, and carrying out the recognition of a multi-mode intention. Standardized audio features, video features and text features are obtained through context alignment processing; fusing the standardized features of the three modalities to obtain a fused feature vector, and mapping the fused feature vector back to the text modal space to be connected with the weighted residual error of the original text feature to obtain a fused semantic vector; extracting global semantic anchor points and mask positions from the fused semantic vector, and splicing the global semantic anchor points and the mask positions with the original text features and the fused semantic vector to obtain input features; and obtaining probability distribution of multiple intention categories by using the input features. Three types of heterogeneous modal input can be supported, and the accuracy and robustness of intention recognition are improved through fine-grained semantic supervision and enhancement strategies.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal intent recognition method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Human-computer interaction (HCI) is a core technology for products like intelligent voice assistants, conversational robots, and virtual customer service. It leverages natural language processing to identify user intent, helping products better respond to user needs. The accuracy of intent recognition directly determines the depth of understanding and quality of responses to user needs.

[0004] Traditional intent recognition methods primarily rely on text-based input, employing deep learning models to classify user input sentences and thereby determine intent. While these methods can achieve high accuracy with standard corpora and standardized expressions, they still face significant limitations in real-world interaction scenarios.

[0005] In real-world applications, user intent is not only reflected in verbal content but is often also expressed through non-verbal signals such as tone, intonation, facial expressions, and gestures. Traditional human-computer interaction technologies rely on converting single-language information into textual modalities. Text analysis methods struggle to capture implicit cues in these non-verbal modalities, which can lead to biased intent recognition and compromise user experience and the quality of decision-making.

[0006] Some existing technologies jointly model modal data such as speech, images (video) and text information. For example, a feature fusion strategy is used to splice or weight the features of each modality and then input them into the model for classification. However, non-linguistic modalities themselves have problems such as low signal-to-noise ratio, high semantic abstraction, and inconsistent temporal spans. Direct modeling is difficult to obtain stable and effective expression and is easily affected by background noise. In addition, there are differences in representation space and semantic structure between data of different modalities, and the lack of an effective alignment mechanism leads to semantic shift after fusion, affecting the accuracy of intent discrimination. Summary of the Invention

[0007] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a multimodal intent recognition method and system, which combines dual-domain feature enhancement and context comparison mechanism, has end-cloud collaborative processing capabilities, supports three types of heterogeneous modal inputs: audio, video and text, and improves the accuracy and robustness of intent recognition through fine-grained semantic supervision and enhancement strategies.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A first aspect of the present invention provides a multimodal intent recognition method, comprising the following steps:

[0010] Obtain the user input command, extract the text features, video features and audio features, perform time domain and frequency domain enhancement respectively, and obtain non-language modality fusion features through splicing;

[0011] The obtained non-language modality fusion features and the original text features are unified into a common feature dimension space, and the temporal synchronization relationship between audio and text and video and text is modeled respectively. The obtained audio similarity matrix and video similarity matrix are used to perform a weighted summation on the original audio features and video features to obtain audio features and video features that are semantically consistent with the text features. After normalization, standardized audio features, video features, and text features are obtained.

[0012] The standardized audio features, video features, and text features are fused separately to obtain the fused feature vector, which is then mapped back to the text modal space and connected with the weighted residual of the original text features to obtain the fused semantic vector.

[0013] Global anchor features and local mask features are extracted from the fused semantic vector and concatenated with the original text features and the fused semantic vector respectively to obtain positive sample pairs for comparative learning. By calculating the similarity between positive and negative sample pairs, semantic information is captured from both global and local perspectives to obtain the probability distribution of multiple intent categories.

[0014] Furthermore, the text features, video features, and audio features are enhanced in the time domain and frequency domain respectively, and then spliced ​​together to obtain non-language modality fusion features; specifically:

[0015] The original text features, video features, and audio features are formed into a three-dimensional tensor. Through parallel convolution branches, dynamic features at different time scales are extracted to obtain time domain output.

[0016] The original text features, video features, and audio features are Fourier transformed to obtain a complex spectrum, which is then decomposed into an amplitude spectrum and a phase spectrum. The enhanced signal is processed through the attention mechanism and mapped back to the time domain to obtain a frequency domain output.

[0017] The time domain output and frequency domain output are spliced ​​in the channel dimension to obtain non-language modality fusion features.

[0018] Furthermore, the obtained non-language modality fusion features and the original text features are unified into a common feature dimension space, and the temporal synchronization relationship between audio and text and video and text is modeled respectively. The obtained audio similarity matrix and video similarity matrix are used to perform weighted summation on the original audio features and video features to obtain audio features and video features that are semantically consistent with the text features. After normalization processing, standardized audio features, video features, and text features are obtained, including:

[0019] The original text features, video features and audio features are standardized and linearly projected respectively, and mapped to a unified semantic space to obtain the aligned text features T p , video features V p , audio feature A p ;

[0020] The aligned features are used in the LSTM model to obtain the aligned prediction output and calculate the probability distribution to obtain the actual audio similarity matrix A. s and the video similarity matrix V s .

[0021] According to the obtained similarity matrix, the audio and video features are weighted, and the features of each modality are weighted and summed by batch matrix multiplication to obtain the audio feature A aligned with the text feature b and video features V b .

[0022] Through linear projection, ensure that the audio feature A b and video features V b Keep consistent with the text features, and get A C and V C ; Normalize the audio, visual and text features to obtain the standardized text features C p , audio feature A p and visual features V p .

[0023] Furthermore, the standardized audio features, video features, and text features are fused separately to obtain a fused feature vector, which is then mapped back to the text modal space and connected with the weighted residual of the original text features to obtain a fused semantic vector. Specifically:

[0024] The Transformer encoder is used to fuse text features, video features, and audio features. Through the multi-head attention mechanism, the query, key, and value are respectively applied to the features corresponding to the three modalities. The interactive attention scores between the modalities are calculated, and normalization and feedforward mapping are performed at each layer to obtain the fused feature vector Z. Z is mapped back to the text modality space through the linear output layer and is weighted residual connected with the original text feature T to obtain the fused semantic vector Achieve semantic integration.

[0025] Furthermore, global anchor features and local mask features are extracted from the fused semantic vector and concatenated with the original text features and the fused semantic vector respectively to obtain positive sample pairs for contrastive learning. The similarity between the positive and negative sample pairs is then calculated to capture semantic information from both global and local perspectives. Specifically:

[0026] From the fusion semantic vector Extract the global anchor feature z cls and local mask feature z mask , build enhanced views;

[0027] The original text features, fused semantic features, z cls Splicing,input MAG-BERT to extract modality-aware representation;

[0028] The original text, z mask It is spliced ​​with the fused features and input into the BERT-processed mask view to generate 2N feature samples. The weighted cosine similarity is used to calculate the matching degree between the positive and negative sample pairs, and the NT-Xent loss function is introduced to shorten the distance between the positive sample pairs and increase the distance between the negative sample pairs, thus capturing semantic information from both global and local perspectives.

[0029] A second aspect of the present invention provides a system for implementing the above method, comprising:

[0030] The modality-aware feature enhancement module is configured to: obtain user input commands, extract text features, video features, and audio features, perform time domain and frequency domain enhancement respectively, and obtain non-language modality fusion features through splicing;

[0031] Multimodal fusion module, including context alignment submodule, semantic fusion module and classification module;

[0032] The context alignment submodule is configured to: unify the obtained non-language modality fusion features and the original text features into a common feature dimension space, model the temporal synchronization relationship between audio and text and video and text respectively, use the obtained audio similarity matrix and video similarity matrix to perform a weighted summation of the original audio features and video features to obtain audio features and video features that are semantically consistent with the text features, and then perform normalization processing to obtain standardized audio features, video features, and text features;

[0033] The semantic fusion module is configured to: fuse the standardized audio features, video features, and text features separately to obtain a fused feature vector, map it back to the text modal space, and concatenate it with the weighted residual of the original text features to obtain a fused semantic vector;

[0034] The classification module is configured to extract global anchor features and local mask features from the fused semantic vector, and concatenate them with the original text features and the fused semantic vector respectively to obtain positive sample pairs for comparative learning. By calculating the similarity between positive and negative sample pairs, the module captures semantic information from both global and local perspectives to obtain the probability distribution of multiple intent categories.

[0035] A third aspect of the present invention provides a computer-readable storage medium.

[0036] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the multimodal intent recognition method described above.

[0037] A fourth aspect of the present invention provides a computer device.

[0038] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the multimodal intent recognition method described above when executing the program.

[0039] Compared with the existing technology, one or more of the above technical solutions have the following beneficial effects:

[0040] 1. A time-frequency dual-channel feature extraction mechanism is proposed. By capturing multi-scale time-dependent features, enhancing key frequency bands and suppressing noise, it effectively improves the robustness and semantic expression quality of non-verbal modalities, provides more stable and richer semantic information for subsequent multimodal fusion and intent detection, and significantly enhances the model's intent recognition capability in complex scenarios.

[0041] 2. Construct a cross-modal semantic alignment mechanism. Through the standardization, linear projection, and similarity matrix calculation of multimodal features, the features of different modalities are mapped to a unified semantic space, thereby effectively alleviating the impact of modal heterogeneity on fusion quality. After obtaining the audio similarity matrix and the video similarity matrix, these similarity matrices are used to perform a weighted summation of the original audio and video features to form alignment features that are consistent with the semantics of the text modality. This is equivalent to "calibrating" the features of different modalities, making them semantically closer, thereby reducing the semantic offset caused by modal heterogeneity. It can significantly improve the effect of multimodal fusion, making the model more robust and accurate when processing complex semantic expressions, and providing high-quality input for subsequent semantic fusion.

[0042] 3. Introducing a multi-layer view comparison supervision method to optimize the consistency of intent expression and discrimination ability. The module extracts global semantic anchors and local mask features from the Transformer fused representation to generate enhanced views. These views capture semantic information from both global and local perspectives, providing the model with richer semantic details. Through the contrastive learning mechanism, the module calculates the similarity between positive and negative sample pairs, and introduces a contrastive loss function (such as NT-Xent loss) to shorten the distance between positive sample pairs and increase the distance between negative sample pairs. This mechanism enables the model to more accurately distinguish the semantic differences between different intents, thereby improving the discrimination ability.

[0043] 4. To address the problem that existing methods mostly use intent-level labels as supervisory signals and lack fine-grained modeling of local features and contextual relationships within the modality, we introduce fine-grained contrastive learning and contextual alignment to achieve in-depth modeling of local features and contextual relationships within the modality. Fine-grained contrastive learning not only focuses on global intent representation but also extracts local semantic features at masked locations to construct positive and negative sample pairs for contrastive learning. This mechanism can enhance the model's ability to learn local features within the modality, allowing the model to better capture semantic details within the modality during training. At the same time, contextual alignment achieves precise alignment and semantic fusion between modalities through normalization of multimodal features, linear projection, and LSTM-based similarity matrix calculation, further strengthening the modeling of contextual relationships between modalities. Through these mechanisms, the model's ability to discriminate and generalize complex semantic expressions can be effectively improved, enabling it to demonstrate higher accuracy and robustness in multimodal intent recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0045] Figure 1 is a schematic diagram of a multimodal intent recognition process provided by one or more embodiments of the present invention;

[0046] Figure 2 A schematic structural diagram of a modality-aware feature enhancement module provided for one or more embodiments of the present invention.

[0047] Figure 3 A schematic diagram of the structure of a multimodal fusion module provided for one or more embodiments of the present invention.

[0048] Figure 4 A schematic diagram of the structure of a comparative analysis module provided in one or more embodiments of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0050] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0051] Example 1:

[0052] During human-computer interaction, user intent is not only reflected in verbal content but is often also expressed through non-verbal signals such as tone, intonation, facial expressions, and gestures. Traditional human-computer interaction technologies rely on converting single verbal information into textual modalities. Text analysis methods struggle to capture implicit cues in these non-verbal modalities, which can lead to biased intent recognition and compromise the user experience and the quality of decision-making.

[0053] This embodiment addresses the problems of existing multimodal intent recognition methods, such as weak non-verbal modal expression capabilities, rough modal alignment, single fusion mechanism, and poor model generalization. It proposes a multimodal intent recognition method that combines dual-domain feature enhancement with a contextual comparison mechanism. This method has end-cloud collaborative processing capabilities and is divided into a modal perception feature enhancement module on the terminal side and a multimodal fusion and comparative analysis module on the server side. It supports three types of heterogeneous modal inputs: audio, video, and text, and improves the accuracy and robustness of intent recognition through fine-grained semantic supervision and enhancement strategies.

[0054] like Figure 1 As shown, the multimodal intent recognition method includes the following steps:

[0055] Obtain the user input command and extract the text features, video features and audio features;

[0056] The obtained video features and audio features are enhanced in the time domain and frequency domain respectively, and then spliced ​​together to obtain non-language modality fusion features;

[0057] The obtained non-language modality fusion features and the original text features are unified into a common feature dimension space, and the temporal synchronization relationship between audio and text and video and text is modeled respectively. The obtained audio similarity matrix and video similarity matrix are used to perform a weighted summation on the original audio features and video features to obtain audio features and video features that are semantically consistent with the text features. After normalization, standardized audio features, video features, and text features are obtained.

[0058] The standardized audio features, video features, and text features are fused separately to obtain the fused feature vector, which is then mapped back to the text modal space and connected with the weighted residual of the original text features to obtain the fused semantic vector.

[0059] Global anchor features and local mask features are extracted from the fused semantic vector and concatenated with the original text features and the fused semantic vector respectively to obtain positive sample pairs for comparative learning. By calculating the similarity between positive and negative sample pairs, semantic information is captured from both global and local perspectives to obtain the probability distribution of multiple intent categories.

[0060] In this embodiment, the modality perception feature enhancement module is deployed on the terminal side. This module is responsible for deep modeling of the input audio and video signals and outputting high-quality non-language modality features, such as Figure 2 This module specifically includes:

[0061] 1) Time domain processing unit:

[0062] The input is a 3D tensor B is the batch size, which represents the number of samples in the input signal. C is the number of channels in the signal, which represents the feature dimension or number of channels in the input signal. L is the sequence length, which represents the time series length of each signal.

[0063] Four convolution branches (Conv) are established using 1×1, 3×3, 7×7, and 9×9 convolution kernels respectively, and each branch is followed by a batch normalization layer (BN) and a ReLU activation function. Each branch can be represented as C i (h)=ReLU(BN(Conv(h))), each branch is progressive, and parallel one-dimensional convolution is performed on the input signal X to extract multi-scale time-dependent features. represents a residual connection.

[0064] The time domain processing unit enhances the ability to model dynamic changes in signals, effectively extracting key features such as rhythm, intonation, pauses in speech and action continuity in video.

[0065] 2) Frequency domain processing unit:

[0066] The input signal X is subjected to a fast Fourier transform (FFT) to obtain a complex spectrum F, which is then decomposed into an amplitude spectrum A and a phase spectrum φ. For A, channel compression is first performed using a 1×1 convolution. Subsequently, global average pooling and a two-layer fully connected network are used to learn channel modulation weights, enhancing key signal bands and suppressing background interference. For φ, a similar processing mechanism is employed, incorporating channel attention to fine-tune the phase distribution. Finally, an inverse Fourier transform (IFFT) is used to reconstruct the modified spectrum back into the enhanced time domain signal.

[0067] In this embodiment, the specific process of the frequency domain processing unit is as follows:

[0068] The input signal X is transformed by fast Fourier transform (FFT) to obtain the complex spectrum Where B is the signal length. Then it is decomposed into the amplitude spectrum and phase spectrum φ=arctan2(Im(X f ),Re(X f )). Arctan2 is the four-quadrant inverse tangent function, Im(X f ) and Re(X f ) represent the imaginary and real parts of the complex spectrum respectively. Using a 1×1 convolution to compress the amplitude spectrum, we can get

[0069] Generate modulation weights through global average pooling and two-layer fully connected network σ is the Sigmoid activation function. After the 1×1 convolution, the bias term is added to obtain the modified amplitude spectrum. Compressing the phase spectrum using a 1×1 convolution yields

[0070] The modulation weights α = σ × σ is the Sigmoid activation function. After the 1×1 convolution, the bias term is added to obtain the modified phase spectrum. ⊙ represents element-by-element multiplication.

[0071] Obtained by complex reconstruction and Represents the cosine and sine parts of the phase spectrum. The spectrum signal is obtained by inverse fast Fourier transform (IFFT) to enhance the signal where j is the imaginary unit j 2 =-1, L is the length of the signal, X p [f] is the value of the spectrum signal at the fth frequency point.

[0072] 3) Dual-domain fusion unit:

[0073] The time-domain output and the frequency-domain enhanced output are concatenated in the channel dimension and fused using 1×1 convolution for dimensionality reduction to generate the final non-language modality fusion feature. This feature integrates multi-scale temporal structure information and key frequency response patterns, significantly improving the contribution and robustness of non-language modalities in downstream semantic modeling tasks.

[0074] In this embodiment, the specific process of the dual-domain fusion unit is as follows: s and y f Perform channel-dimensional splicing and send it to 1×1 convolution to obtain the fused feature vector y dd =c f ([y s ||y f ]).y dd Contains video features Vdd and audio feature A dd ,|| indicates splicing.

[0075] This solution effectively addresses the low signal-to-noise ratio, high semantic abstraction, and inconsistent temporal spans of non-verbal modalities through the time-frequency fusion mechanism within the modality-aware feature enhancement module. The proposed dual-channel time-frequency feature extraction mechanism, which captures multi-scale temporally dependent features through a time-domain processing unit and enhances key frequency bands and suppresses noise through a frequency-domain processing unit, effectively improves the robustness and semantic expression quality of non-verbal modalities. This provides more stable and richer semantic information for subsequent multimodal fusion and intent detection, significantly enhancing the model's intent recognition capabilities in complex scenarios.

[0076] Specifically:

[0077] The time domain processing unit uses multi-scale convolution kernels to extract dynamic features at different time scales, which can capture key information such as rhythm, intonation, pauses in speech and action continuity in video. At the same time, it alleviates the gradient vanishing problem through residual connections and enhances the ability to model dynamic changes of signals.

[0078] The frequency domain processing unit converts the signal into the frequency domain through fast Fourier transform, and performs channel attention mechanism on the amplitude spectrum and phase spectrum respectively to enhance the key frequency bands and suppress noise. At the same time, it introduces a learnable phase offset to optimize timing alignment and further improve the robustness of the signal.

[0079] Finally, the dual-domain fusion unit concatenates and weightedly fuses the time domain features and frequency domain features to generate high-quality non-language modality features.

[0080] This time-frequency dual-channel feature extraction mechanism can not only effectively improve the robustness of non-language modalities, but also enhance the quality of their semantic expression, providing more stable and richer semantic information for subsequent multimodal fusion and intent recognition.

[0081] The multimodal fusion and comparative analysis module is deployed on the server side to receive y dd It is aligned, fused and optimized with text modal features; it includes: context alignment submodule, semantic fusion module, contrastive learning module and classification module.

[0082] 1) Context alignment submodule.

[0083] The context alignment module is responsible for unifying the modeling of enhanced audio and video features uploaded by the terminal with locally extracted text features, completing the contextual alignment, semantic fusion, and joint expression of multimodal information. Its purpose is to establish effective mapping relationships between modalities in heterogeneous feature spaces, explore modal collaborative semantics, and form a unified semantic representation with cross-modal consistency.

[0084] The context alignment submodule first aligns the text and enhanced audio features A dd With video features V dd Normalization and linear projection are performed separately to unify the data into a common feature dimensional space. A bidirectional LSTM is then used to model the temporal synchronization between audio and text and video and text, respectively, outputting aligned prediction sequences. (LSTM is a specialized recurrent neural network.) A Softmax operation is performed to obtain the audio and video similarity matrices.

[0085] Next, the audio and video similarity matrices are used to perform a weighted summation of the original audio and video features to form aligned features that are semantically consistent with the text modality. A linear layer is then used to map the aligned features to a unified embedding dimension and renormalize them to obtain a standardized representation of the audio, video, and text before fusion. This process ensures that the semantic structures of the modalities converge, providing a foundation for subsequent interaction modeling.

[0086] In this embodiment, the specific process of the context alignment module is as follows:

[0087] like Figure 3 As shown in the figure, the text, audio, and video features are standardized and linearly projected respectively, and mapped to a unified semantic space to obtain aligned text features T p =Norm(Linear(Norm(T))), video feature V p =Norm(Linear(Norm(V dd ), audio feature A p =Norm(Linear(Norm(A dd ))), Linear is the linear layer, and Norm is the normalization layer.

[0088] The audio features, video features, and text features are fed into the LSTM model to obtain the aligned prediction output. The probability distribution is calculated using the Softmax activation function, and blank categories are removed to obtain the actual audio similarity matrix A. s =Softmax(LSTM(A p )) and video similarity matrix V s =Softmax(LSTM(V p )).

[0089] According to the obtained similarity matrix, the audio and video features are weighted, and the features of each modality are weighted summed by batch matrix multiplication (BMM) to obtain the audio A aligned with the text features. b =BMM(P T ,A s ) and video features V b=BMM(P T ,V s ).

[0090] After obtaining the aligned audio and video features, linear projection is performed to ensure that they are consistent with the text features to obtain A C =Linear(A b ) and V C =Linear(V b ). Then the audio, visual and text features are normalized. Let the input T∈{C,A C ,V C}, T n =Norm(T). Then pass through a linear layer to get T l =Linear(T n ), and then normalize it again to finally get the transformed text feature C p , audio feature A p and visual features V p .

[0091] This solution builds a cross-modal semantic alignment mechanism. Through multimodal feature normalization, linear projection, and LSTM-based similarity matrix calculation, it maps features from different modalities into a unified semantic space, effectively mitigating the impact of modal heterogeneity on fusion quality. This mechanism can significantly improve the effectiveness of multimodal fusion, making the model more robust and accurate when processing complex semantic expressions.

[0092] Different modalities have different representation spaces and semantic structures. The lack of an effective alignment mechanism can lead to semantic drift after fusion, affecting the accuracy of intent identification. This solution addresses the lack of an effective alignment mechanism in existing multimodal fusion techniques and proposes an innovative contextual alignment module.

[0093] The differences in feature representation space and semantic structure between different modalities (such as text, audio, and video) are called modality heterogeneity. This difference can lead to semantic drift during the fusion process, which in turn affects the accuracy of intent discrimination. For example, text modalities are typically discrete sequences of symbols, while audio and video modalities are continuous signals, with different feature dimensions, time spans, and semantic expressions. This heterogeneity makes it difficult to find an effective mapping relationship when directly fusing features from different modalities, resulting in the fused features being unable to accurately express the semantic information of the original modalities.

[0094] The context alignment module in this solution effectively mitigates the impact of modal heterogeneity on fusion quality through a series of innovative mechanisms. Specifically, the module first normalizes and linearly projects text, audio, and video features into a unified semantic space. This process effectively finds a common "language" for features from different modalities, enabling effective comparison and interaction along the same dimension.

[0095] Secondly, a bidirectional LSTM is used to model the temporal synchronization between audio and text and video and text, respectively, and output aligned prediction sequences. This approach captures the dynamic changes in the temporal dimension of different modalities, further enhancing the semantic consistency between the modalities.

[0096] After obtaining the audio and video similarity matrices through the Softmax operation, these similarity matrices are used to perform a weighted summation of the original audio and video features to form aligned features that are semantically consistent with the text modality. This step is equivalent to "calibrating" the features of different modalities, making them more semantically similar, thereby reducing the semantic shift caused by modal heterogeneity. Finally, a linear layer is used to map the aligned features to a unified embedding dimension and renormalize them to obtain a standardized representation of the audio, video, and text before fusion. This process further optimizes the feature representation and provides high-quality input for subsequent semantic fusion.

[0097] 2) Semantic fusion module.

[0098] The semantic fusion module uses a standard Transformer encoder to fuse and model the features of text, video, and audio modalities. Through a multi-head attention mechanism, the query (Q), key (K), and value (V) are applied to each of the three modalities, calculating the interactive attention scores between the modalities and performing normalization and feed-forward mapping at each layer. The fused feature vector Z represents the modal semantic aggregation result.

[0099] Map Z back to the text modality space through the linear output layer and perform a weighted residual connection with the original text feature T to obtain Achieve semantic integration. The fused semantic vector is used as the final input to the intent recognition task.

[0100] The fusion module supports flexible plug-in and plug-in of new modal channels (such as images, gestures, etc.), and has adjustable structure, adjustable number of layers, and parameter sharing capabilities, adapting to intelligent interaction methods of various business scales.

[0101] In this embodiment, the specific process of the semantic fusion module is as follows:

[0102] The text feature C p , audio feature A p and visual features Vp Cross-modal semantic fusion is performed through the Transformer encoder. The interaction weights between different modalities are calculated through the Transformer's multi-head attention mechanism: in They are text features T p , audio feature A p and visual features V p The corresponding query weight matrix. are the key weight matrices respectively. are value weight matrices respectively, and the fused feature Z = MHA(Q, K, V) is obtained.

[0103] The fused features are layer normalized to stabilize model training and improve performance. The normalized features are input into a feedforward neural network for nonlinear transformation to further enhance the expressive power of the features. The output of the feedforward network is layer normalized again to obtain C g After cross-modal context fusion, the output projection layer returns to the text feature space and performs weighted fusion with the original text features to obtain the final fused semantic features.

[0104] 3) Comparative learning module.

[0105] The contrastive learning module is fused from the Transformer representation. Extraction: Global anchor feature z cls : Select the hidden state corresponding to the CLS tag of the 8th layer as the global intent expression; local mask feature z mask : Select all randomly masked sequence positions in the final layer to capture local semantic details. Then we construct two types of enhanced views: original text features, fused semantic features, z cls Splicing, input MAG-BERT to extract modality perception representation; the original text, z mask Concatenate with the fused features and input BERT to process the mask view.

[0106] BERT is a Transformer-based bidirectional language model. Pre-trained on large amounts of text data, it captures rich linguistic features. MAG-BERT is an extension of BERT that combines multimodal information and graph structures to enhance the model's understanding and processing of cross-modal data. Ultimately, 2N feature samples are generated. Weighted cosine similarity is used to calculate the matching between positive and negative sample pairs, and the NT-Xent loss function is introduced.

[0107] In this embodiment, the specific process of the comparative learning module is as follows:

[0108] like Figure 4 As shown, from Extract the global semantic anchor z cls With mask position z mask . The original text features and the fusion semantic features and the global semantic anchor z cls Concatenated and processed by MAG-BERT

[0109] The original text features and the fusion semantic features and Z mask For splicing, use the BERT encoder layer to get

[0110] Using Z C and Z D Generate enhanced views. Each sample generates two enhanced views, and a total of 2N features are obtained: Where N is the batch size. and as positive sample pairs, and the other samples are regarded as negative samples.

[0111] Using cosine similarity weighted fusion: α is a weight parameter. The NT-Xent loss is used to constrain the feature space, bringing positive pairs closer together and pushing negative pairs apart: N is the number of samples and τ is the temperature parameter.

[0112] This solution introduces a multi-layer view comparison supervision method to optimize the consistency and discriminability of intent expression. It extracts global semantic anchors and local mask features from the fused Transformer representation to generate enhanced views. These views capture semantic information from both global and local perspectives, providing the model with richer semantic details.

[0113] Through contrastive learning, the similarity between positive and negative sample pairs is calculated, and a contrastive loss function (such as NT-Xent loss) is introduced to bring positive pairs closer together and push negative pairs further apart. This mechanism enables the model to more accurately distinguish the semantic differences between different intents, thereby improving its discriminative capabilities.

[0114] To address the problem that existing methods mostly use intent-level labels as supervisory signals and lack fine-grained modeling of local features and contextual relationships within the modality, this paper introduces a fine-grained contrastive learning module and a context alignment module to achieve in-depth modeling of local features and contextual relationships within the modality. The fine-grained contrastive learning module not only focuses on the global intent representation, but also extracts local semantic features at the mask position to construct positive and negative sample pairs for contrastive learning. This mechanism can enhance the model's ability to learn local features within the modality, allowing the model to better capture the semantic details within the modality during training. At the same time, the context alignment module achieves precise alignment and semantic fusion between modalities through the standardization of multimodal features, linear projection, and LSTM-based similarity matrix calculation, further strengthening the modeling of contextual relationships between modalities. Through these innovative mechanisms, this solution can effectively improve the model's ability to discriminate and generalize complex semantic expressions, enabling it to demonstrate higher accuracy and robustness in multimodal intent recognition tasks.

[0115] 4) Classification module:

[0116] In this solution, the output of intent classification is the probability distribution of multi-category intent predictions, which reflects the likelihood that the user input intent will be classified into each category.

[0117] Specifically, this process uses the BERT model to perform deep semantic encoding of the input text and generate masked views. Using BERT's masked language model mechanism, these masked views randomly mask out portions of vocabulary and predict their original content, thereby learning rich contextual information. These masked views, after being processed by BERT's multi-layer Transformer architecture, are able to capture the semantic details and contextual relationships in the text.

[0118] Next, the mask view processed by BERT is average pooled. This process is to reduce the dimension of the feature vector of the mask view by average pooling to extract more representative features.

[0119] These features are then input into the linear layer, which further transforms and classifies the extracted features and finally outputs the predicted probability distribution for each intent category.

[0120] The final result is obtained through the softmax classifier. The softmax function is a commonly used classification function that converts the raw scores of each category output by the linear layer into probability values ​​so that the sum of these probability values ​​is 1.

[0121] The probability value for each category thus represents the likelihood that the user's input intent belongs to that category. Through the softmax classifier, the model can clearly identify the most likely category for the user's intent while also providing the probability distribution of other categories, providing a more comprehensive reference for subsequent decision-making.

[0122] In this embodiment, the specific process of the classification module is: Z D After average pooling, the data is input to the linear layer, which outputs the probability distribution of multi-category intent prediction. The final result is obtained through the softmax classifier.

[0123] All of the aforementioned modules are built using a standard neural network architecture, offering excellent scalability and portability, making them suitable for online deployment, edge inference, and cross-device deployment. Through module decoupling and standardized communication interfaces, they are adaptable to distributed deployment and heterogeneous modal input, resulting in a high degree of engineering practicality.

[0124] Existing multimodal intent recognition technologies struggle to operate stably in complex environments and are unable to simultaneously address non-verbal modality enhancement, modal semantic alignment, and deep fusion modeling. They also suffer from low recognition accuracy, poor robustness, and weak model generalization. Therefore, this proposal proposes a novel multimodal intent recognition method that possesses modality awareness, context fusion, and semantic comparison optimization to meet the practical needs for efficient semantic understanding in multimodal interaction scenarios.

[0125] Example 2:

[0126] A system for implementing the above method includes:

[0127] The modality-aware feature enhancement module is configured to: obtain user input commands, extract text features, video features, and audio features, perform time domain and frequency domain enhancement respectively, and obtain non-language modality fusion features through splicing;

[0128] Multimodal fusion module, including context alignment submodule, semantic fusion module and classification module;

[0129] The context alignment submodule is configured to: unify the obtained non-language modality fusion features and the original text features into a common feature dimension space, model the temporal synchronization relationship between audio and text and video and text respectively, use the obtained audio similarity matrix and video similarity matrix to perform a weighted summation of the original audio features and video features to obtain audio features and video features that are semantically consistent with the text features, and then perform normalization processing to obtain standardized audio features, video features, and text features;

[0130] The semantic fusion module is configured to: fuse the standardized audio features, video features, and text features separately to obtain a fused feature vector, map it back to the text modal space, and concatenate it with the weighted residual of the original text features to obtain a fused semantic vector;

[0131] The classification module is configured to extract global anchor features and local mask features from the fused semantic vector, and concatenate them with the original text features and the fused semantic vector respectively to obtain positive sample pairs for comparative learning. By calculating the similarity between positive and negative sample pairs, the module captures semantic information from both global and local perspectives to obtain the probability distribution of multiple intent categories.

[0132] Example 3:

[0133] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the multimodal intent recognition method in the above-mentioned embodiment 1 are implemented.

[0134] Example 4:

[0135] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the multimodal intent recognition method in the first embodiment described above are implemented.

[0136] The steps or modules involved in Examples 2 to 4 above correspond to those in Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media that includes one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and cause the processor to perform any method of the present invention.

[0137] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A multimodal intent recognition method, characterized in that: The following steps are involved: Obtain the user input command, extract the text features, video features and audio features, perform time domain and frequency domain enhancement respectively, and obtain non-language modality fusion features through splicing; The obtained non-language modality fusion features and the original text features are unified into a common feature dimension space, and the temporal synchronization relationship between audio and text and video and text is modeled respectively. The obtained audio similarity matrix and video similarity matrix are used to perform a weighted summation on the original audio features and video features to obtain audio features and video features that are semantically consistent with the text features. After normalization, standardized audio features, video features, and text features are obtained. The standardized audio features, video features, and text features are fused separately to obtain the fused feature vector, which is then mapped back to the text modal space and connected with the weighted residual of the original text features to obtain the fused semantic vector. Global anchor features and local mask features are extracted from the fused semantic vector and concatenated with the original text features and the fused semantic vector respectively to obtain positive sample pairs for comparative learning. By calculating the similarity between positive and negative sample pairs, semantic information is captured from both global and local perspectives to obtain the probability distribution of multiple intent categories.

2. The multimodal intent recognition method according to claim 1, wherein: Text features, video features, and audio features are enhanced in the time domain and frequency domain respectively, and then concatenated to obtain non-language modality fusion features. Specifically: The original text features, video features, and audio features are formed into a three-dimensional tensor. Through parallel convolution branches, dynamic features at different time scales are extracted to obtain time domain output. The original text features, video features, and audio features are Fourier transformed to obtain a complex spectrum, which is then decomposed into an amplitude spectrum and a phase spectrum. The enhanced signal is processed through the attention mechanism and mapped back to the time domain to obtain a frequency domain output. The time domain output and frequency domain output are spliced ​​in the channel dimension to obtain non-language modality fusion features.

3. The multimodal intent recognition method according to claim 1, wherein: The obtained non-language modality fusion features include enhanced video features and audio features. The enhanced video features and audio features are normalized and linearly projected with the original text features, and mapped to a unified semantic space to obtain aligned text features T p , video features V p , audio feature A p ; Alignment The features of the LSTM model are used to obtain the aligned prediction output and calculate the probability distribution to obtain the actual audio similarity matrix A. s and the video similarity matrix V s .

4. The multimodal intention recognition method according to claim 3, wherein: According to the obtained similarity matrix, the audio and video features are weighted, and the features of each modality are weighted and summed by batch matrix multiplication to obtain the audio feature A aligned with the text feature b and video features V b .

5. The multimodal intention recognition method according to claim 4, wherein: By linear projection, ensure that the audio feature A b and video features V b Keep consistent with the text features, and get A C and V C ; Normalize the audio, visual and text features to obtain the standardized text features C p , audio feature A p and visual features V p .

6. The multimodal intention recognition method according to claim 1, wherein: The standardized audio features, video features, and text features are fused separately to obtain the fused feature vector, which is then mapped back to the text modal space and connected with the weighted residual of the original text features to obtain the fused semantic vector. Specifically: The Transformer encoder is used to fuse text features, video features, and audio features. Through the multi-head attention mechanism, the query, key, and value are respectively applied to the features corresponding to the three modalities. The interactive attention scores between the modalities are calculated, and normalization and feedforward mapping are performed at each layer to obtain the fused feature vector Z. Z is mapped back to the text modality space through the linear output layer and is weighted residual connected with the original text feature T to obtain the fused semantic vector Achieve semantic integration.

7. The multimodal intention recognition method according to claim 1, wherein: From the fusion semantic vector Extract the global anchor feature z cls and local mask feature z mask , build enhanced views; The original text features, fused semantic features, z cls Splicing,input MAG-BERT to extract modality-aware representation; The original text, z mask It is spliced ​​with the fused features and input into the BERT-processed mask view to generate 2N feature samples. The weighted cosine similarity is used to calculate the matching degree between the positive and negative sample pairs, and the NT-Xent loss function is introduced to shorten the distance between the positive sample pairs and increase the distance between the negative sample pairs, thus capturing semantic information from both global and local perspectives.

8. A multimodal intention recognition method system, characterized in that: include: The modality-aware feature enhancement module is configured to: obtain user input commands, extract text features, video features, and audio features, perform time domain and frequency domain enhancement respectively, and obtain non-language modality fusion features through splicing; Multimodal fusion module, including context alignment submodule, semantic fusion module and classification module; The context alignment submodule is configured to: unify the obtained non-language modality fusion features and the original text features into a common feature dimension space, model the temporal synchronization relationship between audio and text and video and text respectively, use the obtained audio similarity matrix and video similarity matrix to perform a weighted summation of the original audio features and video features to obtain audio features and video features that are semantically consistent with the text features, and then perform normalization processing to obtain standardized audio features, video features, and text features; The semantic fusion module is configured to: fuse the standardized audio features, video features, and text features separately to obtain a fused feature vector, map it back to the text modal space, and concatenate it with the weighted residual of the original text features to obtain a fused semantic vector; The classification module is configured to extract global anchor features and local mask features from the fused semantic vector, and concatenate them with the original text features and the fused semantic vector respectively to obtain positive sample pairs for comparative learning. By calculating the similarity between positive and negative sample pairs, the module captures semantic information from both global and local perspectives to obtain the probability distribution of multiple intent categories.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal intent recognition method according to any one of claims 1 to 7.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the multimodal intent recognition method according to any one of claims 1 to 7 are implemented.

Images