Multi-modal intention recognition method and system based on consistency and difference decoupling

Abstract

The invention relates to a multi-mode intention recognition method and system based on consistency and difference decoupling, and the method comprises the steps: carrying out the consistency modeling and difference modeling of the features of text, video and audio modes, and enabling the consistency modeling to be used for extracting common information expressing similar meanings in each mode; the difference modeling is used for retaining information which is unique in each mode but possibly has important supplementary value; by constructing a shared-private feature space, carrying out explicit modeling on common information in the consistency features and specific information in the difference features, and fusing the consistency features and the difference features; and based on the fused feature representation, judging the real intention of the user through a classifier, and outputting a result. Through consistency and difference collaborative modeling, in combination with a shared-private feature space, semantic consistency and difference among all modalities are explicitly modeled, and the accuracy and robustness of intention recognition are fundamentally improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal intent recognition method and system based on decoupling of consistency and difference. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Human-computer interaction (HCI) is a core technology for products like intelligent voice assistants, conversational robots, and virtual customer service. It leverages natural language processing to identify user intent, helping products better respond to user needs. The accuracy of intent recognition directly determines the depth of understanding and quality of responses to user needs.

[0004] Traditional intent recognition methods primarily rely on text-based input, employing deep learning models to classify user input sentences and thereby determine intent. While these methods can achieve high accuracy with standard corpora and standardized expressions, they still face significant limitations in real-world interaction scenarios.

[0005] In real-world applications, user intent is not only reflected in verbal content but is often also expressed through non-verbal signals such as tone, intonation, facial expressions, and gestures. Traditional human-computer interaction technologies rely on converting single-language information into textual modalities. Text analysis methods struggle to capture implicit cues in these non-verbal modalities, which can lead to biased intent recognition and compromise user experience and the quality of decision-making.

[0006] Some existing technologies jointly model modal data such as speech, images (video) and text information. Mainstream multimodal intent recognition methods can be divided into three categories:

[0007] (1) Early fusion method: immediately after feature extraction, the features of each modality are concatenated to form a joint vector and sent to the downstream classifier;

[0008] (2) Late fusion method: first train each modality network independently, and then fuse the prediction probabilities of each modality;

[0009] (3) Mid-term fusion method: Use attention mechanism, tensor fusion, cross-modal alignment mechanism, etc. to model the deep interaction between modal features.

[0010] Although these methods have alleviated the problem of information inconsistency between modalities to a certain extent, they still face the following technical challenges:

[0011] (1) The modality consistency problem is not fully modeled: Different modalities (such as intonation in speech and body movements in video) often have the same semantic trends as text, that is, they are consistent. If the consistency is not explicitly modeled, redundant or contradictory semantic interference may be introduced during the fusion process.

[0012] (2) Modal differences are ignored: In some scenarios, information expressed in different modalities is complementary or conflicting. For example, a user's textual content of "thank you" expresses gratitude, but the voice, tone, and facial expression may actually express irony. The lack of differential modeling will lead to a decline in the model's ability to understand complex intent;

[0013] (3) Modal features lack semantic separation and decoupling mechanisms: Existing methods mostly use a unified space for modal alignment, which cannot take into account both the "shared semantics" and "private semantics" semantic subspaces, ultimately resulting in a double reduction in the accuracy of semantic expression and the ability to interpret it;

[0014] (4) Existing model structures lack modularity and portability: Many models have complex structures and high coupling, making them difficult to directly migrate and reuse in different tasks or platforms.

[0015] In summary, the current multimodal intent recognition algorithm has an unclear structure, unsatisfactory expressiveness, and its interpretability and module reusability cannot meet the requirements. Summary of the Invention

[0016] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a multimodal intent recognition method and system based on consistency and difference decoupling. By introducing modal consistency modeling and difference modeling mechanisms, the shared semantics and private semantics in multimodal data are extracted respectively, effectively alleviating the redundant conflict problem in the modal fusion process; at the same time, a semantic decoupling structure is designed to clearly distinguish between shared and private semantic subspaces, thereby improving the model's expressiveness and interpretability; in addition, the overall system structure adopts a modular design with good portability and cross-task reuse capabilities. Through the collaborative modeling of consistency and difference, the information structure between multimodalities is explicitly decoupled, fundamentally improving the accuracy and robustness of intent recognition.

[0017] In order to achieve the above object, the present invention adopts the following technical solutions:

[0018] A first aspect of the present invention provides a multimodal intent recognition method based on consistency and difference decoupling, comprising the following steps:

[0019] Obtain data in three modalities: text, video, and audio, input by the user, and extract corresponding text features, video features, and audio features;

[0020] The obtained features are modeled for consistency and difference, respectively. Consistency modeling is used to extract common information that expresses similar meanings in each modality, while difference modeling is used to retain information that is unique to each modality but may have important supplementary value.

[0021] By constructing a shared-private feature space, the common information in the consistent features and the unique information in the differentiated features are explicitly modeled, thus fusing the consistent features and differentiated features.

[0022] Based on the fused feature representation, the classifier determines the user's true intention and outputs the result.

[0023] Furthermore, the obtained features are modeled differentially, specifically:

[0024] The original features of each modality are independently encoded to obtain sequence feature representation;

[0025] A multi-head self-attention mechanism is used to model the long-range dependencies between time steps within the modality and capture the unique temporal structure of the modality itself. Specifically, the query, key, and value vectors are calculated using the obtained sequence feature representation. The feature dimension is then split into multiple attention heads, and the attention weights and weighted feature outputs of each attention head are calculated. The intra-modal representation is obtained by concatenating all attention heads and performing linear transformations and residual connections.

[0026] Using the cross-modal feature modulation module, the semantic representation of the audio modality is used to guide the dynamic adjustment of the video modality features to enhance the ability to perceive the feature differences between the modalities. Specifically, the audio features are used as modulation factors, and the importance weights of the video features are adjusted by element-by-element multiplication.

[0027] Furthermore, the obtained features are modeled for consistency, specifically:

[0028] Through the channel-space joint calculation mechanism, the complementary information between different feature dimensions in each modality is fully explored to enhance local and global perception capabilities;

[0029] Through the dual-temporal feature aggregation mechanism, local short-term and overall long-term temporal dependencies are modeled separately, effectively capturing the temporal dynamic features in speech and video modalities.

[0030] A dynamic gating strategy is adopted to adaptively adjust the fusion weight of each feature according to the contextual semantic relevance of different modalities to achieve more refined semantic alignment.

[0031] Furthermore, the channel-space joint calculation mechanism is as follows:

[0032] The features of video and audio modalities are jointly modeled by channel attention and spatial attention to obtain the feature representation after attention enhancement; as shown in the following formula: Among them, Γ cs (·) represents the joint computation of channel and spatial attention, T i represents the input feature, v represents the video modality, a represents the audio modality, and ⊙ represents the element-by-element multiplication operation;

[0033] During this process, global average pooling and global maximum pooling are used to extract global semantic features, multi-layer perceptron is used for feature transformation, and the attention weights required for joint modeling are calculated through learnable parameter matrix.

[0034] Furthermore, during the channel-space joint calculation, the characteristics of the audio modality also use the SE mechanism to perform adaptive channel weight adjustment, specifically:

[0035] The feature representation after attention enhancement is globally averaged pooled along the spatial dimension to obtain the response value of each channel. The response value is input into a two-layer fully connected network for activation to obtain the channel weight;

[0036] The obtained channel weights are multiplied element-wise with the original features of the audio modality to obtain the modality feature representation optimized by the joint attention mechanism and SE.

[0037] Furthermore, the bi-temporal feature aggregation mechanism is as follows:

[0038] Obtain the features of the video and audio modalities processed by the channel-space joint calculation mechanism, calculate the three-dimensional average pooling representation of each modality, and construct a cross-modal feature representation, as shown in the following formula:

[0039]

[0040] in, and are the features of the video modality and audio modality processed by the channel-space joint calculation mechanism, is the three-dimensional average pooling representation, Represents the result of spatiotemporal three-dimensional maximum pooling of video and audio features;

[0041] Use dilated causal convolution DConv for modeling and generate temporal fusion weights:

[0042]

[0043] w s =Softmax(DConv dil＝2 (S c ⊙S c ));

[0044] in, and ⊙ represent channel concatenation and element-wise multiplication, respectively, and dil represents the dilation rate.

[0045] Furthermore, a dynamic gating strategy is proposed, specifically: according to the contextual semantic relevance of different modalities, the temporal fusion weights obtained in the bi-temporal feature aggregation mechanism are fused with video or audio features; during the fusion period, a learnable gating coefficient λ is used to balance the linear mapping and the nonlinear mapping.

[0046] Furthermore, by constructing a shared-private feature space, the common information in the consistent features and the unique information in the differentiated features are explicitly modeled, thus fusing the consistent features and differentiated features. Specifically:

[0047] Obtain the input features of each modality and obtain the sequence feature representation;

[0048] Construct a parameter-sharing encoder to extract common representations from different modalities;

[0049] Build a modality-specific encoder to extract modality-specific private features;

[0050] Set orthogonality constraints to force the shared and private features of each modality to remain orthogonal in the latent space;

[0051] In the encoder, bilinear mapping is used to map the original features to a unified latent space representation, and explicit partitioning is performed through shared and private subspace basis matrices. The obtained features are concatenated to obtain the final fused representation.

[0052] A second aspect of the present invention provides a system for implementing the above method, comprising:

[0053] The multimodal feature acquisition module is configured to: obtain data in three modalities of text, video, and audio input by the user, and extract corresponding text features, video features, and audio features;

[0054] The difference and consistency processing module is configured to perform consistency modeling and difference modeling on the obtained features, respectively. The consistency modeling is used to extract the common information expressing similar meanings in each modality, and the difference modeling is used to retain the information that is unique to each modality but may have important supplementary value.

[0055] The shared-private space decoupling module is configured to: explicitly model the common information in the consistent features and the unique information in the differentiated features by constructing a shared-private feature space, thereby fusing the consistent features with the differentiated features;

[0056] The classification prediction module is configured to: judge the user's true intention based on the fused feature representation through the classifier and output the result.

[0057] The third aspect of the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the multimodal intent recognition method based on decoupling of consistency and difference are implemented.

[0058] Compared with the existing technology, one or more of the above technical solutions have the following beneficial effects:

[0059] 1. Through collaborative modeling of consistency and difference, consistency modeling is used to extract common information that expresses similar meanings in each modality, while difference modeling is used to retain information that is unique to each modality but may have important supplementary value. By explicitly decoupling the information structure between multiple modalities, both "shared semantics" and "private semantics" semantic subspaces are taken into account, fundamentally improving the accuracy and robustness of intent recognition.

[0060] 2. In multimodal learning tasks, modalities such as text, speech, and video often have their own independent information distribution, expression structure, noise type, and feature space. Therefore, effectively modeling the semantic differences between modalities not only helps to avoid information redundancy and semantic conflict, but also fully explores the complementarity between modalities, thereby improving the model's discriminative and generalization capabilities. This solution proposes a differential feature extraction method that combines a multi-head self-attention mechanism within the modality with a cross-modal feature modulation mechanism. The multi-head self-attention mechanism improves the internal representation capabilities of each modality, enabling the model to capture the fine-grained semantic associations and deep dependency structures of the modality from multiple subspaces. Then, using the cross-modal feature modulation mechanism, semantic comparison and modulation between modalities are achieved by dynamically adjusting the attention and response of one modality to other modalities. This mechanism can explicitly identify and strengthen the differential semantics between modalities, suppress redundant information, and retain complementary features, ensuring that the fusion process is more targeted and selective.

[0061] 3. In multimodal data, each modality, such as speech, video, and text, usually has a different feature space. The feature space is divided into channel dimensions: for example, color channels in images, motion features in videos, etc. And spatial dimensions: for example, the spatial layout in image and video data, especially when processing, the pixel position of the image and the frame position of the video are key features. Therefore, during semantic consistency modeling, the joint calculation mechanism of channel and spatial information enhances the model's ability to perceive local and global information simultaneously, thereby providing a more comprehensive feature representation. It can help the model retain the details of local features and integrate the macro information of global features when fusing cross-modal data, thereby enhancing the model's understanding of complex scenes and multimodal features.

[0062] 4. The dual-temporal feature aggregation mechanism introduced during semantic consistency modeling models short-term and long-term temporal dependencies, effectively capturing the rhythmicity and motion continuity in speech and video modalities. Temporal features are crucial when processing time-series data such as video and speech. They not only exhibit temporal continuity but also may include rhythmic and dynamic changes. By dually modeling short-term and long-term temporal dependencies, the model can more accurately capture rhythmicity and motion continuity in different modalities (such as speech and video), thereby better understanding the evolution of time-series data.

[0063] 5. The dynamic gating strategy during semantic consistency modeling adaptively adjusts the fusion weight of each modal feature according to the contextual semantic relevance between modalities, avoiding the interference of redundant information and retaining key semantic content.

[0064] 6. Compared to traditional fusion methods, progressive multi-dimensional fusion enables the model to finely fuse features across spatial, channel, and temporal dimensions, further optimizing information extraction and integration, thereby improving semantic alignment accuracy. These designs not only enhance the model's contextual modeling capabilities but also improve its robustness, resulting in significant advantages in both accuracy and model stability for multimodal intent recognition tasks. Through the holistic effect of semantic consistency modeling, the feature expression capabilities of multimodal data are enhanced, providing more accurate semantic modeling for intent recognition tasks.

[0065] 7. By constructing a shared-private feature space, the features of each modality are mapped to a shared semantic subspace and a private semantic subspace respectively. The module uses two multi-layer perceptron (MLP) networks to complete the mapping, increasing the network's nonlinear expression capabilities. MLP is a basic feedforward neural network consisting of multiple linear layers (fully connected layers) and nonlinear activation functions. Orthogonality constraints are introduced to ensure that the feature representations of the two spaces are independent of each other. A cross-modal reconstruction mechanism is used to reconstruct the original modality features from the shared semantic subspace to improve the information coverage of the shared representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0067] Figure 1 Schematic diagram of a multimodal intent recognition process based on consistency and difference decoupling provided by one or more embodiments of the present invention;

[0068] Figure 2 It is a schematic diagram of the intra-modal feature learning and cross-modal interaction process provided by one or more embodiments of the present invention.

[0069] Figure 3 It is a schematic diagram of the multimodal intent recognition process provided by one or more embodiments of the present invention.

[0070] Figure 4 is a schematic diagram of a dual-semantic spatiotemporal perception fusion network provided by one or more embodiments of the present invention;

[0071] Figure 5 Schematic diagram of a channel-space joint calculation mechanism provided for one or more embodiments of the present invention;

[0072] Figure 6 Schematic diagram of bi-temporal feature aggregation provided for one or more embodiments of the present invention. DETAILED DESCRIPTION

[0073] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0074] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0075] Example 1:

[0076] During human-computer interaction, user intent is not only reflected in verbal content but is often also expressed through non-verbal signals such as tone, intonation, facial expressions, and gestures. Traditional human-computer interaction technologies rely on converting single verbal information into textual modalities. Text analysis methods struggle to capture implicit cues in these non-verbal modalities, which can lead to biased intent recognition and compromise the user experience and the quality of decision-making.

[0077] This embodiment presents a multimodal intent recognition method based on consistency and difference decoupling. By introducing modal consistency modeling and difference modeling mechanisms, it extracts shared semantics and private semantics from multimodal data, effectively alleviating the redundancy conflict problem in the modal fusion process. At the same time, it designs a semantic decoupling structure to clearly distinguish between shared and private semantic subspaces, improving the model's expressiveness and interpretability. Furthermore, the overall system structure adopts a modular design, which provides good portability and cross-task reuse. By collaboratively modeling consistency and difference, the information structure between multiple modalities is explicitly decoupled, fundamentally improving the accuracy and robustness of intent recognition.

[0078] The multimodal intent recognition method based on consistency and difference decoupling includes the following steps:

[0079] Obtain multimodal input data from users, including voice, video, and text;

[0080] The three types of data are processed separately, extracting audio features such as intonation and speaking speed from speech, visual features such as facial expressions and body movements from video, and semantic information from text;

[0081] Consistency modeling and difference modeling are performed on the obtained features respectively: consistency modeling is used to extract common information that expresses similar meanings in each modality, while difference modeling is used to retain information that is unique to each modality but may have important supplementary value;

[0082] By integrating consistent features with differential features and optimizing multiple loss functions, the information of each modality can be effectively coordinated in a unified framework.

[0083] Based on the fused feature representation, the classifier determines the user's true intention and outputs the result.

[0084] like Figure 1 As shown, the core parts of the above solution are as follows:

[0085] 1) Multimodal feature extraction layer: The text modality uses the BERT pre-trained language model to extract context-aware vector representations. BERT is a pre-trained language model proposed by Google in 2018. The video modality uses sLSTM (structured LSTM), an improved recurrent neural network, to extract dynamic features such as visual behavior and facial expressions. The audio modality uses sLSTM to extract prosodic features such as pitch, speaking rate, and pauses. All modal features are mapped to the same dimensional space through linear transformation.

[0086] 2) Modal Difference Modeling Method: This method is used to emphasize the differences between modalities and includes: multi-head attention to strengthen local representation within the modality and a cross-modal feature modulation mechanism: using audio signals to guide the weighting of video frames to highlight potential irony or polysemy signals; and outputting a private semantic representation vector.

[0087] 3) Modal consistency modeling network: The full name of this network is "Dual Semantic Spatiotemporal Attention Fusion Network", which is used to process the temporal and semantic alignment between video and audio; the dual semantic spatiotemporal perception fusion network consists of three core mechanisms: Channel-space joint calculation mechanism: Fusion of channel attention and spatial attention to extract highly expressive semantic features within the modality from different dimensions, improve the structural understanding ability within the modality and enhance the global semantic interaction features; Dual temporal feature aggregation design: Modeling local temporal dependencies through dilated convolutions, and combining average pooling to extract global temporal trends to achieve dynamic semantic fusion at multiple time scales; Adaptively adjust the fusion weights of each modal feature based on the importance of the feature, so that the model can flexibly allocate the proportion and contribution of semantic expression in the final consistent representation.

[0088] 4) Shared-private space decoupling mechanism: Each modal feature is mapped to a shared semantic subspace and private semantic subspace m represents different modalities; the module uses two multi-layer perceptron (MLP) networks to complete the mapping, increasing the nonlinear expression capability of the network; orthogonality constraints are introduced to ensure that the feature representations of the two spaces are independent of each other; the cross-modal reconstruction mechanism is used to Reconstruct the original modal feature X m , to improve the information coverage of shared representations.

[0089] 5) Classification prediction layer. The concatenation of the six subspace representations (shared and private representations of text, audio, and video) is used as input. A Transformer encoder is used to capture cross-modal dependencies. The Transformer is a neural network architecture based on a self-attention mechanism that excels at parallel processing and modeling long-range dependencies in sequences. The prediction result is finally output through a fully connected layer followed by a Softmax classifier.

[0090] Joint loss function design: classification loss Cross entropy loss function is used to supervise the intention prediction results; consistency loss Use central moment matching (CMD) to align distributions between shared feature spaces; difference loss Improve decoupling by penalizing the orthogonality of shared and private features; reconstruction loss The reconstruction ability is measured by the mean square error (MSE). The final objective function is: Among them, α, β, and γ are adjustable hyperparameters.

[0091] like Figure 3 As shown in the figure, taking the actual algorithm architecture as an example, the implementation process of this solution is explained in detail:

[0092] 1. Data input and preprocessing: During the multimodal input stage, the system receives the user's audio, video, and text data, and samples, frames, and normalizes the three modal data separately.

[0093] The text modality is first segmented and token embedded before being fed into the pre-trained language model BERT to generate context-aware semantic representations.

[0094] The video modality uses the visual perception network sLSTM to extract the temporal feature representation of facial movements;

[0095] The audio modality is processed through frame-level decomposition and Mel spectrogram to extract frame-level acoustic features containing information such as pitch and speaking speed.

[0096] Specifically expressed as:

[0097] Given an input text S, text features T = BERT (S), video and audio features are extracted by sLSTM, and for each non-text modality feature M = sLSTM (u m ), where M∈{V,A}. In order for multimodal features to have the same embedding dimension, the subsequent fusion mechanism can effectively align and process information from different modalities;

[0098] Use linear transformation to reduce the dimension of video and audio, expressed as: M L =W M M+b M , M L Represents the features after dimensionality reduction, where W M and b M are the projection matrix and bias term, T m is the time step of the modality (length of video frame sequence or number of audio feature time steps).

[0099] 2. Semantic Difference Modeling: To effectively mine the differential feature expressions between modalities in multimodal data, this solution proposes a differential feature extraction method that combines the intra-modal multi-head self-attention mechanism with the cross-modal feature modulation mechanism. This method aims to enhance the independent expression capability of each modality and promote differential modeling and selective fusion between different modalities.

[0100] In multimodal learning tasks, modalities such as text, speech, and video often have independent information distributions, expression structures, noise types, and feature spaces. Therefore, effectively modeling the semantic differences between modalities not only helps avoid information redundancy and semantic conflicts, but also fully exploits the complementarity between modalities, thereby improving the model's discriminative and generalization capabilities.

[0101] To address the problems of modal heterogeneity and information asymmetry in multimodal data, this solution proposes a differential feature extraction method that combines an intra-modal multi-head self-attention mechanism with a cross-modal feature modulation mechanism. This method can enhance the independent expression capability of each modality and guide the model to selectively focus on the differences and complementarities between modalities during the fusion process.

[0102] Different modalities inherently carry differences in semantic expression: speech tends to convey dynamic information such as emotion, intonation, and speech rate; video focuses on capturing body movements and visual content; and text excels at expressing structured language and logical semantics. Indiscriminately treating these modal features uniformly can easily lead to over-alignment or semantic conflicts between modal features, weakening the model's ability to express key features and even introducing invalid or interfering information.

[0103] To this end, this solution first enhances the internal representation capabilities of each modality through a multi-head self-attention mechanism within the modality, enabling the model to capture the fine-grained semantic associations and deep dependency structures of the modality from multiple subspaces. Next, a cross-modal feature modulation mechanism is employed to dynamically adjust the attention and response of one modality to the other, enabling semantic comparison and modulation between modalities. This mechanism explicitly identifies and enhances the differential semantics between modalities, suppresses redundant information, and preserves complementary features, ensuring a more targeted and selective fusion process.

[0104] This differential modeling strategy not only effectively avoids the information confusion caused by early fusion, but also enhances the independent expression of each modality and the ability to collaboratively model across modalities. This approach demonstrates greater robustness and generalization capabilities, especially in complex scenarios such as modality loss, data noise, or semantic conflicts, providing a more robust and efficient feature modeling path for multimodal systems.

[0105] Specifically, such as Figure 2 As shown in the multimodal semantic understanding system of this scheme, for the three modal inputs of text (T), video (V) and audio (A), the original features of each modality are first independently encoded to obtain the sequence feature representation Where m∈{T,V,A}.

[0106] In order to enhance the feature expression capability within a single modality, a multi-head self-attention mechanism is used to model the long-range dependencies between time steps within the modality, thereby capturing the unique temporal structure of the modality itself. This process includes the following steps:

[0107] Compute query Q on input features m , key K m , value vector V m : Split the feature dimension into N attention heads, and is the weight matrix, and the dimension of each head is d m is the input feature dimension, N represents the number of attention heads, and the attention weight and weighted feature output are calculated for each attention head: is the attention score of the i-th attention head, is the attention weight of the i-th attention head, is the output of the ith attention head. All attention heads are concatenated and linearly transformed and residually connected to obtain the intra-modal representation:

[0108]

[0109] The above-mentioned multi-head self-attention mechanism operation can enhance the structural modeling capabilities within each modal feature, thereby highlighting the differences between modalities at the expression level and laying the foundation for subsequent cross-modal interaction.

[0110] To further enhance the contrast between modalities, this solution designs a cross-modal feature modulation module. This module uses the semantic representation of the audio modality to guide the dynamic adjustment of video modal features, thereby enhancing the ability to perceive feature differences between modalities. The cross-modal feature modulation module uses audio features as modulation factors and uses element-by-element multiplication to adjust the importance weights of video features.

[0111] The process is specifically expressed as:

[0112] Among them, ⊙ represents the element-by-element multiplication operation, The new video representation is modulated by the audio features. This approach can achieve local enhancement of modal differences and improve the selectivity and adaptability of modal interaction.

[0113] 3. Semantic consistency modeling: The dual-semantic spatiotemporal perception fusion network proposed in this scheme adopts a progressive fusion framework to extract and fuse multimodal features from multiple levels of channel dimension, spatial dimension and temporal dimension.

[0114] The architecture of the dual semantic spatiotemporal perception fusion network is as follows Figure 4 As shown, specifically:

[0115] Through channel-space joint calculation, the complementary information between different feature dimensions in each modality is fully exploited to enhance local and global perception capabilities;

[0116] A dual-temporal feature aggregation mechanism is introduced to model local short-term and overall long-term temporal dependencies respectively, effectively capturing temporal dynamic features such as rhythm and action continuity in speech and video modalities.

[0117] A dynamic gating strategy is adopted to adaptively adjust the fusion weight of each feature according to the contextual semantic relevance of different modalities, thereby achieving more refined semantic alignment.

[0118] The dual-semantic spatiotemporal perception fusion network proposed in this solution significantly improves the feature expression capabilities of multimodal data and provides more accurate semantic modeling by combining multi-level feature fusion, temporal feature modeling, and dynamic adjustment strategies. It also enhances local and global perception capabilities through joint channel-space computation, mining complementary information between modalities from multiple dimensions and improving the ability to understand complex scenes.

[0119] In multimodal data, each modality, such as speech, video, and text, typically has a distinct feature space. This feature space is divided into channel dimensions, such as color channels in images and motion features in videos, and spatial dimensions, such as the spatial layout of image and video data. In particular, pixel positions in images and frame positions in videos are key features during processing.

[0120] The joint computation of channel and spatial information enhances the model's ability to perceive both local and global information, providing a more comprehensive feature representation. This helps the model preserve local feature details while integrating global macro information when fusing cross-modal data, thereby enhancing the model's understanding of complex scenes and multimodal features.

[0121] The introduced dual-temporal feature aggregation mechanism models short-term and long-term temporal dependencies respectively, effectively capturing the rhythmicity and motion continuity in speech and video modalities. Temporal features are very important factors when processing time-series data such as video and speech, because they not only have temporal continuity but may also contain rhythmic and dynamic changes.

[0122] Traditional time series modeling methods are typically unable to simultaneously account for both short-term and long-term temporal dependencies. Local short-term temporal dependencies help the model understand rapidly changing parts of the data, such as pauses in speech or rapid movements in videos. Global long-term temporal dependencies help capture more persistent trends, such as changes in speech intonation or sustained motion patterns in videos. By dually modeling short-term and long-term temporal dependencies, the model can more accurately capture the rhythmicity and motion continuity of different modalities (such as speech and video), thereby better understanding the evolution of time series data.

[0123] The dynamic gating strategy adaptively adjusts the fusion weights of each modal feature based on the contextual semantic relevance between modalities, avoiding interference from redundant information and retaining key semantic content. Compared with traditional fusion methods, progressive multi-dimensional fusion enables the model to finely fuse features in spatial, channel, and temporal dimensions, further optimizing information extraction and integration, thereby improving semantic alignment accuracy. These designs not only enhance the contextual modeling capabilities of the model, but also improve the robustness of the model, resulting in significant advantages in both accuracy and model stability for multimodal intent recognition tasks. Through the above overall effects, this method improves the feature expression capabilities of multimodal data and provides more accurate semantic modeling for intent recognition tasks.

[0124] like Figure 5 As shown in Figure 2, the channel-space joint calculation mechanism is as follows:

[0125] Input modal feature T i(i∈{v,a}), v represents the video mode, a represents the audio mode) is input into the attention mechanism to obtain the feature representation after attention enhancement This process is completed through the joint modeling of channel attention and spatial attention. The calculation formula is as follows:

[0126]

[0127] Among them, Γ cs (·) denotes the joint computation of channel and spatial attention. ⊙ denotes the element-wise multiplication operation.

[0128] The attention mechanism uses global average pooling (GAP) and global maximum pooling (GMP) to extract global semantic features, and then uses a multi-layer perceptron (MLP) to transform the features. The attention weight is calculated through the learnable parameter matrices W1 and W2. The calculation expression is as follows:

[0129]

[0130] where f 7×7 represents a 7×7 convolution operation, represents channel dimension splicing, and σ represents the Sigmoid activation function.

[0131] Based on the characteristics of the audio modality, the Squeeze-and-Excitation (SE) mechanism is further introduced to perform adaptive adjustment of channel weights.

[0132] The SE process includes the following steps:

[0133] Feature compression: enhancing features with attention Perform GAP operation along the spatial dimension to obtain the response value of each channel: Where H represents the height of the feature map, W represents the width of the feature map, C represents the number of channels of the feature map, and F iatt (c,h,w) represents the value of the i-th attention-enhanced feature map at channel c, height h, and width w.

[0134] Feature incentive: Input to a two-layer fully connected network with a bottleneck structure: s = σ(W2·δ(W1·z)). Where δ(·) is the ReLU activation function, the weight matrix Weight Matrix r is the scaling factor.

[0135] Feature rescaling: Apply channel weights s to the original features: After the above operations are completed, the modal feature representation optimized by the joint attention mechanism and SE can be obtained.

[0136] Bi-temporal feature aggregation, such as Figure 6 As shown: In order to further model the temporal information, this method adopts the joint spatiotemporal representation method to splice and fuse the features of different modalities. Specifically:

[0137] Compute the 3D average pooled representation for each modality: The cross-modal feature representation is constructed in the following way:

[0138]

[0139] in and They are the video and audio features processed by the attention module, Represents the video and audio features ( and ) is the result of spatiotemporal 3D maximum pooling.

[0140] In order to capture long-term temporal dependencies, dilated causal convolution (DConv) is introduced for modeling and temporal fusion weights are generated:

[0141]

[0142] w s =Softmax(DConv dil＝2 (S c ⊙S c ));

[0143] in, and ⊙ represent channel concatenation and element-wise multiplication, respectively. dil=2 indicates a dilation rate of 2.

[0144] Softmax is used to ensure weight normalization.

[0145] Dynamic gating fusion module: This solution further designs a dynamic gating mechanism to achieve weighted fusion of spatiotemporal features.

[0146] The fusion expression is as follows:

[0147]

[0148] in, is a video or audio feature, Expand(·) broadcasts the spatial weights to match the feature dimension, represents the channel weight of the kth modality, represents the spatial weight of the kth mode, is a tensor contraction operation. To avoid information loss, a learnable gating coefficient λ is used to balance the linear mapping and the nonlinear mapping:

[0149] λ=σ(wλ F fusion );

[0150] F out =λ·tanh(W g F fusion )+(1-λ)F fusion ;

[0151] in is the projection matrix, σ is the Sigmoid activation function, w λ is a learnable weight vector, W g is a projection matrix.

[0152] 4. Feature Decoupling and Spatial Modeling: To enhance the multimodal semantic understanding model's ability to model common information and modal difference information, this solution proposes a multimodal feature decoupling mechanism. This mechanism explicitly models the common and unique information in multimodal data through shared-private feature decomposition, thereby effectively improving the alignment effect and complementary expression between modalities.

[0153] This solution designs the following decoupling structure:

[0154] Arbitrary modal input features Among them, text (T), video (V) and audio (A) are three modal inputs, and the original feature sequence features of each modality are expressed as Where m∈{T,V,A}.

[0155] Shared Feature Extractor: Constructs a parameter-sharing encoder to extract common representations from different modalities:

[0156] Private Feature Extractor: Build a modality-specific encoder to extract private features unique to the modality:

[0157] Orthogonality constraint design: To ensure that decoupled features do not interfere with each other, this solution introduces orthogonality constraints to force the shared and private features of each modality to remain orthogonal in the latent space, that is, to satisfy: This constraint helps to enhance the modal decoupling effect and ensure that the expressions between different feature spaces are not redundant.

[0158] Projection transformation and space division: To further explicitly divide the feature components, based on the above encoding representation, this solution designs a modal projection module to map the original features to a unified latent space representation through bilinear mapping: in, and is the modal projection parameter, which is the intermediate feature of modal unification.

[0159] The representation is then explicitly partitioned into shared and private subspace basis matrices:

[0160]

[0161] in, is the modal private subspace basis matrix, is the cross-modal private subspace basis matrix, is the cross-modal shared subspace basis matrix, and σ(·) is the nonlinear activation function.

[0162] In order to ensure the orthogonality of the private subspace and the shared subspace, further constraints are imposed This constraint ensures that the private subspace and the shared subspace do not produce redundant overlap after transformation, thereby improving the expressive power of feature decoupling.

[0163] 5-classification prediction layer: After feature decoupling (shared-private decomposition), 6 features are obtained:

[0164]

[0165] After Transformer encoding, 6 feature vectors are obtained:

[0166]

[0167] Finally, they are concatenated into the final fusion representation:

[0168]

[0169] The intent category is calculated using the fully connected layer and the SoftMax function. The final formula is expressed as: Where W cls is the weight matrix of the fully connected layer, b cls is the bias vector of the fully connected layer, and e is the base of the natural logarithm.

[0170] 6 Joint loss function design: Finally, a multi-loss joint optimization strategy is adopted, combining task loss Similarity loss Orthogonalization loss Reconstruction losses Optimize to ensure that the model can effectively learn the correlation of multimodal features while maintaining the independence between different modalities and enhancing the generalization ability of the model. The final optimization objective function is as follows:

[0171]

[0172] in, Cross entropy loss is used to optimize classification tasks to minimize the error between the model prediction results and the true labels:

[0173]

[0174] Among them, y i is the true label, is the class probability predicted by the model, and N is the number of samples.

[0175] The goal is to reduce the difference in the distribution of different modal features to enhance the modal alignment capability. We use central moment matching (CMD) to measure the distance between the distributions of different modal features. The CMD loss is used to align the distributions of different modalities and can be expressed as:

[0176]

[0177] in and Represents the mean of the kth order moment of different modes. In order to maintain the independence of different modal features, we introduce Constrain the feature representations of different modalities to be as orthogonal as possible to reduce redundant information interference:

[0178]

[0179] Among them, f i ,f j The feature representations of different modalities are respectively expressed, and it is expected that they are as orthogonal as possible in the high-dimensional space.

[0180] In addition, the introduction To optimize the reconstruction ability of features, to ensure that the information learned by the model can still effectively restore the original features after dimensionality reduction, the mean square error (MSE) is used for calculation:

[0181]

[0182] Among them, x i is the original input feature, Rebuild features for the model.

[0183] By jointly optimizing the loss function, the model balances classification performance, modality alignment, and information redundancy while enhancing generalization. Hyperparameters α, β, and γ control the impact of each loss term and can be fine-tuned to achieve optimal multimodal fusion.

[0184] Example 2:

[0185] A system for implementing the above method includes:

[0186] The multimodal feature acquisition module is configured to: obtain data in three modalities of text, video, and audio input by the user, and extract corresponding text features, video features, and audio features;

[0187] The difference and consistency processing module is configured to perform consistency modeling and difference modeling on the obtained features, respectively. The consistency modeling is used to extract the common information expressing similar meanings in each modality, and the difference modeling is used to retain the information that is unique to each modality but may have important supplementary value.

[0188] The shared-private space decoupling module is configured to: explicitly model the common information in the consistent features and the unique information in the differentiated features by constructing a shared-private feature space, thereby fusing the consistent features with the differentiated features;

[0189] The classification prediction module is configured to: judge the user's true intention based on the fused feature representation through the classifier and output the result.

[0190] Example 3:

[0191] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the multimodal intent recognition method based on decoupling of consistency and difference as described in the first embodiment above are implemented.

[0192] Example 4:

[0193] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the multimodal intent recognition method based on decoupling of consistency and difference as described in the first embodiment above are implemented.

[0194] The steps or modules involved in Examples 2 to 4 above correspond to those in Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media that includes one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and cause the processor to perform any method of the present invention.

[0195] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A multimodal intent recognition method based on decoupling of consistency and difference, characterized by: The following steps are involved: Obtain data in three modalities: text, video, and audio, input by the user, and extract corresponding text features, video features, and audio features; The obtained features are modeled for consistency and difference, respectively. Consistency modeling is used to extract common information that expresses similar meanings in each modality, while difference modeling is used to retain information that is unique to each modality but may have important supplementary value. By constructing a shared-private feature space, the common information in the consistent features and the unique information in the differentiated features are explicitly modeled, thus fusing the consistent features and differentiated features. Based on the fused feature representation, the classifier determines the user's true intention and outputs the result.

2. The multimodal intent recognition method based on consistency and difference decoupling according to claim 1, characterized in that: Perform differential modeling on the obtained features, specifically: The original features of each modality are independently encoded to obtain sequence feature representation; A multi-head self-attention mechanism is used to model the long-range dependencies between time steps within the modality and capture the unique temporal structure of the modality itself. Specifically, the query, key, and value vectors are calculated using the obtained sequence feature representation. The feature dimension is then split into multiple attention heads, and the attention weights and weighted feature outputs of each attention head are calculated. The intra-modal representation is obtained by concatenating all attention heads and performing linear transformations and residual connections. Using the cross-modal feature modulation module, the semantic representation of the audio modality is used to guide the dynamic adjustment of the video modality features to enhance the ability to perceive the feature differences between the modalities. Specifically, the audio features are used as modulation factors, and the importance weights of the video features are adjusted by element-by-element multiplication.

3. The multimodal intent recognition method based on consistency and difference decoupling according to claim 1, characterized in that: The obtained features are modeled for consistency, specifically: Through the channel-space joint calculation mechanism, the complementary information between different feature dimensions in each modality is fully explored to enhance local and global perception capabilities; Through the dual-temporal feature aggregation mechanism, local short-term and overall long-term temporal dependencies are modeled separately, effectively capturing the temporal dynamic features in speech and video modalities. A dynamic gating strategy is adopted to adaptively adjust the fusion weight of each feature according to the contextual semantic relevance of different modalities to achieve more refined semantic alignment.

4. The multimodal intent recognition method based on consistency and difference decoupling according to claim 3 is characterized in that: The channel-space joint calculation mechanism is as follows: The features of video and audio modalities are jointly modeled by channel attention and spatial attention to obtain attention-enhanced feature representations; As shown in the following formula: i∈{v,a}; where Γ cs (·) represents the joint computation of channel and spatial attention, ⊙ represents the element-wise multiplication operation, T i is the input feature, v represents the video modality, and a represents the audio modality; During this process, global average pooling and global maximum pooling are used to extract global semantic features, multi-layer perceptron is used for feature transformation, and the attention weights required for joint modeling are calculated through learnable parameter matrix.

5. The multimodal intent recognition method based on consistency and difference decoupling according to claim 4 is characterized in that: During the joint channel-space computation, the audio modality features also use the SE mechanism to adaptively adjust the channel weights, specifically: The feature representation after attention enhancement is globally averaged pooled along the spatial dimension to obtain the response value of each channel. The response value is input into a two-layer fully connected network for activation to obtain the channel weight; The obtained channel weights are multiplied element-wise with the original features of the audio modality to obtain the modality feature representation optimized by the joint attention mechanism and SE.

6. The multimodal intent recognition method based on consistency and difference decoupling according to claim 3, characterized in that: The bi-temporal feature aggregation mechanism is as follows: Obtain the features of the video and audio modalities processed by the channel-space joint calculation mechanism, calculate the three-dimensional average pooling representation of each modality, and construct a cross-modal feature representation, as shown in the following formula: in, and are the features of the video modality and audio modality processed by the channel-space joint calculation mechanism, is the three-dimensional average pooling representation, Represents the result of spatiotemporal three-dimensional maximum pooling of video and audio features; Use dilated causal convolution DConv for modeling and generate temporal fusion weights: w s =Softmax(DConv dil＝2 (S c ⊙S c )); in, and ⊙ represent channel concatenation and element-wise multiplication, respectively, and dil represents the dilation rate.

7. The multimodal intent recognition method based on consistency and difference decoupling according to claim 3, characterized in that: The dynamic gating strategy is as follows: based on the contextual semantic relevance of different modalities, the temporal fusion weights obtained from the bi-temporal feature aggregation mechanism are fused with video or audio features; during the fusion process, a learnable gating coefficient λ is used to balance the linear mapping and the nonlinear mapping.

8. The multimodal intent recognition method based on consistency and difference decoupling according to claim 1, characterized in that: By constructing a shared-private feature space, the common information in the consistent features and the unique information in the differentiated features are explicitly modeled, and the consistent features and differentiated features are fused. Specifically: Obtain the input features of each modality and obtain the sequence feature representation; Construct a parameter-sharing encoder to extract common representations from different modalities; Build a modality-specific encoder to extract modality-specific private features; Set orthogonality constraints to force the shared and private features of each modality to remain orthogonal in the latent space; In the encoder, bilinear mapping is used to map the original features to a unified latent space representation, and explicit partitioning is performed through shared and private subspace basis matrices. The obtained features are concatenated to obtain the final fused representation.

9. A multimodal intention recognition method system based on decoupling of consistency and difference, characterized by: include: The multimodal feature acquisition module is configured to: obtain data in three modalities of text, video, and audio input by the user, and extract corresponding text features, video features, and audio features; The difference and consistency processing module is configured to perform consistency modeling and difference modeling on the obtained features, respectively. The consistency modeling is used to extract the common information expressing similar meanings in each modality, and the difference modeling is used to retain the information that is unique to each modality but may have important supplementary value. The shared-private space decoupling module is configured to: explicitly model the common information in the consistent features and the unique information in the differentiated features by constructing a shared-private feature space, thereby fusing the consistent features with the differentiated features; The classification prediction module is configured to: judge the user's true intention based on the fused feature representation through the classifier and output the result.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the multimodal intent recognition method based on decoupling of consistency and difference as claimed in any one of claims 1 to 8 are implemented.

Images