Multi-modal intent recognition method and system fusing intent center consistency and semantic difference learning

Abstract

The application provides a multi-modal intention recognition method and system fusing intention center consistency and semantic difference learning, and the method comprises the following steps: obtaining multi-modal data of a sample to be recognized; performing a preset operation on the multi-modal data to obtain multi-modal fusion features, and extracting intention level representations from the multi-modal fusion features; constructing multi-modal fusion feature variants, and performing consistency constraint learning on intention level representations corresponding to the multi-modal fusion feature variants with the intention level representations as references; according to contribution degrees, suppressing features of specified modes to obtain intention level representations after mode suppression, and performing difference constraint learning on differences between the intention level representations before and after the suppression; training a multi-modal intention recognition network based on a total loss formed by intention classification cross-entropy loss, consistency constraint loss and difference constraint loss; and inputting the multi-modal data of the sample to be recognized into the trained multi-modal intention recognition network to output an intention category.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal intent recognition method and system that integrates intent-centered consistency and semantic difference learning. Background Technology

[0002] Multimodal intent recognition technology determines intent by collaboratively analyzing multi-source information such as text, vision, and audio, and has significant application value in scenarios such as intelligent human-computer interaction, customer service systems, and behavior understanding. This technology can integrate complementary information from different modalities, significantly improving the robustness of intent understanding in complex environments. Current mainstream methods rely on deep learning frameworks to construct a unified representation space through feature extraction, temporal alignment, and fusion modeling. However, practical applications face a core challenge: different modalities have fundamental differences in semantic expression, noise distribution, and information density, resulting in a dynamic imbalance in modal contributions. Specifically, the text modality usually carries the core intent semantics, while the visual and audio modalities mainly provide auxiliary contextual information, and this difference in contribution fluctuates significantly across different samples.

[0003] Existing technologies mainly fall into three categories: The first category employs cross-modal interaction and adaptive fusion mechanisms, dynamically adjusting modal weights using attention or gating strategies. While this can alleviate modal imbalance in some samples, its optimization objective implicitly assumes that the fused representation should maintain overall semantic consistency, failing to explicitly model the structured semantic changes caused by differences in modal contributions. The second category introduces contrastive learning for cross-modal representation alignment, treating different modal views as semantically equivalent positive samples. This overemphasizes representation consistency constraints, failing to distinguish between reasonable perturbations in intent preservation and true semantic shifts caused by modal differences, resulting in excessive contraction of intent representation and weakening the model's ability to utilize diverse semantic cues. The third category prioritizes high-information modalities through modality selection or dynamic attention mechanisms, but this adjustment is only implemented during the inference phase, lacking a systematic modeling of "how intent semantics should change during modality suppression," and failing to guide the model to understand the relative roles of each modality in the semantic space from the training level. These shortcomings make it difficult for existing methods to effectively characterize semantic differences caused by changes in modality dominance relationships. In scenarios with multimodal semantic heterogeneity and dynamically imbalanced contributions, the discriminativeness and stability of intent representation are constrained. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a multimodal intent recognition method and system that integrates intent center consistency and semantic difference learning, so as to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of the present invention provides a multimodal intent recognition method that integrates intent center consistency and semantic difference learning, comprising: Acquire multimodal data of the sample to be identified; The multimodal data is subjected to preset operations to obtain multimodal fusion features, and intent-level representations are extracted from them; Construct a multimodal fusion feature variant, and use the intent-level representation as a reference to perform consistency constraint learning on the intent-level representation corresponding to the multimodal fusion feature variant; Based on the information from the multimodal data fusion process, the contribution of different modalities to the intent-level representation is calculated; Based on the contribution, the features of the specified modality are suppressed to obtain the intention-level representation after modality suppression, and the difference between the intention-level representations before and after suppression is subjected to difference constraint learning. Calculate the consistency constraint loss and the difference constraint loss, and train the multimodal intent recognition network based on the intention classification cross-entropy loss, the consistency constraint loss and the difference constraint loss. The multimodal data of the sample to be identified is input into the trained multimodal intent recognition network, which outputs the intent category.

[0006] In some embodiments of the present invention, the multimodal data undergoes a preset operation to obtain multimodal fusion features, and intent-level representations are extracted from them, including: The text modal data, visual modal data, and audio modal data in the multimodal data are temporally aligned, and the aligned multimodal features are respectively encoded to obtain aligned text feature sequences, visual feature sequences, and audio feature sequences. The aligned text feature sequence, visual feature sequence, and audio feature sequence are input into a dynamic attention fusion network to obtain multimodal fusion features, and intent-level representations are extracted from the intent alignment regions corresponding to intent labels in the multimodal fusion features.

[0007] In some embodiments of the present invention, constructing a multimodal fusion feature variant includes: Construct a local context window covering the intended alignment region, and set the features within the local context window to zero or reduce their weight to obtain a multimodal fusion feature variant of the semantic local mask; and / or, The visual feature sequence and / or audio feature sequence are randomly masked at time step ratios r, and then re-fused with the text feature sequence to obtain a multimodal fusion feature variant with random feature masks; and / or, The visual feature sequence and the audio feature sequence are subjected to suppression processing, which involves setting all features to zero or masking them. Then, they are re-fused with the other sequence and the text feature sequence to obtain a multimodal fusion feature variant with modal suppression.

[0008] In some embodiments of the present invention, the calculation of the consistency constraint loss is performed using the following formula: in, The loss function; This represents the intent-level representation extracted from multimodal fusion features. This represents the intent-level representation of the multimodal fusion feature variants. This represents the similarity measurement function; B represents the preset temperature parameter; B represents the sample batch size.

[0009] In some embodiments of the present invention, it further includes: Extract the corresponding intent-level representation from each modal branch of the multimodal data; Taking the intent-level representation corresponding to the text modality data carrying intent labels as a reference, consistency constraint learning is performed on the intent-level representations corresponding to other modality branches, or consistency constraint learning is performed simultaneously on the intent-level representations corresponding to other modality branches and the intent-level representations extracted from multimodal fusion features. The consistency constraint loss is calculated using the following formula: in, An intent-level representation of text modal data carrying intent labels; It can be either the intent-level representation corresponding to other modal branches, or the set of intent-level representations corresponding to other modal branches and intent-level representations extracted from multimodal fusion features; These are the weighting coefficients.

[0010] In some embodiments of the present invention, the step of suppressing the features of a specified modality based on the contribution degree to obtain an intent-level representation after modality suppression, and performing difference constraint learning on the difference between the intent-level representations before and after suppression, includes: Suppression processing, namely zeroing or masking, is performed on all features of one of the visual feature sequences and the audio feature sequence, respectively. Then, the two sequences are re-fused with the other sequence and the text feature sequence, and the suppressed intent-level representation is extracted from the fused features. The semantic offset between the intent-level representation before and after suppression is calculated to obtain the strong suppression semantic offset and the weak suppression semantic offset, and an asymmetric interval constraint is applied to the strong suppression semantic offset and the weak suppression semantic offset.

[0011] In some embodiments of the present invention, the calculation of the difference constraint loss is performed using the following formula: in, To strongly suppress semantic shift; This represents the intention level before suppression; This is the suppressed intention-level representation, in which high-contribution modalities are suppressed; This represents a weakly suppressed semantic shift. The suppressed intention level representation, in which low-contribution modalities are suppressed.

[0012] Another aspect of the present invention provides a multimodal intent recognition system that integrates intent center consistency and semantic difference learning, comprising: The acquisition module is used to acquire multimodal data of the sample to be identified; The extraction module is used to perform preset operations on the multimodal data to obtain multimodal fusion features and extract intent-level representations from them; The consistency constraint learning module is used to construct a multimodal fusion feature variant and, with the intent-level representation as a reference, perform consistency constraint learning on the intent-level representation corresponding to the multimodal fusion feature variant. The calculation module is used to calculate the contribution of different modalities to the intent-level representation based on the information in the multimodal data fusion process. The difference constraint learning module is used to suppress the features of a specified modality based on the contribution degree, obtain the intention-level representation after modality suppression, and perform difference constraint learning on the difference between the intention-level representations before and after suppression. The training module is used to calculate the consistency constraint loss and the difference constraint loss, and to train the multimodal intent recognition network based on the total loss formed by the intent classification cross-entropy loss, the consistency constraint loss and the difference constraint loss. The intent recognition module is used to input the multimodal data of the sample to be recognized into the trained multimodal intent recognition network and output the intent category.

[0013] This invention presents a multimodal intent recognition method and system that integrates intent center consistency and semantic difference learning. It effectively addresses the problem of traditional multimodal intent recognition methods lacking explicit modeling of semantic changes when dealing with imbalances and dynamic variations in modal contributions. This method ensures that the intent-level representation remains semantically stable even in the face of local perturbations through consistency constraint learning. Simultaneously, through difference constraint learning, the model can systematically understand the structured contributions of different modalities to intent semantics, avoiding excessive contraction of the intent representation, thereby improving the robustness and accuracy of intent recognition in complex multimodal scenarios.

[0014] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0015] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. The components in the drawings are not drawn to scale but are merely illustrative of the principles of the invention. For ease of illustration and description of certain parts of the invention, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to the invention. In the drawings: Figure 1 This is a flowchart of a multimodal intent recognition method that integrates intent center consistency and semantic difference learning in one embodiment of the present invention.

[0017] Figure 2 This is a flowchart of a multimodal intent recognition method that integrates intent center consistency and semantic difference learning in another embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0019] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0020] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0021] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0022] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0023] Traditional multimodal intent recognition methods, when dealing with the uneven contribution of different modalities to the semantic meaning of intent and the dynamic changes in modal contributions, mainly focus on weight allocation or consistency constraints at the representation level during the fusion process. They lack explicit modeling of the semantic changes caused by differences in modal contributions, making it difficult to characterize the structured semantic differences dominated by different modalities. Furthermore, existing methods do not distinguish between reasonable perturbations in intent preservation and the true semantic shifts caused by differences in modal contributions, which can easily lead to excessive contraction of intent representation, weakening the ability to utilize diverse semantic cues. Moreover, they do not systematically model the changes in intent semantics that should occur when a modality is suppressed or removed, making it difficult to guide the model to understand the relative semantic roles of different modalities from the training level.

[0024] To address this issue, this application proposes a multimodal intent recognition method that integrates intent center consistency and semantic difference learning. This method acquires multimodal data of the sample to be recognized and performs pre-defined operations to obtain multimodal fusion features and intent-level representations. Further, it constructs variants of the multimodal fusion features and performs consistency constraint learning with the intent-level representation as a reference. Simultaneously, based on information from the multimodal data fusion process, it calculates the contribution of different modalities to the intent-level representation and suppresses specified modal features according to this contribution, obtaining the modality-suppressed intent-level representation. Then, it performs difference constraint learning on the difference between the intent-level representations before and after suppression. Finally, by calculating the total loss formed by the consistency constraint loss, difference constraint loss, and intent classification cross-entropy loss, the multimodal intent recognition network is trained, thereby achieving accurate output of intent categories.

[0025] The above approach effectively addresses the problem of traditional multimodal intent recognition methods lacking explicit modeling of semantic changes when dealing with imbalances and dynamic variations in modal contributions. This method ensures that the intent-level representation remains semantically stable in the face of local perturbations through consistency constraint learning. Simultaneously, through difference constraint learning, the model can systematically understand the structured contributions of different modalities to intent semantics, avoiding excessive contraction of the intent representation, thereby improving the robustness and accuracy of intent recognition in complex multimodal scenarios.

[0026] For ease of understanding, the following explains some key terms in this embodiment: Multimodal data refers to a collection of data from different information sources, such as text, visual data, and audio data. In multimodal intent recognition tasks, this data is used for comprehensive analysis to understand the user's intent.

[0027] Multimodal fusion features refer to a comprehensive representation formed in a unified feature space by encoding, aligning, and fusing the original features of different modalities. This feature aims to capture complementary information and correlations between different modalities, providing comprehensive semantic information for subsequent intent determination.

[0028] Intent-level representation refers to high-level abstract features extracted from multimodal fusion features that directly reflect the semantic intent of a sample. This representation is typically designed to be discriminative, enabling intent classifiers to accurately determine intent.

[0029] Multimodal fusion feature variants refer to new feature representations generated by locally perturbing or adjusting the modal information of multimodal fusion features. These variants are used as contrastive samples in consistency constraint learning to enhance the model's robustness to the core semantics of intent.

[0030] Consistency constraint learning refers to constructing a contrastive objective that ensures the semantic consistency between the original intent-level representation and the intent-level representation extracted by its corresponding multimodal fusion feature variant in the feature space. This learning process aims to ensure that the model can still stably identify the same intent when faced with modal local perturbations.

[0031] Contribution refers to the relative importance or influence of different modalities on the formation of the final intent-level representation during multimodal data fusion. This contribution reflects the weight of each modality in expressing the semantic intent and can be used to guide modality suppression operations.

[0032] Difference-constrained learning refers to guiding the model to understand the structured contribution of different modalities to the semantics of intent by explicitly modeling the semantic shift between the intent-level representation before and after modality suppression. This learning process aims to enable the model to distinguish between reasonable perturbations in intent preservation and the true semantic shift caused by differences in modal contributions.

[0033] Intent classification cross-entropy loss is a loss function used in multimodal intent recognition tasks to measure the difference between the distribution of intent categories predicted by the model and the distribution of the true intent categories. This loss function is commonly used in supervised learning to drive the model to learn the correct intent classification boundary.

[0034] A multimodal intent recognition network is a deep learning model designed to receive multimodal data input, process it through feature extraction, fusion, intent-level representation generation, and classification, and finally output the intent category of the sample to be recognized. This network achieves automatic recognition of multimodal intents in an end-to-end manner.

[0035] The embodiments of this disclosure are described below with reference to the accompanying drawings.

[0036] Figure 1This is a flowchart of a multimodal intent recognition method that integrates intent center consistency and semantic difference learning in one embodiment of the present invention. Figure 1 As shown, the multimodal intent recognition method that integrates intent center consistency and semantic difference learning includes the following steps S101-S107: In step S101, multimodal data of the sample to be identified is obtained.

[0037] Multimodal data can include various forms such as text, visual data, and audio data. For example, in an intelligent customer service scenario, the multimodal data of the sample to be identified could be a text query entered by a user, a video clip of a user's facial expressions, or an audio recording of a user's voice commands. This data can be collected independently by different sensors or input interfaces and stored in its raw data format.

[0038] In step S102, a preset operation is performed on the multimodal data to obtain multimodal fusion features, and intent-level representations are extracted from them.

[0039] The preset operation may include independent feature extraction for each modality of data, such as temporal alignment and encoding operations, and then inputting them into the dynamic attention fusion module to form preliminary multimodal fusion features. Next, an intent-level representation representing the overall intent of the sample can be extracted from this preliminary multimodal fusion feature through a fully connected layer or average pooling operation.

[0040] In step S103, a multimodal fusion feature variant is constructed, and consistency constraint learning is performed on the intent-level representation corresponding to the multimodal fusion feature variant, with the intent-level representation as a reference.

[0041] Variants can be constructed by adding random noise to the original multimodal fusion features or by randomly zeroing out certain dimensions of the feature vector to simulate slight perturbations in the data. For example, a subset of feature dimensions from the multimodal fusion features can be randomly selected and their values ​​replaced with random numbers or zero. These variants are then fed into the same network structure as the original feature extraction to obtain the intent-level representations corresponding to the variants. Consistency constraint learning can be achieved by calculating the Euclidean distance or cosine similarity between the original intent-level representation and the variant intent-level representation, and minimizing or maximizing this distance to ensure that the model's intent recognition results remain stable even with slight changes in features.

[0042] In step S104, based on the information from the multimodal data fusion process, the contribution of different modalities to the intent-level representation is calculated.

[0043] This contribution can be determined by analyzing the impact of each modal feature on the final intent-level representation during the fusion process. For example, its contribution can be approximately evaluated by the weight of each modal feature in the fusion layer, or it can be measured by performing sensitivity analysis on the model to observe the degree of change in the intent-level representation when a certain modal feature is removed or perturbed.

[0044] Specifically, the dynamic attention fusion module assigns corresponding attention weights to features of different modalities during the fusion process. These attention weights are time-step or word-level weights used to characterize the semantic contribution of each modality at different time positions. Average pooling is performed on the attention weights of the same modality across all time steps to obtain a scalar or vector-based measure of modality importance, which serves as the contribution to characterize the overall importance of different modalities in the sample intent recognition task.

[0045] In step S105, based on the contribution degree, the features of the specified modality are suppressed to obtain the intention-level representation after modality suppression, and difference constraint learning is performed on the difference between the intention-level representation before and after suppression.

[0046] Feature suppression for a specific modality can be achieved by setting all feature vectors of that modality to zero or multiplying them by a coefficient close to zero. For example, if the visual modality is determined to contribute the most, features of the audio modality can be suppressed. After suppression, the features of the remaining modalities are re-fused, and the modality-suppressed intent-level representation is extracted from them. Differential constraint learning can guide the model to learn the structured influence of different modalities on intent semantics by calculating the distance between the intent-level representation before and after suppression and setting a target distance range.

[0047] In step S106, the consistency constraint loss and the difference constraint loss are calculated, and the multimodal intent recognition network is trained based on the total loss formed by the intent classification cross-entropy loss, the consistency constraint loss and the difference constraint loss.

[0048] Intent classification cross-entropy loss is used to supervise the model to correctly classify intents. Consistency constraint loss is used to enhance the model's robustness to the core semantics of intents. Difference constraint loss is used to guide the model to understand the semantic changes caused by differences in modality contributions. By weighted summing of these three losses, a total loss function is formed, and the parameters of the multimodal intent recognition network are iteratively updated using optimization algorithms such as gradient descent until the network converges.

[0049] In step S107, the multimodal data of the sample to be identified is input into the trained multimodal intent recognition network, and the intent category is output.

[0050] The trained multimodal intent recognition network can automatically analyze and determine the intent category implied by the input multimodal data, such as "play music," "check the weather," or "set an alarm." The output can be directly used in subsequent intelligent interaction or decision-making systems.

[0051] This embodiment effectively addresses the problem of traditional multimodal intent recognition methods lacking explicit modeling of semantic changes when dealing with imbalances and dynamic variations in modal contributions by integrating intent center consistency and semantic difference learning. This method ensures that the intent-level representation remains semantically stable even in the face of local perturbations through consistency constraint learning. Simultaneously, through difference constraint learning, the model can systematically understand the structured contributions of different modalities to intent semantics, avoiding excessive contraction of the intent representation, thereby improving the robustness and accuracy of intent recognition in complex multimodal scenarios.

[0052] In some embodiments of the present invention, step S102 performs a preset operation on the multimodal data to obtain multimodal fusion features, and extracts intent-level representations from them, specifically including the following steps: First, the text, visual, and audio modal data in the multimodal data are temporally aligned, and the aligned multimodal features are then encoded to obtain aligned text, visual, and audio feature sequences. This step aims to transform the original, heterogeneous multimodal data into a unified, high-dimensional feature representation, making it suitable for subsequent machine learning processing. Since the data from different modalities may differ in acquisition frequency and temporal granularity, temporal alignment is necessary to ensure that the features of different modalities correspond to each other in the temporal dimension, thereby accurately capturing cross-modal semantic relationships during fusion. Temporal alignment can be achieved through resampling, interpolation, dynamic time warping (DTW), or implicit alignment based on attention mechanisms. Subsequently, for text modal data, pre-trained language models based on the Transformer architecture (such as BERT, RoBERTa, etc.) can be used for encoding, mapping each word or subword to a context-dependent vector representation; for visual modal data, convolutional neural networks (such as ResNet, VGG, etc.) or visual Transformers (such as ViT) can be used to extract spatial and temporal features from image or video frames; for audio modal data, Mel-frequency cepstral coefficients (MFCCs) or deep learning-based acoustic models (such as Wav2Vec 2.0, Conformer) can be used to capture the acoustic and semantic information of speech.

[0053] Next, the aligned text feature sequence, visual feature sequence, and audio feature sequence are input into a dynamic attention fusion network to obtain multimodal fusion features. Intent-level representations are then extracted from the intent alignment regions corresponding to intent labels within these multimodal fusion features. The dynamic attention fusion network adaptively learns the importance weights between different modalities and within different time steps or feature dimensions, thereby generating a comprehensive and information-rich multimodal fusion feature. This network can employ a Transformer structure with multi-head self-attention and cross-attention mechanisms, or a fusion unit based on gating mechanisms (such as GRU or LSTM). By dynamically adjusting attention weights, the network can focus on the modal information most relevant to the current task, effectively suppressing redundancy and noise. After obtaining the multimodal fusion features, to extract the core semantic information directly related to the user's intent, intent alignment regions need to be identified and extracted from these fusion features. Intent alignment regions refer to local or global feature representations in the fusion features that are strongly correlated with a specific intent label. Extracting intent-level representations can be achieved by applying an additional attention pooling layer to the fused feature sequence and learning a weight distribution for weighted summation, resulting in a compact and discriminative intent-level representation.

[0054] By temporally aligning multimodal features, the synchronicity of different modalities in the time dimension is ensured, enabling accurate capture of cross-modal semantic correspondences during fusion and avoiding information confusion caused by temporal misalignment. Furthermore, by independently encoding textual, visual, and audio modal data, this application transforms heterogeneous raw multimodal data into a unified and semantically rich feature representation, laying the foundation for subsequent fusion. The aligned feature sequence is input into a dynamic attention fusion network, allowing the network to adaptively adjust the weights of different modalities and intramodal information based on the input content, thereby generating a high-quality, information-rich multimodal fusion feature that effectively suppresses redundancy and noise. Finally, intent-level representations are extracted from the intent-aligned regions corresponding to intent labels in the multimodal fusion features, ensuring that the obtained intent-level representations are highly focused on the core semantic information of the user's intent, significantly improving the purity and discriminative power of the intent-level representations. This refined feature processing and intent-level representation extraction mechanism provides high-quality input for subsequent consistency constraint learning and difference constraint learning, thereby effectively improving the accuracy and robustness of multimodal intent recognition.

[0055] In some embodiments of the present invention, the method for constructing a multimodal fusion feature variant in step S103 includes: constructing a local context window covering the intended alignment region, setting the features within the local context window to zero or reducing their weights to obtain a multimodal fusion feature variant with a semantic local mask; and / or, randomly masking the time-step features of the visual feature sequence and / or audio feature sequence at a random ratio r, and then re-fusing them with the text feature sequence to obtain a multimodal fusion feature variant with a random feature mask; and / or, performing suppression processing by setting zero or masking all features of one of the visual feature sequences and the audio feature sequence respectively, and then re-fusing them with the other sequence and the text feature sequence to obtain a multimodal fusion feature variant with modality suppression.

[0056] Specifically, when constructing multimodal fusion feature variants for semantic local masks, the aim is to generate variants by locally perturbing the multimodal fusion features. Intent alignment markers are set in the multimodal input sequence to explicitly indicate the sample intent semantics. These intent alignment markers correspond to a continuous feature unit after multimodal fusion, constituting an intent alignment region, which serves as the central representation of the intent semantics. Intent-related local regions refer to feature segments closely related to a specific intent expression in the multimodal fusion features. These regions are typically located in the context of the intent alignment region, i.e., the auxiliary information region surrounding the core intent expression. The intent-related local regions are determined by the position of the pre-introduced intent alignment markers in the fusion feature sequence of the multimodal fusion features. These local regions can be systematically identified and selected through a localization mechanism based on intent alignment markers. The local context window is a fixed-range feature region centered on the intent alignment region. By using it to cover the context region of the intent alignment region and part of the intent alignment region, the features of these regions are zeroed (i.e., their information is completely removed) or deweighted (i.e., their importance or intensity is reduced), introducing local information perturbation into the core intent semantics. This perturbation forces the model to learn a more robust intent representation, making it independent of a single, potentially unstable, local feature.

[0057] When constructing multimodal fusion feature variants with random feature masks, the focus is on introducing random perturbations within the modality to enhance the model's robustness to intramodal noise and information loss. Visual and audio feature sequences are typically time-series data, containing features across multiple time steps. The random masking operation involves randomly selecting a subset of time steps from these sequences and zeroing or replacing them with random values ​​according to a preset random ratio r. This operation simulates the instantaneous loss or interference that may occur in visual or audio information in the real world. After masking the visual and / or audio sequences, they are re-fused with undisturbed text feature sequences. Since the text modality is generally considered the core and relatively stable modality for intent recognition, this re-fusion ensures the preservation of core intent semantics while forcing the model to extract accurate intent from the remaining information even when visual or audio information is incomplete. For example, in a video of "playing music," if the visual image is briefly blurred or the audio has background noise, the model should still be able to recognize the intent through text information and some visual / audio information.

[0058] When constructing modality-suppressed multimodal fusion feature variants, the aim is to generate variants by completely removing information from a non-textual modality. This allows for the evaluation of the model's intent recognition capability in the case of modality absence and encourages the model to learn the complementarity between modalities. Suppression means zeroing out or completely masking all features in a visual or audio feature sequence, so that they no longer contribute information to the fusion process. For example, information from the visual modality can be completely removed, retaining only the text and audio modalities; or information from the audio modality can be completely removed, retaining only the text and visual modalities. The remaining modalities (e.g., text and audio, or text and visual) are then re-fused. This operation simulates situations where specific modal data may be completely missing in certain application scenarios. Variants generated in this way primarily carry the intent semantics of the unsuppressed modality (especially the text modality), thus ensuring the stability of the intent semantics and enabling the model to perform accurate intent recognition even when modal information is incomplete.

[0059] Through the above approach, this application provides a diverse and robust method for constructing multimodal fusion feature variants. Specifically, by utilizing a local context window mechanism to zero out or reduce the weight of features in intent-related local regions, local information loss or noise can be simulated, prompting the model to focus on more essential semantic features of the intent rather than over-relying on easily perturbed information. Simultaneously, time-step feature masking of visual and / or audio feature sequences at random proportions effectively enhances the model's resistance to random noise and incomplete information within modalities, improving its generalization performance in real-world complex environments. Furthermore, by globally suppressing visual or audio feature sequences and re-fusing them with other modalities, the model can learn how to extract complementary information from other modalities to accurately identify intents even when a specific modality is completely missing, thus significantly improving the robustness and usability of the multimodal intent recognition system in modal incomplete scenarios. These multimodal fusion feature variants ensure that the generated variants effectively perturb the input features during consistency constraint learning, enabling the model to learn more stable, robust, and generalizable intent-level representations.

[0060] In some embodiments of the present invention, step S106 calculates the consistency constraint loss using the following formula: in, The loss function; This represents the intent-level representation extracted from multimodal fusion features. This represents the intent-level representation of the multimodal fusion feature variants. This represents the similarity measurement function; B represents the preset temperature parameter; B represents the sample batch size; and i represents the sample number.

[0061] The above formula defines a contrastive learning loss that aims to force the model to learn semantically consistent intent representations by maximizing the similarity between positive sample pairs (i.e., the original intent-level representation and its semantically stable variants) while minimizing the similarity with negative sample pairs (intent-level representations of other samples within a batch). Specifically, it takes the form of a negative log-likelihood, encouraging the model to learn semantically consistent intent representations. With corresponding multimodal fusion feature variants The increased similarity makes it stand out among all possible negative samples. By minimizing The model is able to better identify and distinguish semantically consistent representations. It is the core intent representation extracted from the original, undisturbed multimodal fusion features. As the "anchor point" in consistency constraint learning, it represents the true intent semantics of the current sample. It is an intent-level representation extracted from the original multimodal fusion features after applying specific perturbations (such as masking, weight reduction, etc.). These perturbations aim to maintain the semantic stability of the core intent, therefore Should be with It exhibits high semantic consistency and serves as a "positive sample" in consistency constraint learning, used to... Conduct comparative learning. Functions are used to quantify the similarity between two intent-level representations. Common implementations include cosine similarity or dot product. Cosine similarity measures the directional consistency of two vectors by calculating the cosine of the angle between them, while the dot product measures the projected lengths of the vectors. Choosing an appropriate similarity function helps to more accurately capture the semantic proximity of the intent. Temperature parameter This is a hyperparameter used to adjust the distribution of similarity scores. It affects the sensitivity of the loss function to distinguishing between positive and negative samples by scaling the similarity values. Smaller... A higher value makes the model more sensitive to differences in similarity, thus prompting the model to learn more refined features; while a higher value... A value of B makes the model less sensitive to differences in similarity, helping it learn more generalized features. B represents the number of samples processed in one training iteration. In contrastive learning, other samples within a batch are typically considered negative samples. Therefore, the batch size B directly affects the number of negative samples, thus impacting the difficulty and efficiency of contrastive learning.

[0062] In some of the above embodiments, multimodal intent recognition methods learn consistency constraints on multimodal fusion features and their variants to ensure that the fused intent-level representation remains semantically stable. However, this constraint mainly applies to the fused feature level and may fail to fully utilize the intent information carried by each individual modality, especially when the text modality is generally considered to be the modality with the most explicit intent expression. This may lead to potential inconsistencies between the intent-level representations of each modality branch and the overall intent semantics during the fusion process, thereby affecting the accuracy and robustness of the final intent recognition.

[0063] To address this, this application further proposes extracting the corresponding intent-level representations from each modality branch of the multimodal data; using the intent-level representations corresponding to the text modality data carrying intent tags as a reference, performing consistency constraint learning on the intent-level representations corresponding to other modality branches, or simultaneously performing consistency constraint learning on the intent-level representations corresponding to other modality branches and the intent-level representations extracted from the multimodal fusion features; the consistency constraint loss is calculated using the following formula: in, An intent-level representation of text modal data carrying intent labels; It can be either the intent-level representation corresponding to other modal branches, or the set of intent-level representations corresponding to other modal branches and intent-level representations extracted from multimodal fusion features; These are the weighting coefficients.

[0064] In multimodal intent recognition networks, each modality (e.g., text, vision, audio) typically has its own independent encoder or processing path, known as modal branches. Extracting intent-level representations from each modal branch involves further refining the core semantic information that represents the intent expressed by each modality after the data has undergone its own feature extraction and encoding process. This can be achieved by adding an intent representation layer (e.g., a fully connected layer or attention mechanism layer) to the output of each modality encoder. This layer maps modality-specific features to a unified intent semantic space, thus obtaining the intent-level representation for that modality.

[0065] Textual modality, due to its direct semantic expressiveness, is often regarded as a reliable carrier of intent labels in multimodal intent recognition. The intent-level representation corresponding to textual modality data carrying intent labels is used. Using it as a reference means taking it as an "anchor" or "benchmark" for the semantic meaning of intent. This applies to the intent-level representation of other modal branches (such as visual and audio modalities). Consistency constraint learning aims to force the intent representations of these modalities to align with the intent representations of the text modality, ensuring that the intent-level representations of different modalities remain highly similar in semantic space when expressing the same intent. Furthermore, it can constrain the intent-level representations of other modal branches. With text intent level representation Maintaining consistency also allows for synchronized constraints on the intent-level representation extracted from multimodal fusion features. and Maintain consistency, that is, here. It includes not only intent-level representations of other modal branches, but also... This ensures that both single-modal intent understanding and the overall intent understanding after multimodal fusion maintain a high degree of consistency with the most reliable textual intent semantics. This consistency constraint learning is usually achieved through contrastive learning, that is, maximizing the similarity between positive sample pairs while minimizing the similarity between negative sample pairs.

[0066] The formula for calculating the consistency constraint loss above indicates that the total consistency constraint loss consists of two parts: the first part... The first part is the original consistency constraint loss for multimodal fusion features and their variants, designed to ensure the semantic stability of the fusion features; the second part... This is a newly added consistency constraint loss between the text modality intent-level representation and the intent-level representations of other modality branches (or including fused feature intent-level representations). By adding these two parts of the loss, intent constraints can be applied simultaneously at both the fused feature level and the modality branch level. Figure 1 Consistency constraints form a more comprehensive and robust intent semantic alignment mechanism.

[0067] In some embodiments of the present invention, step S105 suppresses the features of a specified modality according to the contribution degree to obtain a modality-suppressed intent-level representation, and performs difference constraint learning on the difference between the intent-level representations before and after suppression. Specifically, it includes: performing zeroing or masking suppression processing on all features of one of the visual feature sequences and the audio feature sequence respectively, then re-fusing them with the other sequence and the text feature sequence, and extracting the suppressed intent-level representation from the fused features; and calculating the semantic offset between the intent-level representations before and after suppression to obtain strong suppression semantic offset and weak suppression semantic offset, and applying asymmetric interval constraints to the strong suppression semantic offset and the weak suppression semantic offset.

[0068] Specifically, when generating the modality-suppressed intent-level representation, suppression processing can be performed on all features of any modality in both the visual and audio feature sequences. This suppression processing can be implemented in various ways. For example, by setting all feature vector elements in the feature sequence of the selected modality to zero, the information of that modality can be completely removed, preventing it from participating in the subsequent fusion process. Alternatively, a masking approach can be used, introducing a mask matrix and multiplying the feature sequence of the selected modality element-wise with the mask matrix. The elements of the mask matrix can be zero or small values ​​close to zero, thus significantly weakening the influence of the modality without completely zeroing it. After the suppression processing is completed, a new intent-level representation is extracted from the fusion features of the suppressed modality feature sequence and another unsuppressed modality feature sequence and the text feature sequence (zeroing means it does not participate in the fusion, masking means it participates or does not participate in the fusion). This new intent-level representation is the suppressed intent-level representation, which reflects the model's understanding of intent when specific modality information is missing or weakened.

[0069] Building upon this foundation, to achieve more refined differential constraint learning, this application further proposes calculating the semantic shift between the intent-level representations before and after suppression, distinguishing between strong and weak suppression semantic shifts, and then applying an asymmetric margin constraint. The semantic shift can be quantified by calculating the distance or similarity between the intent-level representations before and after suppression, for example, using metrics such as Euclidean distance or cosine distance. The determination of strong and weak suppression semantic shifts is typically related to the modality's contribution. For example, if a modality contributes significantly to the intent-level representation, the semantic shift resulting from its suppression is defined as a strong suppression semantic shift; conversely, if the contribution is low, it is a weak suppression semantic shift. This contribution can be dynamically calculated during training or pre-set. The asymmetric margin constraint aims to ensure that the semantic shift caused by strong suppression is greater than that caused by weak suppression, and that there is a pre-defined margin between the two. This can be achieved through a loss function, such as a variant of the triplet loss, setting a margin such that the strong suppression semantic shift is significantly greater than the weak suppression semantic shift, and at least greater than this margin.

[0070] Through the above scheme, this application can accurately isolate the influence of a single modality on the overall intent representation, thereby clearly comparing the semantic changes between the original intent-level representation and the intent-level representation after modality suppression. By calculating and distinguishing between strong and weak suppression semantic shifts, and applying asymmetric margin constraints, the model is explicitly guided to understand the relative importance of different modalities to the semantics of the intent. When a modality with a high contribution is suppressed, the model is encouraged to produce a larger semantic shift; while when a modality with a low contribution is suppressed, the model is encouraged to produce a smaller semantic shift. This refined differential constraint learning mechanism enables the multimodal intent recognition network to learn more robust and modality-independent intent representations. It allows the model to better maintain its understanding of intent when faced with missing modal information or noise interference, thereby improving the robustness and accuracy of multimodal intent recognition. In this way, the model can not only recognize intents but also deeply understand the contribution of different modalities to intent recognition, thus enabling more reliable judgments in practical applications even when some modal data is of poor quality or missing.

[0071] In some embodiments of the present invention, step S106 calculates the difference constraint loss using the following formula: in, To strongly suppress semantic shift; This represents the intention level before suppression; This is the suppressed intention-level representation, in which high-contribution modalities are suppressed; This represents a weakly suppressed semantic shift. The suppressed intention level representation, in which low-contribution modalities are suppressed.

[0072] The core function of this difference-constrained loss formula is to quantify and enforce the asymmetric margin constraint between strongly suppressed and weakly suppressed semantic shifts. By introducing a preset margin parameter *m*, this loss function aims to ensure that when a high-contribution mode is suppressed, the resulting semantic shift (strongly suppressed semantic shift) must be significantly greater than the semantic shift produced when a low-contribution mode is suppressed (weakly suppressed semantic shift), and that there is at least a margin of *m* between the two. This helps the model learn the differences in modal contributions and makes its dependence on key modes more explicit.

[0073] Among them, the intention level representation before suppression Compared to the original intent-level representation extracted directly from multimodal fusion features same. This serves as a benchmark reference point for measuring the effectiveness of modality suppression. When calculating semantic shifts, all intent-level representations after modality suppression are compared to this benchmark reference point. The margin parameter *m* is a preset positive value used to enforce a minimum margin between strongly suppressed and weakly suppressed semantic shifts in the difference-constrained loss. Specifically, it requires that the strongly suppressed semantic shift be at least *m* greater than the weakly suppressed semantic shift. If this condition is not met, the loss function will produce a positive value, prompting the model to adjust its parameters during training to satisfy this constraint. By adjusting the value of *m*, the model's sensitivity to differences in modality contribution can be controlled.

[0074] By employing the aforementioned formula for calculating the difference-constrained loss, this application can accurately quantify and effectively enforce the asymmetric margin constraint between strong and weak suppression semantic shifts. Specifically, the formula, by introducing a margin parameter *m*, explicitly requires that the semantic shift caused by modalities with high suppression contributions must be significantly greater than the semantic shift caused by modalities with low suppression contributions. This allows the multimodal intent recognition network to more clearly learn the true differences in the contributions of different modalities to the final intent-level representation during training. When the model attempts to minimize this loss, it is guided to construct an intent-level representation where the absence of key modalities leads to greater semantic changes, while the absence of non-key modalities leads to smaller semantic changes. This explicit distinction helps improve the model's ability to perceive modal contributions, enhances the model's robustness to partial modal information loss or noise, and improves the accuracy and interpretability of multimodal intent recognition.

[0075] Figure 2 This is a flowchart of a multimodal intent recognition method that integrates intent center consistency and semantic difference learning in another embodiment of the present invention. Figure 2The process of consistency constraint learning (i.e., intention-centered consistency learning) is illustrated in detail below. Figure 2 Further explanation of embodiments of the present invention follows.

[0076] like Figure 2 As shown, the multimodal intent recognition method that integrates intent center consistency and semantic difference learning includes the following steps: Encoding and Alignment Steps: Multimodal data, such as audio, visual, and text data, are temporally aligned and then input into their respective audio encoder, visual encoder, and text encoder for feature encoding, resulting in text feature sequences. Visual feature sequences and audio feature sequences .

[0077] Intent-centered consistency learning steps: Four multimodal fusion feature variants are constructed using three methods: semantic local masking, random feature masking, and adaptive modality suppression. These are the multimodal fusion feature variants based on semantic local masking. , a multimodal fusion feature variant of random feature mask and a multimodal fusion feature variant with two modes of suppression. Among them, It is a multimodal fusion feature based on the output of the dynamic attention fusion module. Constructed. It integrates information from three modalities: text, visual, and audio. Random feature masking involves randomly masking partial features from the visual and audio modalities to obtain the masked audio features. and the visual features behind the mask ; , With the above text feature sequences The input dynamic attention fusion module outputs a multimodal fusion feature variant with a random feature mask. Adaptive modality suppression distinguishes visual and audio modalities into stronger and weaker modalities based on modality importance, and employs an adaptive modality dropout mechanism to suppress the stronger modality and combine it with the other modality to form a set of suppressed strong modality features. The weaker mode is suppressed and then combined with another mode to form a set to form a suppressed weak mode feature. ; , Compared with the above text feature sequences respectively The input dynamic attention fusion module outputs a multimodal fusion feature variant with two modal suppressions. Multimodal fusion features Intent-level representation is obtained after averaging the intents. The four constructed multimodal fusion feature variants are then averaged using intent to obtain the multimodal fusion feature variants of the semantic local mask. Corresponding intent-level representation Intent-level representation corresponding to multimodal fusion feature variants of random feature masks And the intent-level representation corresponding to the multimodal fusion feature variants of the two modal suppressions. The intent averaging refers to the aggregation of feature vectors from multiple consecutive feature units within a predetermined intent alignment region in the fused feature sequence to obtain an intent-level representation of the sample. This aggregation process includes average pooling operations on the intent-related feature subsequences along the feature dimension or the time dimension to obtain an intent-level representation that characterizes the semantic intent of the current sample.

[0078] Intent classification steps: Training is performed using a unified objective that combines supervised intent classification with intent-center consistency learning (i.e., consistency constraint learning) and semantic difference learning (i.e., difference constraint learning) objectives. Through joint optimization of the intent-center consistency learning loss (i.e., consistency constraint loss), the semantic difference learning loss (differential constraint loss), and the intent classification cross-entropy loss, the training objective simultaneously enforces intent-level consistency across views while preserving structured semantic differences caused by modality contribution imbalances. Based on the final intent representation, intent category prediction is performed, and recognition results are output, such as the recognition results for "thank you," "agree," "greet," "joke," and "stop."

[0079] For detailed technical information regarding the above embodiments, please refer to [link / reference]. Figure 1 The embodiments shown are not described in detail here.

[0080] The following example will provide a more detailed explanation of the above technical solution: In a smart home scenario, user A issues a command to the smart assistant, such as saying "Turn on the living room lights," accompanied by a visual gesture pointing to the living room lights. The smart assistant needs to accurately recognize user A's intention.

[0081] First, the system acquires multimodal data from user A, including voice and visual data. This raw data then enters a pre-defined operational process. Specifically, the multimodal data is temporally aligned to ensure that information from different modalities corresponds to each other in the time dimension. Next, the voice data is processed by an audio encoder to generate an audio feature sequence; the visual data is processed by a visual encoder to generate a visual feature sequence; if user A simultaneously inputs text (e.g., via a touchscreen), the text data is processed by a text encoder to generate a text feature sequence. The aligned text feature sequence, visual feature sequence, and audio feature sequence are then input into a dynamic attention fusion network. This network adaptively allocates attention weights to deeply fuse information from different modalities, resulting in a multimodal fusion feature. From this multimodal fusion feature, the system extracts the intent alignment region corresponding to the intent label "turn on the living room lights," thus obtaining an intent-level representation, which is the core semantic carrier of user A's intent.

[0082] To enhance the robustness and stability of intent-level representations, the system constructs multimodal fusion feature variants. For example, the system utilizes local context windowing techniques to zero out or reduce the weight of features within a local context window, generating a multimodal fusion feature variant with a semantic local mask. Alternatively, the system can randomly mask time-step features in visual or audio feature sequences at a random ratio *r*, and then re-fuse them with the text feature sequence to obtain a multimodal fusion feature variant with a random feature mask. Or, the system can perform zeroing or masking suppression on all features of one sequence in the visual or audio feature sequence, and then re-fuse it with the other sequence and the text feature sequence to obtain a multimodal fusion feature variant with modality suppression. Using the original intent-level representation as a reference, the system performs consistency constraint learning on the intent-level representations corresponding to these multimodal fusion feature variants. By calculating the consistency constraint loss, for example using contrastive learning, the system ensures that the intent-level representations maintain high similarity even under reasonable perturbations. This avoids the problem in existing technologies where all augmented views are treated as semantically equivalent positive samples, leading to excessive shrinkage of the intent representation, and improves the model's ability to recognize reasonable perturbations that preserve intent.

[0083] During multimodal data fusion, the system calculates the contribution of each modality to the intent-level representation. For example, if user A's voice command is very clear, while the visual gesture is relatively blurry, the audio modality may contribute more to the intent "turn on the living room lights" than the visual modality.

[0084] Based on the calculated contribution, the system suppresses features of a specified modality to obtain a modality-suppressed intent-level representation. For example, if the visual modality's contribution is relatively low, the system will perform zeroing or masking suppression on all features of the visual feature sequence, then re-fuse the suppressed visual sequence with the audio and text feature sequences, and extract the suppressed intent-level representation from the fused features. Subsequently, the system calculates the semantic shift between the intent-level representations before and after suppression. Specifically, the system calculates the semantic shift between the original intent-level representation and the intent-level representation obtained after suppressing high-contribution modalities (e.g., assuming the audio modality has the highest contribution and suppressing it), which is called strong suppression semantic shift. Simultaneously, the system also calculates the semantic shift between the original intent-level representation and the intent-level representation obtained after suppressing low-contribution modalities (e.g., assuming the visual modality has a low contribution and suppressing it), which is called weak suppression semantic shift. The system applies an asymmetric margin constraint to both strong and weak suppression semantic shifts. This means the system is trained to understand that suppressing a modality with a high contribution to intent should result in a larger semantic shift than suppressing a modality with a low contribution. This explicit modeling of semantic changes caused by differences in modal contributions solves the problem that existing technologies struggle to characterize the structured semantic differences dominated by different modalities, and guides the model to understand the relative semantic roles of different modalities in the intent space during the training phase.

[0085] Finally, the system calculates the consistency constraint loss and the difference constraint loss. These losses, together with the standard intent classification cross-entropy loss, constitute the total loss. Based on this total loss, the multimodal intent recognition network is trained. Through this multi-objective optimization, the network not only learns how to accurately classify intents, but also learns how to handle semantic consistency and differences between modalities, thereby obtaining a more stable and discriminative intent representation in scenarios with multimodal semantic heterogeneity and uneven contributions.

[0086] After sufficient training, when multimodal data of new samples to be identified (such as user B's "play music" command and its accompanying visual gesture) is input into the network, the network can output an accurate intent category. This method models intent simultaneously in the intent-level representation space. Figure 1 By combining consistency and modal contribution differences, the overall performance and generalization ability of multimodal intent recognition are improved, overcoming the shortcomings of existing technologies that only focus on weight allocation during the fusion process and lack explicit modeling of semantic changes caused by modal contribution differences.

[0087] Corresponding to the above method, the present invention also provides a multimodal intent recognition system that integrates intent center consistency and semantic difference learning. The system includes an acquisition module for acquiring multimodal data of the sample to be recognized; an extraction module for performing preset operations on the multimodal data to obtain multimodal fusion features and extracting intent-level representations from them; a consistency constraint learning module for constructing multimodal fusion feature variants and performing consistency constraint learning on the intent-level representations corresponding to the multimodal fusion feature variants, using the intent-level representations as a reference; a calculation module for calculating the contribution of different modalities to the intent-level representation based on information from the multimodal data fusion process; a difference constraint learning module for suppressing features of a specified modality according to the contribution, obtaining a modality-suppressed intent-level representation, and performing difference constraint learning on the difference between the intent-level representations before and after suppression; a training module for calculating the consistency constraint loss and the difference constraint loss, and training the multimodal intent recognition network based on the intent classification cross-entropy loss, the consistency constraint loss, and the total loss formed by the difference constraint loss; and an intent recognition module for inputting the multimodal data of the sample to be recognized into the trained multimodal intent recognition network and outputting the intent category.

[0088] The core innovation of this embodiment lies in its collaborative combination of intent center consistency learning and semantic difference learning. This explicitly models the semantic changes caused by modal contribution differences and distinguishes between intent preservation perturbations and true semantic shifts, achieving a more stable and discriminative intent representation in scenarios with multimodal semantic heterogeneity and uneven contributions. Specifically, the system constructs multimodal fusion feature variants through the consistency constraint learning module, ensuring the model maintains the stability of intent semantics even when facing local perturbations. Simultaneously, the computation module accurately calculates the contribution of each modality based on the fusion process information, and the difference constraint learning module performs suppression operations on specified modal features based on this contribution, explicitly modeling the semantic shift difference before and after suppression. This dual-track learning mechanism effectively solves the problem of excessive contraction of intent representation in existing technologies, enabling the model to distinguish between reasonable semantic perturbations and true semantic shifts, thereby guiding the model to understand the relative semantic roles of different modalities at the training level.

[0089] In practical applications, the system first receives multimodal data input containing text, visual, and audio data through an acquisition module. The extraction module performs feature encoding and temporal alignment on this data, generating aligned feature sequences. These sequences are then processed by a dynamic attention fusion network to obtain multimodal fusion features, thereby extracting an intent-level representation representing the sample's intent. Based on this, a consistency constraint learning module uses a local context window to determine the intent alignment region and, after covering it, sets the features to zero or reduces their weight. Alternatively, it randomly masks the visual / audio feature sequences, constructing multimodal fusion feature variants with semantic local masks or random feature masks. A contrastive learning mechanism constrains the semantic consistency between the variants and the original representation. The computation module analyzes the weight distribution or sensitivity of each modality during the fusion process, quantifying the contribution of different modalities to the intent-level representation. The difference constraint learning module, based on this contribution information, performs feature suppression operations on high-contribution modalities or low-contribution modalities respectively, generating modality-suppressed intent-level representations and calculating the asymmetric margin constraint between strongly suppressed semantic shifts and weakly suppressed semantic shifts. The training module weights and sums the cross-entropy loss, consistency constraint loss, and difference constraint loss for intent classification to form a total loss function, driving the parameter optimization of the multimodal intent recognition network. Finally, the intent recognition module uses the trained network model to determine the intent category of newly input multimodal data.

[0090] Through the above approach, this system can effectively capture the structured differences in intent semantics in complex scenarios with multimodal semantic heterogeneity and uneven contributions, avoiding the semantic information loss caused by overemphasizing representation consistency in traditional methods. This system not only improves the accuracy and robustness of intent recognition but also enhances the model's ability to utilize diverse semantic cues, providing more reliable technical support for applications such as intelligent human-computer interaction and intelligent customer service.

[0091] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0092] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0093] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0094] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal intent recognition method integrating intent center consistency and semantic difference learning, characterized in that, include: Acquire multimodal data of the sample to be identified; The multimodal data is subjected to preset operations to obtain multimodal fusion features, and intent-level representations are extracted from them; Construct a multimodal fusion feature variant, and use the intent-level representation as a reference to perform consistency constraint learning on the intent-level representation corresponding to the multimodal fusion feature variant; Based on the information from the multimodal data fusion process, the contribution of different modalities to the intent-level representation is calculated; Based on the contribution, the features of the specified modality are suppressed to obtain the intention-level representation after modality suppression, and the difference between the intention-level representations before and after suppression is subjected to difference constraint learning. Calculate the consistency constraint loss and the difference constraint loss, and train the multimodal intent recognition network based on the intention classification cross-entropy loss, the consistency constraint loss and the difference constraint loss. The multimodal data of the sample to be identified is input into the trained multimodal intent recognition network, which outputs the intent category.

2. The method according to claim 1, characterized in that, The multimodal data undergoes preset operations to obtain multimodal fusion features, from which intent-level representations are extracted, including: The text modal data, visual modal data, and audio modal data in the multimodal data are temporally aligned, and the aligned multimodal features are respectively encoded to obtain aligned text feature sequences, visual feature sequences, and audio feature sequences. The aligned text feature sequence, visual feature sequence, and audio feature sequence are input into a dynamic attention fusion network to obtain multimodal fusion features, and intent-level representations are extracted from the intent alignment regions corresponding to intent labels in the multimodal fusion features.

3. The method according to claim 2, characterized in that, The multimodal fusion feature variants include: Construct a local context window covering the intended alignment region, and set the features within the local context window to zero or reduce their weight to obtain a multimodal fusion feature variant of the semantic local mask; and / or, The visual feature sequence and / or audio feature sequence are randomly masked at time step ratios r, and then re-fused with the text feature sequence to obtain a multimodal fusion feature variant with random feature masks; and / or, The visual feature sequence and the audio feature sequence are subjected to suppression processing, which involves setting all features to zero or masking them. Then, they are re-fused with the other sequence and the text feature sequence to obtain a multimodal fusion feature variant with modal suppression.

4. The method according to claim 1, characterized in that, The consistency constraint loss is calculated using the following formula: in, The loss function; This represents the intent-level representation extracted from multimodal fusion features. This represents the intent-level representation of the multimodal fusion feature variants. This represents the similarity measurement function; B represents the preset temperature parameter; B represents the sample batch size.

5. The method according to claim 4, characterized in that, Also includes: Extract the corresponding intent-level representation from each modal branch of the multimodal data; Taking the intent-level representation corresponding to the text modality data carrying intent labels as a reference, consistency constraint learning is performed on the intent-level representations corresponding to other modality branches, or consistency constraint learning is performed simultaneously on the intent-level representations corresponding to other modality branches and the intent-level representations extracted from multimodal fusion features. The consistency constraint loss is calculated using the following formula: in, An intent-level representation of text modal data carrying intent labels; It can be either the intent-level representation corresponding to other modal branches, or the set of intent-level representations corresponding to other modal branches and intent-level representations extracted from multimodal fusion features; These are the weighting coefficients.

6. The method according to claim 2, characterized in that, The step of suppressing features of a specified modality based on the contribution level to obtain a modality-suppressed intent-level representation, and performing difference constraint learning on the difference between the intent-level representations before and after suppression, includes: Suppression processing, namely zeroing or masking, is performed on all features of one of the visual feature sequences and the audio feature sequence, respectively. Then, the two sequences are re-fused with the other sequence and the text feature sequence, and the suppressed intent-level representation is extracted from the fused features. The semantic offset between the intent-level representation before and after suppression is calculated to obtain the strong suppression semantic offset and the weak suppression semantic offset, and an asymmetric interval constraint is applied to the strong suppression semantic offset and the weak suppression semantic offset.

7. The method according to claim 6, characterized in that, The difference constraint loss is calculated using the following formula: in, To strongly suppress semantic shift; This represents the intention level before suppression; This is the suppressed intention-level representation, in which high-contribution modalities are suppressed; This represents a weakly suppressed semantic shift. The suppressed intention level representation, in which low-contribution modalities are suppressed.

8. A multimodal intent recognition system integrating intent center consistency and semantic difference learning, characterized in that, include: The acquisition module is used to acquire multimodal data of the sample to be identified; The extraction module is used to perform preset operations on the multimodal data to obtain multimodal fusion features and extract intent-level representations from them; The consistency constraint learning module is used to construct a multimodal fusion feature variant and, with the intent-level representation as a reference, perform consistency constraint learning on the intent-level representation corresponding to the multimodal fusion feature variant. The calculation module is used to calculate the contribution of different modalities to the intent-level representation based on the information in the multimodal data fusion process. The difference constraint learning module is used to suppress the features of a specified modality based on the contribution degree, obtain the intention-level representation after modality suppression, and perform difference constraint learning on the difference between the intention-level representations before and after suppression. The training module is used to calculate the consistency constraint loss and the difference constraint loss, and to train the multimodal intent recognition network based on the total loss formed by the intent classification cross-entropy loss, the consistency constraint loss and the difference constraint loss. The intent recognition module is used to input the multimodal data of the sample to be recognized into the trained multimodal intent recognition network and output the intent category.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images