Multimodal intent recognition method and apparatus, computer device, and storage medium

Abstract

The application relates to a multi-modal intention recognition method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises: acquiring modal features of each mode; acquiring semantics of the modal features at each time step; determining the causal edge weight of the semantics at each time step and an intention label in a directed acyclic graph according to an adjacency matrix and the semantics at each time step; screening the modal features of each mode according to the causal edge weight to obtain key modal features of each mode; projecting the key modal features of each mode into hyperbolic space through a hyperbolic embedding function to obtain the modal features of each mode in the hyperbolic space; performing cross-attention fusion on the semantics of the multi-modal features in the hyperbolic space to obtain fused features; and classifying the fused features to obtain an intention label. The method can screen key modal features and more accurately recognize intentions through the multi-modal features in the hyperbolic space.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a multimodal intent recognition method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] Multimodal intent recognition (MIR) aims to infer user intent by jointly analyzing linguistic, visual, and audio cues. As a core task of multimodal understanding, MIR has been widely applied in virtual assistants and conversational artificial intelligence. Traditional techniques capture asynchronous interactions through cross-modal converters, untangling the entanglement between modality invariance and modality-specific representations, and integrate pre-trained language models and gated multimodal attention to achieve multimodal intent recognition.

[0003] However, in traditional methods, pre-trained language models and gated multimodal attention lack multimodal noise reduction techniques, resulting in relatively large noise levels. Consequently, the alignment effect of multimodal intent recognition is relatively poor and needs improvement.

[0004] They often struggle to filter out irrelevant or noisy mode-specific signals and rely excessively on Euclidean calibration, which limits their ability to model causality and structural heterogeneity. Summary of the Invention

[0005] Therefore, it is necessary to provide a multimodal intent recognition method, apparatus, computer device, computer-readable storage medium, and computer program product that can reduce noise and improve alignment performance in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a multimodal intent recognition method, the method comprising:

[0007] Obtain the modal features for each mode;

[0008] Obtain the semantics of the modal features at each time step; determine the causal edge weights of the semantics and intent labels at each time step in the directed acyclic graph based on the adjacency matrix and the semantics of each time step; filter the modal features of each modality based on the causal edge weights to obtain the key modal features of each modality.

[0009] By using a hyperbolic embedding function, the key modal features of each modality are projected into the hyperbolic space to obtain the modal features of each modality in the hyperbolic space;

[0010] The semantics of multiple modal features in the hyperbolic space are fused by cross-attention to obtain fused features;

[0011] The fused features are classified to obtain intent labels.

[0012] In one embodiment, the training step of the adjacency matrix includes:

[0013] Based on the initial adjacency matrix, the semantic samples of the modal features at each time step are adjusted to obtain the reconstructed values ​​of the semantic samples;

[0014] A first loss value is determined based on the difference between the reconstructed value of the semantic sample and the semantic sample at each time step;

[0015] A second loss value is determined based on the difference between the predicted value of the intent label corresponding to the semantic sample and the intent label sample.

[0016] Based on the first loss value and the second loss value, the initial adjacency matrix is ​​adjusted to obtain the trained adjacency matrix.

[0017] In one embodiment, adjusting the initial adjacency matrix based on the first loss value and the second loss value to obtain the trained adjacency matrix includes:

[0018] An exponential operation is performed using the element-wise product of the initial adjacency matrix as the exponent, yielding the exponential operation result; a matrix tracking operation is performed on the exponential operation result, yielding the matrix tracking result; the difference between the matrix tracking result and the total number of time steps of the semantic samples is determined; the initial adjacency matrix is ​​adjusted based on the first loss value and the second loss value, constrained by the difference value, to obtain the trained adjacency matrix; and / or,

[0019] The absolute value of the vector transformed from the initial adjacency matrix is ​​determined, and the absolute value of the vector is adjusted by sparse coefficients to obtain a third loss value; based on the first loss value, the second loss value and the third loss value, the initial adjacency matrix is ​​adjusted to obtain the trained adjacency matrix.

[0020] In one embodiment, the training method for the hyperbolic embedding function includes:

[0021] The key modal feature samples of each modality are projected into each modal feature sample in the hyperbolic space through the initial hyperbolic embedding function; the geodesic distance is calculated based on the common semantics between each modal feature sample in the hyperbolic space to obtain the first hyperbolic loss value;

[0022] Obtain intent label samples and cue symbols for classification, concatenate the intent label samples and the cue symbols and convert them into label pin samples in the hyperbolic space, calculate the geodesic distance based on the semantics between each modal feature sample in the hyperbolic space and the label pin, and obtain the second hyperbolic loss value;

[0023] Based on the first hyperbolic loss value and the second hyperbolic loss value, the initial hyperbolic embedding function is trained to obtain the hyperbolic embedding function.

[0024] In one embodiment, the semantics of the various modal features in the hyperbolic space includes each target modal semantic and non-target modal semantic; the non-target modal semantic is the semantic concatenation result of various modalities other than the target modal semantic among the semantics of the various modal features in the hyperbolic space;

[0025] The semantic cross-attention fusion of multiple modal features in the hyperbolic space yields fused features, including:

[0026] Cross-attention calculation is performed based on the query vector corresponding to each target modality semantic, and the key vector and value vector corresponding to the non-target modality semantic, to obtain each feature to be fused;

[0027] Multiple features to be fused are spliced ​​together to obtain fused features.

[0028] In one embodiment, obtaining the modal features of each modality includes:

[0029] Obtain the raw data for each modality;

[0030] The original data is split and features are extracted according to the unit length of each modality to obtain the embedding features of each modality at each unit length;

[0031] Temporal granularity alignment is performed based on the embedding features of each modality at each unit length to obtain the modal features of each modality.

[0032] Secondly, this application also provides a multimodal intent recognition device, comprising:

[0033] The feature extraction module is used to obtain the modal features of each modality;

[0034] The causal filtering module is used to obtain the semantics of the modal features at each time step; determine the causal edge weights of the semantics and intent labels at each time step in the directed acyclic graph based on the adjacency matrix and the semantics of each time step; and filter the modal features of each modality based on the causal edge weights to obtain the key modal features of each modality.

[0035] The hyperbolic embedding module is used to project the key modal features of each modality into the hyperbolic space through the hyperbolic embedding function, so as to obtain each modal feature in the hyperbolic space;

[0036] An attention module is used to perform cross-attention fusion on the semantics of multiple modal features in the hyperbolic space to obtain fused features;

[0037] The classification module is used to classify the fused features to obtain intent labels.

[0038] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the multimodal intent recognition steps described in any of the above embodiments.

[0039] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of multimodal intent recognition in any of the above embodiments.

[0040] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of multimodal intent recognition in any of the above embodiments.

[0041] The aforementioned multimodal intent recognition method, apparatus, computer device, computer-readable storage medium, and computer program product determine causal edge weights based on the adjacency matrix and the semantics of each modality at each time step. These causal edge weights represent the relationship between modal features and intent labels, thereby determining whether the semantics of modal features at each time step affect the intent label. By using causal edge weights for filtering, the causal effectiveness of each modality can be identified, and non-causal components can be explicitly filtered out. This allows for enhanced intent understanding through the selected key modal features, resulting in stable key modal features highly correlated with the intent label. Furthermore, the key modal features are transformed into a hyperbolic space, where semantic alignment and fusion of different modalities are performed. This allows for a more accurate reflection of the structural heterogeneity of each modal feature through the various modal features in the hyperbolic space, enhancing cross-modal coherence and intent-level discriminability. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram illustrating the application of multimodal intent recognition methods in traditional technologies.

[0044] Figure 2 This is a diagram illustrating the application environment of a multimodal intent recognition method in one embodiment.

[0045] Figure 3 This is a flowchart illustrating a multimodal intent recognition method in one embodiment;

[0046] Figure 4 This is a schematic diagram of the adjacency matrix training process in one embodiment;

[0047] Figure 5 This is a schematic diagram of the adjacency matrix training process in another embodiment;

[0048] Figure 6 This is a schematic diagram of the C-HAP framework process in an exemplary embodiment;

[0049] Figure 7 This is a schematic diagram of the hyperbolic embedding function training process in one embodiment;

[0050] Figure 8 This is a schematic diagram illustrating the effect of the C-HAP framework in one embodiment and the corresponding comparative example.

[0051] Figure 9 This is a structural block diagram of a multimodal intent recognition device in one embodiment;

[0052] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0055] In some related technologies, intent recognition (IR) is a fundamental ability that enables artificial intelligence to develop human-like empathy. Its main goal is to enable AI systems to accurately infer users' potential intentions, thereby promoting more natural and intelligent human-computer interaction. With the rapid development of applications such as virtual assistants, multimodal dialogue systems, and video-based question-answering, relying solely on linguistic cues is no longer sufficient to capture complex and subtle user intentions. Therefore, multimodal intent recognition has emerged, aiming to integrate complementary signals from language, vision, and audio, enabling AI to perceive human intentions from multiple perspectives.

[0056] Multimodal Intent Recognition (MIR) introduces a cross-modal converter to capture asynchronous interactions, untangling the entanglement between modality invariance and modality-specific representations. It integrates a pre-trained language model and gated multimodal attention to improve robustness in text-dominated settings. To enhance intermodal consistency, a marker-level contrastive objective and modality-aware projection are proposed to mitigate feature transfer. A cross-video semantic memory is constructed to handle ambiguous expressions and support global consistency. Multi-view contrastive learning is combined with dynamic attention fusion to better capture informational cues under complex conditions.

[0057] Intent recognition remains a significant challenge in real-world multimodal interactions. This embodiment highlights two important but often overlooked factors behind this difficulty. First, existing methods typically assume that all modalities contribute equally to intent prediction, often aggregating features through simple connectivity or attention mechanisms. However, nonverbal morphology—such as facial expressions or audio—often contains irrelevant or misleading signals (e.g., irrelevant gestures or background noise), such as… Figure 1 As shown, this blurs the clues related to intent and reduces prediction performance.

[0058] Most MIR methods embed all modalities into a Euclidean space, making it difficult to capture their structural heterogeneity. Language is inherently discrete and syntactic, while visual and auditory forms are continuous, hierarchical, and often ambiguous. Projecting these heterogeneous signals into a shared Euclidean space oversimplifies their structural differences, leading to suboptimal semantic alignment.

[0059] Therefore, research has found that existing methods often struggle to filter out irrelevant or noisy modality-specific signals and rely excessively on Euclidean calibration, limiting their ability to model causal relationships and structural heterogeneity. These limitations are addressed through causality-aware feature selection and hyperbolic embedding to improve multimodal alignment. Hyperbolic geometry, characterized by exponential volume growth and hierarchical inductive bias, is increasingly used in deep learning due to its ability to model structured data. Recent research has shown that it outperforms Euclidean space in both aspects. In particular, hyperbolic space can better organize features of hierarchical and sparse data distributions, especially in capturing complex causal relationships across modalities.

[0060] The multimodal intent recognition method provided in this application embodiment can be applied to, for example... Figure 2 In the application environment shown, terminal 202 communicates with server 204 via a network. The data storage system can store the data that server 204 needs to process. The data storage system can be integrated onto server 204, or it can be located in the cloud or on other network servers.

[0061] The terminal 202 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 204 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0062] In one exemplary embodiment, such as Figure 3 As shown, a multimodal intent recognition method is provided, which can be applied to... Figure 2 Taking server 204 as an example, the explanation includes the following steps 302 to 310, wherein:

[0063] Step 302: Obtain the modal features of each modality.

[0064] Modal features are the results of feature extraction from data of each modality. Through modal features, data from different modalities can be transformed into modal features that the model can process, thereby facilitating subsequent feature extraction, fusion, and alignment.

[0065] In some embodiments, each modality includes at least one modality among language, video, or audio; for example, the semantics of the tags can be incorporated into the context model based on the number of text tokens to obtain the modal features of the language, i.e., language features.

[0066] Step 304: Obtain the semantics of modal features at each time step; determine the causal edge weights of the semantics and intent labels at each time step in the directed acyclic graph based on the adjacency matrix and the semantics of each time step; filter the modal features of each modality based on the causal edge weights to obtain the key modal features of each modality.

[0067] The semantics of a time step is the semantic representation of each unit of time. Each modality feature can exist at multiple time steps, and each time step has its own semantics. For example, each time step of a video feature is divided according to video frames, so feature extraction can be performed on the semantics of each video frame to obtain the semantics of the video feature at each time step.

[0068] An adjacency matrix represents the mapping relationship between modal features and intent labels, and can be a structure of a neural network model. Causal edge weights are the directed edge weights in a directed acyclic graph (DAG). These weights represent the relationship between modal features and intent labels, thus determining whether the semantics of the modal features at each time step affect the intent label. DAGs provide constraints for the causal edge weights, preventing self-loops during adjacency matrix training, thus enabling faster training and ensuring relatively accurate causal edge weights.

[0069] Key modal features are the filtering results based on causal edge weights, representing modal features that have a strong correlation with the intent label. By using causal edge weights for filtering, the causal effectiveness in each modality can be identified, and non-causal components can be explicitly filtered out. This enhances intent understanding through the selected key modal features, resulting in stable key modal features that are highly correlated with the intent label.

[0070] In some embodiments, determining the causal edge weights of the semantics and intent labels of each time step in the directed acyclic graph based on the adjacency matrix and the semantics of each time step includes: adjusting the eigenvalues ​​of the semantics of each time step according to the adjacency matrix to obtain the semantic reconstruction value of each time step; the semantic reconstruction value is the causal edge weight.

[0071] In some embodiments, modal features of each modality are filtered according to causal edge weights to obtain key modal features of each modality, including: filtering key modal features with causal edge weights greater than a threshold from the modal features of each modality; wherein, the causal edge weights are positively correlated with the correlation between the modal features and the intent label.

[0072] Step 306: Project the key modal features of each modality into the hyperbolic space using the hyperbolic embedding function to obtain the modal features of each modality in the hyperbolic space.

[0073] Hyperbolic embedding functions include trainable parameters that transform key modal features into a hyperbolic space. Hyperbolic space is characterized by exponential volume growth and hierarchical inductive bias; exponential volume growth means that hyperbolic space has more space to accommodate more points or features, thus accommodating the relationships that each modal feature can form with its causal edges; and hierarchical structure means that modal features closer to the origin represent more important influences, while modal features farther from the origin represent smaller causal influences.

[0074] Since Euclidean space is not characterized by exponential volume growth and hierarchical inductive bias, it cannot simulate causal relationships and structural heterogeneity, making it difficult to comprehensively display causal edge weights and key modal features. However, when performing semantic alignment and fusion of different modalities in hyperbolic space, the structural heterogeneity of each modal feature can be more accurately reflected through multiple modal features in hyperbolic space, enhancing cross-modal coherence and intent-level discriminability, so that cross-modal semantic fusion has a more accurate alignment effect.

[0075] In some embodiments, the key modal features of each modality are projected into each modal feature in hyperbolic space by using a hyperbolic embedding function, including: projecting the key modal features of each modality from Euclidean space into hyperbolic space by using the projection parameters of the hyperbolic embedding function to obtain each modal feature in hyperbolic space.

[0076] In some embodiments, key modal features of each modality are projected into a hyperbolic space using a hyperbolic embedding function to obtain each modal feature in the hyperbolic space. This includes: determining each modal feature in the hyperbolic space and projecting causal edges into the hyperbolic space using the hyperbolic embedding function to obtain each modal feature in the hyperbolic space. Thus, projection using key modal features and causal edges ensures effective comparison.

[0077] Step 308: Perform cross-attention fusion on the semantics of multiple modal features in hyperbolic space to obtain fused features.

[0078] The fused features are the result of cross-attention fusion of semantics from at least two modalities. When using cross-attention fusion, semantics from different modalities can be fused, so that the intra-modal causal reasoning process in steps 302-306 is changed to the cross-modal alignment and collaborative fusion process in step 308.

[0079] In some embodiments, the semantics of multiple modal features in hyperbolic space are fused by cross-attention to obtain fused features, including: using the semantics of the modal features of one modality in hyperbolic space as a query vector, and using the semantic concatenation results of the modal features of other modalities as key vectors and value vectors to perform cross-attention calculation to obtain the features to be fused for this modality; and then fusing the features to be fused for each modality to obtain fused features.

[0080] Step 310: Classify the fused features to obtain intent labels.

[0081] Intent labels are the results of multimodal intent recognition, used to represent the content of the multimodal intent. During training, intent labels are the predicted values ​​of the multimodal intent labels; after training is complete, intent labels are the recognized output results. Intent labels can be the output label values.

[0082] In some embodiments, classifying the fused features to obtain an intent label includes: predicting the probability of the category to which the fused features belong using a classifier model to obtain a probability value; and using the category corresponding to the probability value as the intent label. The classifier model may be a multilayer perceptron (MLP).

[0083] In the aforementioned multimodal intent recognition method, causal edge weights are determined based on the adjacency matrix and the semantics of each modality at each time step. These weights represent the relationship between modal features and intent labels, thereby determining whether the semantics of modal features at each time step affect the intent label. By using causal edge weights for filtering, the causal effectiveness of each modality can be identified, and non-causal components can be explicitly filtered out. This allows for enhanced intent understanding through the selected key modal features, resulting in stable key modal features highly correlated with the intent label. Furthermore, the key modal features are transformed into a hyperbolic space, where semantic alignment and fusion of different modalities are performed. This allows for a more accurate reflection of the structural heterogeneity of each modal feature through multiple modal features in the hyperbolic space, enhancing cross-modal coherence and intent-level discriminability. This filters out false and redundant signals, paving the way for a more robust and interpretable intent understanding system.

[0084] In some embodiments, obtaining modal features for each modality includes: obtaining the original data for each modality; splitting and extracting features from the original data according to the unit length of each modality to obtain the embedding features of each modality at each unit length; and performing temporal granular alignment based on the embedding features of each modality at each unit length to obtain the modal features for each modality.

[0085] The raw data is data that has not been converted into feature vectors; it can be text tokens, video, or audio. The unit length is the time length corresponding to the modality. The unit length for text can be divided according to text tokens, the unit length for video is divided according to video frames, and the unit length for audio is divided according to audio frames.

[0086] Temporal granularity represents the time length corresponding to a unit length. By aligning temporal granularity, the modal features of each modality can have time steps of the same length, which facilitates the semantic extraction of each time step and the cross-attention fusion processing.

[0087] In some embodiments, temporal granular alignment is performed based on the embedding features of each modality at each unit length to obtain modal features of each modality, including: determining the label probability distribution at each time step based on the embedding features of each modality at each unit length; searching the label probability distribution at each time step by greedy decoding or bundle search, and adjusting the temporal granularity by merging duplicate labels or ignoring blank labels to obtain uniform modal features per unit time step for each modality.

[0088] In some embodiments, a connectionist temporal classification model can be used to align the embedding features of each modality at the temporal granularity for each unit length, thereby obtaining the modal features for each modality.

[0089] In this embodiment, for the unit length of data characteristics of different modalities, the embedding features of each modality at each unit length are accurately obtained; then, a connectionist temporal classification model is used to perform temporal alignment on the embedding features of each modality at each unit length, so that the modal features of various modalities have temporal step alignment.

[0090] In some embodiments, such as Figure 4 As shown, the training steps for the adjacency matrix include:

[0091] Step 402: Based on the initial adjacency matrix, adjust the semantic samples of the modal features at each time step to obtain the reconstructed values ​​of the semantic samples.

[0092] The initial adjacency matrix is ​​the adjacency matrix before or during training; it can convert semantic samples into reconstructed values. Specifically, the initial adjacency matrix may include a classifier head with trainable causal weights and a basic adjacency matrix; the initial adjacency matrix is ​​obtained by optimizing the parameters of the classifier head with trainable causal weights, and the result of combining the basic adjacency matrix with the semantic sample and the classifier head is the reconstructed value of the semantic sample.

[0093] The reconstructed value of a semantic sample is the estimated causal weight value of the initial adjacency matrix and the semantic sample, used to characterize the causal relationship between the semantic sample and the intent label sample. The reconstructed value of the semantic sample can represent the accuracy of the potential factors of the causal relationship, and the reconstructed value is the causal edge weight determined by the semantic sample and the initial adjacency matrix to be adjusted.

[0094] Semantic samples are feature samples that may have a causal relationship with intent label samples. Semantic samples are divided according to each modality, and each modality feature is connected to the intent label sample respectively, thus obtaining the sample of the causal graph node set.

[0095] Intent label samples are label samples used to classify and identify user intent. Intent label samples can be set according to time steps, with one or more intent label samples existing at each time step. The training process of the initial adjacency matrix is ​​supervised and guided by intent label samples.

[0096] In some embodiments, semantic samples can be data from a preset original sample set for each modality, or they can be obtained based on the original data samples for each modality. Specifically, the original data samples for each modality are obtained; the original data samples are split and feature extracted according to the unit length of each modality to obtain the embedded feature samples for each modality; a connectionist temporal classification model is used to temporally align the embedded feature samples for each modality to obtain the modal feature samples for each modality; the semantics of the modal feature samples for each modality at each time step are detected to obtain the semantic samples of the modal features at each time step.

[0097] In some embodiments, based on the initial adjacency matrix, the semantic samples of the modal features at each time step are adjusted to obtain reconstructed values ​​of the semantic samples. This includes: concatenating the semantic samples of the modal features at each time step with the intent label samples to obtain samples of the causal graph node set; and adjusting the feature values ​​of the samples of the causal graph node set at the corresponding time steps based on the initial adjacency matrix to obtain reconstructed values ​​of the node set samples. The samples of the causal graph node set include features of both semantic samples and intent label samples to facilitate subsequent analysis of the causal relationship detection performance of the initial adjacency matrix.

[0098] In one embodiment, the semantic samples of modal features at each time step are concatenated with the intent label samples to obtain samples of the causal graph node set, including: taking the union of the semantic samples of modal features at each time step and the intent label samples as samples of the causal graph node set.

[0099] In some embodiments, the initial adjacency matrix is ​​a one-dimensional vector comprising multiple adjacency matrix elements, each of which is used to adjust the semantic sample of the modality feature at the corresponding time step to obtain the reconstructed value of the semantic sample.

[0100] Step 404: Determine the first loss value based on the difference between the reconstructed value of the semantic sample and the semantic sample at each time step.

[0101] The first loss value is the feature difference between the semantic sample and its reconstructed value in the causal relationship dimension. The first loss value is determined based on the difference between the semantic sample and its reconstructed value, and it can be the L2 norm. Since the difference between the reconstructed value and the semantic sample is actually determined by the initial adjacency matrix, the first loss value can characterize the potential factors of the causal edge weights between the semantic sample and the reconstructed value, helping to more accurately determine the causal edge weights.

[0102] In some embodiments, determining a first loss value based on the difference between the reconstructed value of the semantic sample and the semantic sample at each time step includes: calculating the difference between the reconstructed value of the semantic sample of each modality and the semantic sample at each time step at the same time step to obtain the difference of the semantic reconstructed sample of each modality at each time step; substituting the difference of the semantic reconstructed sample at each time step into the L2 loss function for calculation to obtain the first loss value.

[0103] Step 406: Determine the second loss value based on the difference between the predicted value of the intent label corresponding to the semantic sample and the intent label sample.

[0104] The intent label prediction value is the result of predicting the intent of semantic samples for each modality based on the latent nonlinear structure function of the intent label. Using the intent label prediction value, the cross-entropy function for intent recognition can be directly formed, thus obtaining the second loss value more accurately.

[0105] The second loss value is the loss value that optimizes the initial adjacency matrix directly from the perspective of intent recognition results; using the second loss value strengthens the learning signal on the causal path between semantic samples and intent label samples.

[0106] Step 408: Based on the first loss value and the second loss value, adjust the initial adjacency matrix to obtain the trained adjacency matrix.

[0107] The trained adjacency matrix is ​​a matrix that can accurately represent the causal edge weights. Optionally, the trained adjacency matrix can be the adjacency matrix in step 304, or it can be an adjacency matrix determined in conjunction with the first loss value, the second loss value, and other factors.

[0108] In some embodiments, adjusting the initial adjacency matrix based on the first loss value and the second loss value to obtain the trained adjacency matrix includes: adjusting the initial adjacency matrix until the first loss value and the second loss value satisfy a stopping training condition, and then using the initial adjacency matrix at the time of the training condition as the trained adjacency matrix. The stopping training condition can be that the sum of the first loss value and the second loss value converges, or that the number of calculations of the first loss value and the second loss value reaches a preset value.

[0109] In this embodiment, a first loss value is determined based on the difference between the reconstructed value of the semantic sample and the semantic sample at each time step. The first loss value reflects the classification accuracy of potential factors and helps to determine the causal edge weights more accurately. Using the first loss value and the second loss value can strengthen the training signal on the causal path between semantic factors and intent label samples, thereby enabling the adjacency matrix to determine the causal edge weights more accurately.

[0110] In some embodiments, such as Figure 5 As shown, based on the first loss value and the second loss value, the initial adjacency matrix is ​​adjusted to obtain the trained adjacency matrix, including steps 502 and 504. Steps 502 and 504 can be implemented separately or simultaneously, wherein:

[0111] Step 502: Perform exponential operation using the product of the elements of the initial adjacency matrix as the exponent to obtain the exponential operation result; perform matrix tracking operation on the exponential operation result to obtain the matrix tracking result; determine the difference between the matrix tracking result and the total number of time steps of the semantic samples; use the difference as a preset value as a constraint, and adjust the initial adjacency matrix based on the first loss value and the second loss value to obtain the trained adjacency matrix.

[0112] Element-wise multiplication involves multiplying the elements at each corresponding position. Element-wise multiplication and matrix multiplication are calculated differently. For example, multiplying elements at the same time step yields the Hadamard product. The result of exponentiation is obtained by multiplying the elements of the initial adjacency matrix and itself using a preset value as the base and the product of the elements as the exponent. The preset value can be the natural logarithm.

[0113] The matrix tracking result represents the number of causal edges of all possible lengths that start from and return to a given time step node, thus forming a loop in the directed acyclic graph at that time step node, representing the angular contribution in the initial adjacency matrix. By using the difference between the matrix tracking result and the total number of time steps of the semantic samples as a constraint, the diagonal contribution of matrix powers is suppressed, thereby preventing self-loops in the causal graph. This forms an exponential tracking penalty as an acyclic constraint, allowing the trained adjacency matrix to be used to transform into a directed acyclic graph.

[0114] In some embodiments, the initial adjacency matrix is ​​adjusted based on the first loss value and the second loss value, constrained by the difference value, to obtain the trained adjacency matrix. This includes: adjusting the initial adjacency matrix when the difference between the matrix tracking result and the total number of time steps of the semantic samples is 1, until the first loss value and the second loss value meet the stopping training condition, and obtaining the trained adjacency matrix based on the initial adjacency matrix when the stopping training condition is met.

[0115] Step 504: Determine the absolute value of the vector transformed from the initial adjacency matrix, adjust the absolute value of the vector through sparse coefficients to obtain the third loss value; based on the first loss value, the second loss value and the third loss value, adjust the initial adjacency matrix to obtain the trained adjacency matrix.

[0116] The absolute value of a vector can be the L1 norm of the initial adjacency matrix, which is the sum of the absolute values ​​of all elements in the initial adjacency matrix. Sparse coefficients indicate that the proportion of zero elements in the matrix is ​​greater than that of non-zero elements. By using sparse coefficients, the sparsity of the vector's absolute value can be increased, thereby strengthening the relationship between key semantic features and intent labels, and ensuring the accuracy of causal edge weight calculation. The third loss value is the result of adjusting the absolute value of the vector using the sparse coefficients; for example, the matrix multiplication of the sparse coefficients and the absolute value of the vector results in the third loss value.

[0117] In some embodiments, adjusting the initial adjacency matrix based on the first loss value, the second loss value, and the third loss value to obtain the trained adjacency matrix includes: adjusting the initial adjacency matrix until the sum of the first loss value, the second loss value, and the third loss value satisfies the stopping training condition, and obtaining the trained adjacency matrix based on the initial adjacency matrix when the stopping training condition is met.

[0118] In this case, steps 502 and 504 can be implemented simultaneously. That is, the initial adjacency matrix is ​​adjusted based on the first loss value, the second loss value and the third loss value, constrained by the difference between the matrix tracking result and the total number of time steps of the semantic samples, to obtain the trained adjacency matrix.

[0119] In this embodiment, using element-wise multiplication to construct the matrix exponent increases the trend of the exponentiation result, ensuring a more stable trend. Furthermore, by using the difference between the matrix tracking result and the total number of time steps of the semantic samples as a constraint, the diagonal contribution of the matrix exponentiation is suppressed, preventing self-looping in the directed acyclic graph (DAG) that serves as a causal graph. Using sparse coefficients to obtain the third loss value allows the initial adjacency matrix to obtain more accurate causal edge weights.

[0120] In some embodiments, the training method for the hyperbolic embedding function includes: projecting key modal feature samples of each modality into each modal feature sample in hyperbolic space using an initial hyperbolic embedding function; calculating geodesic distance based on the common semantics between each modal feature sample in hyperbolic space to obtain a first hyperbolic loss value; acquiring intent label samples and cue symbols for classification, concatenating the intent label samples and cue symbols and converting them into label pin samples in hyperbolic space, calculating geodesic distance based on the semantics between each modal feature sample in hyperbolic space and the label pin to obtain a second hyperbolic loss value; and training the initial hyperbolic embedding function based on the first hyperbolic loss value and the second hyperbolic loss value to obtain the hyperbolic embedding function.

[0121] Key modal feature samples are the key modal features used in the training process. These samples can be obtained by filtering modal feature samples based on causal edge weights, representing modal feature samples with a strong correlation to the intent label. Key modal feature samples can also be obtained from a sample set. Cue symbols are used to enhance the perceived effectiveness of the intent label samples; these symbols can be related to the intent label samples and are used to increase the guidance strength of the label samples.

[0122] The initial hyperbolic embedding function is the hyperbolic embedding function used before or during training. It transforms key modal feature samples from Euclidean space into hyperbolic space. However, since the parameters of the initial hyperbolic embedding function are adjustable, each modal feature sample transformed into hyperbolic space may not yet accurately represent the intent label. Conversely, after training with the first and second hyperbolic loss values, the trained hyperbolic embedding function can more accurately represent the intent label for each modal feature sample transformed into hyperbolic space.

[0123] Geodesic distance is a distance determined along hyperbolic space; it is based on the inverse hyperbolic cosine function. Same semantics means that the semantics of different modal feature samples are the same or similar; for example, semantics with a similarity less than a preset similarity threshold are considered the same semantics.

[0124] The first hyperbolic loss value is the semantic loss value between different modalities, reflecting the degree of semantic alignment between them. The second hyperbolic loss value is the geodesic distance between the semantics of the modal feature samples in hyperbolic space and the semantics between the label pins in each modality. The second hyperbolic loss value reflects the degree of semantic alignment between the feature samples of each modality and the label pins.

[0125] The intent label samples are set according to each time step, and the label pin samples can be obtained by combining the intent label samples of each time step. The label pin samples can reflect the actual situation of intent recognition. Therefore, the second hyperbolic loss value is used to train the hyperbolic embedding function. This is a result-oriented process to adjust the initial hyperbolic embedding function.

[0126] In some embodiments, projecting key modal feature samples of each modality into each modal feature sample in hyperbolic space using an initial hyperbolic embedding function includes: projecting key modal features of each modality from Euclidean space into hyperbolic space using the projection parameters of the initial hyperbolic embedding function to obtain each modal feature in hyperbolic space.

[0127] In some embodiments, concatenating intent label samples with cue symbols and converting them into label pin samples in hyperbolic space includes: concatenating intent label samples with multiple cue symbols to obtain a concatenated sequence; encoding the concatenated sequence using a BERT encoder to obtain an encoded concatenated sequence; and converting the encoded concatenated sequence into hyperbolic space using a function for label encoding to obtain label pin samples in hyperbolic space.

[0128] In some embodiments, training an initial hyperbolic embedding function based on a first hyperbolic loss value and a second hyperbolic loss value to obtain a hyperbolic embedding function includes: adjusting the projection parameters of the initial hyperbolic embedding function until the sum of the losses of the first hyperbolic loss value and the second hyperbolic loss value is minimized, and using the initial hyperbolic embedding function with the minimized sum of losses as the trained hyperbolic embedding function.

[0129] In this embodiment, a first hyperbolic loss value is determined based on the distance between the same semantics of different modalities in hyperbolic space. This allows for analysis from the potential hierarchical structure and global topological structure of the same semantics, which helps to align the semantics of different modalities. A second hyperbolic loss value is determined based on the distance between the semantics of different modalities and the label pins in hyperbolic space. This allows the hyperbolic embedding function to move closer to the label pins of the intended label samples, thereby improving the accuracy of classification. As a result, when using the hyperbolic embedding function for feature extraction, both discriminativeness and structural consistency are improved.

[0130] In some embodiments, the semantics of multiple modal features in hyperbolic space includes each target modal semantic and non-target modal semantic; the non-target modal semantic is the semantic concatenation result of various modalities other than the target modal semantic among the semantics of multiple modal features in hyperbolic space.

[0131] The semantics of multiple modal features in hyperbolic space are fused by cross-attention to obtain fused features, including: cross-attention calculation based on the query vector corresponding to each target modal semantics and the key vector and value vector corresponding to the non-target modal semantics to obtain each feature to be fused; and concatenation of multiple features to be fused to obtain fused features.

[0132] The target modal semantics includes the semantics of a modal feature in hyperbolic space; the semantics of each modal feature can be used as the target modal semantics, and the corresponding non-target modal semantics can be determined for each target modal semantics.

[0133] Non-target modal semantics include the semantic concatenation result of at least two modal features in hyperbolic space; that is, non-target modal semantics is the concatenation result of other modal semantics. When using each target modal semantic and non-target modal semantic, features can be extracted progressively for each target modal semantic to form corresponding cross-attention features.

[0134] In some embodiments, cross-attention calculation is performed based on the query vector corresponding to each target modality semantic and the key vector and value vector corresponding to the non-target modality semantic to obtain each feature to be fused, including: performing dot product and normalization processing on the query vector corresponding to each target modality semantic and the key vector corresponding to the non-target modality semantic to obtain the attention weight corresponding to each target modality semantic; and performing cross-attention extraction on the non-target modality semantic of each target modality semantic according to the attention weight corresponding to each target modality semantic to obtain each feature to be fused corresponding to each target modality semantic.

[0135] In an exemplary embodiment, the semantics of multiple modal features in hyperbolic space include a first modal semantic, a second modal semantic, and a third modal semantic; the first modal semantic, the second modal semantic, and the third modal semantic are each a target modal semantic; correspondingly, cross-attention calculation is performed based on the query vector corresponding to each target modal semantic, and the key vector and value vector corresponding to the non-target modal semantic, to obtain each feature to be fused, including: using the first modal semantic as the query vector, and using the concatenation result of the second modal semantic and the third modal semantic as the key vector and value vector to perform cross-attention calculation to obtain the first feature to be fused; using the second modal semantic as the query vector, and using the concatenation result of the first modal semantic and the third modal semantic as the key vector and value vector to perform cross-attention calculation to obtain the second feature to be fused; using the third modal semantic as the query vector, and using the concatenation result of the first modal semantic and the second modal semantic as the key vector and value vector to perform cross-attention calculation to obtain the third feature to be fused; concatenating the first feature to be fused, the second feature to be fused, and the third feature to be fused to obtain the fused feature.

[0136] Among them, the first modality semantics, the second modality semantics, and the third modality semantics are the semantics of different modal features in hyperbolic space; for example, if multiple modal features include language modality, video modality, and audio modality, then the first modality semantics can be the semantics of language modal features in hyperbolic space, the second modality semantics can be the semantics of video modal features in hyperbolic space, and the third modality semantics can be the semantics of audio modal features in hyperbolic space.

[0137] In this embodiment, each target modal semantic in the hyperbolic space is used as a query vector, and cross-attention calculation is performed on the splicing results of other modal semantics in the hyperbolic space. This can more accurately fuse feature data of multiple modalities to form the fused features corresponding to the hyperbolic space.

[0138] In one exemplary embodiment, multimodal intent recognition (MIR) attempts to infer user intent by jointly analyzing linguistic, visual, and acoustic signals. However, this task remains fundamentally challenging due to the pervasive presence of intent-irrelevant cues in non-linguistic patterns and the inherent dissonance between heterogeneous patterns. Existing methods often indiscriminately fuse patterns and align them in Euclidean space, ignoring the structural heterogeneity between patterns and oversimplifying the causal relationships between different patterns regarding intent expression.

[0139] To address these issues, C-HAP, a novel framework for causal relationship-aware hyperbolic alignment with immediate supervision, is proposed. Specifically, C-HAP first constructs a modality-specific causal graph, identifying intent-related features by separating causal information signals from irrelevant or noisy inputs. Subsequently, C-HAP utilizes a cue-guided hyperbolic alignment module to embed modality representations and intent labels into a shared hyperbolic space, achieving discriminative and structurally consistent cross-modal alignment. Extensive experiments on three benchmark MIR datasets (mintrec, MintRec2.0, and melD-Da) demonstrate the superiority of C-HAP, highlighting the importance of causal inference and geometric alignment in multimodal intent recognition. Steps 302-310 and corresponding embodiments constitute the C-HAP framework as follows: Figure 6 As shown. It includes the following:

[0140] First, a specific embodiment of step 302 will be discussed. Raw data for three modalities are acquired: text, image, and audio. These modalities show significant differences in time scale, semantic content, and expression granularity over three months. To achieve unified cross-modal modeling, the embedding preprocesses the three modalities, focusing on two main aspects: the acquisition of embedding features and the alignment of temporal granularity. The acquisition of embedding features may include the acquisition of text modality, visual modality, and audio modality.

[0141] In text modality, text input is represented as The intention sentence, in which This represents the number of text tokens. To incorporate label semantics into context modeling, a masked label input sequence is constructed in the following format. :

[0142]

[0143] Wherein, [CLS] represents the classification marker at the beginning of the sequence. Indicates the first Each text token, representing a unit of text length, is counted starting from the first text token, with [SEP] indicating the end of the sentence. The constructed sequence is fed into the BERT encoder to obtain a contextualized, high-dimensional text representation:

[0144]

[0145] in It is the hidden size of the BERT model. This represents the embedding features of a text input sequence containing a masked tag token.

[0146] In the visual modality, the raw input data can be represented as a series of frame-level features. ,in Indicates the frame number. Pre-extracted visual frame features are used. By constructing a frame-level embedding matrix through the embedding layer, the embedding features of the video frame are obtained, and their expression is as follows:

[0147]

[0148] In audio modality, the audio modal input is represented as an audio frame sequence. ,in This indicates the number of audio frames. First, the embedding features of the audio frames are obtained using a pre-trained model (e.g., wav2vec2). Then, the embedding layer is applied to obtain the initial high-dimensional features, and the expression is as follows:

[0149]

[0150] In some embodiments, due to the differences in temporal granularity between textual, visual, and acoustic modes, connectionist temporal classification (CTC) is used to process the embedded features to obtain modal features for each modality.

[0151] The CTC module temporarily aligns the embedded features of all three modes, thereby generating a standardized feature sequence:

[0152]

[0153] here, and Represents alignment features for text, visual, and acoustic patterns. To ensure a uniform time length after alignment; The embedding dimension is the same as the output dimension of the respective modal encoder.

[0154] Then align the feature sequences The data is fed into a modality-specific encoder to extract high-level semantics. Formally, this process can be unified as follows:

[0155]

[0156] in Representing modes Aligned input sequence of {text, video, audio}.

[0157] Furthermore, steps 304 and 306 integrate the mode-specific causal DAG with a two-embedding hyperbolic alignment strategy to support robust intent recognition across audio, video, and text. Step 304 allows each modal stream to extract causal features through a trainable DAG adjacency matrix while suppressing spurious signals. Step 306 projects these features into a hyperbolic space, where modality-to-label alignment pulls them toward the label cue, while cross-modal alignment enforces consistency between modalities. Finally, step 308 predicts the semantics of the alignment through a hybrid modality and cross-self-attention model, resulting in fused features; these are then classified in step 310. C-HAP jointly optimizes cross-modal and modality-to-intent-label alignment to enhance multimodal intent recognition, enabling modality-based causal intent modeling to obtain the aforementioned adjacency matrix and hyperbolic embedding function.

[0158] In an exemplary embodiment, specific implementations of steps 402-408 are described. By using the adjacency matrix and corresponding causal edge weights, effective causal factors in each modality can be identified, and non-causal components can be explicitly filtered out, thereby selecting key modal features for each modality to enhance intent understanding. Wherein:

[0159] For any mode Let the embedded feature sequence be ,in Indicates the number of time steps. This represents the embedding dimension. For each modality (e.g., text, image frame, or audio), the input is represented as sequential embedding features:

[0160]

[0161] in Indicates the first The semantics of each time step. In the formula, each time step is considered as a single node in a modality-specific causal graph.

[0162] Then, obtain intent label samples for classification. The sample for constructing the causal graph node set is:

[0163]

[0164] Based on the structural causal model (SCM), the nodes of each factor are... The generation process is defined as follows:

[0165]

[0166] in, for causal parent set, For the potential nonlinear structure function of the neural network parameterization, This is an independent noise term.

[0167] Therefore, by explicitly modeling the causal structure, it is possible to infer which factors truly contribute to the generation of the intention label prediction value, rather than just statistical correlation.

[0168] Specifically, the above formula (8) is implemented by formula (10), and the above formula (8) can also be expressed as the following formula (11).

[0169] To learn the causal graph structure for each modality, a learnable adjacency matrix is ​​introduced:

[0170]

[0171] Each element represents a factor node and an intent label node. The directed edge weights between them. All potential factors and intention labels are connected into a unified vector representation, which is another form of Equation (8):

[0172]

[0173] Reconstruction function for each node Defined as:

[0174]

[0175] in Adjacency matrix of For example, the elements in the first row; This represents element-wise multiplication; A classification head for trainable causal weight values.

[0176] Based on this, the intramodal reconstruction loss is defined as:

[0177]

[0178] in, The first loss value mentioned above, This is the second loss value mentioned above; The result of formula (12) is... For intent label nodes The potential nonlinear structure function. It is the node set of the cause-effect graph, i.e., in formula (11). The first loss value reconstructs each latent factor, while the second loss value is the cross-entropy loss of the intent label prediction, which reinforces the training signal on the causal path from factor to intent label.

[0179] To ensure the trained adjacency matrix causal graph Following the Directed Acyclic Graph (DAG) structure, the aforementioned difference value is required as a constraint. This constraint suppresses the diagonal contribution of matrix powers, thereby preventing self-loops in the causal graph. For example, the exponential tracking penalty is used as an acyclic constraint, and its expression is as follows:

[0180]

[0181] in, Represents the Hadamard (element) product. It is a matrix tracking operator. To ensure a uniform time length after alignment.

[0182] To further improve the sparsity of the graph structure, a... The regularization term is used as the third loss value, thus yielding the target loss function used to train the initial adjacency matrix. The expression for the target loss function is as follows:

[0183]

[0184] in, The sparsity coefficient is . This is the vectorization of a matrix.

[0185] After completing the corresponding training process through formula (15), the trained adjacency matrix can be obtained, so as to execute formula (16).

[0186] Extract the intent label from each modality. Key semantic factors with significant causal influence form a high-quality intent representation subspace as key modal features for each modality, expressed as follows:

[0187]

[0188] in, As a factor Nodes to intent labels Causal edge weights A threshold is set to filter causality strength. In this work, it is set... This eliminates weakly correlated, non-discriminatory factors and retains only core semantic units meaningful for intent modeling. The resulting modality-specific causal intent subspace... It serves as the input basis for subsequent cross-modal aligned embedding.

[0189] In one exemplary embodiment, such as Figure 7 As shown, to mitigate the persistent cross-modal mismatch after causal filtering, a cue-guided hyperbolic alignment module is proposed to perform the training process of the hyperbolic embedding function. This training process uses intent-cued samples to embed mode-specific features into a shared hyperbolic space, thereby achieving supervised cross-modal structural alignment and label consistency. Figure 7 The causal features in the graph include those found in directed acyclic graphs.

[0190] First, hyperbolic projection of causal features is used to obtain each modality feature or each modality feature sample in the hyperbolic space. That is, the causal information subspace of each modality is obtained through an initial hyperbolic embedding function. Projected onto hyperbolic space to preserve its underlying hierarchical structure and global topology. Let... Representing modes The initial hyperbolic embedding function or hyperbolic embedding function. The projection is defined as:

[0191]

[0192] in, For modality Hyperbolic feature sequences are used for semantic distance calculation and contrast optimization in hyperbolic space. In the case of the initial hyperbolic embedding function, the hyperbolic feature sequence consists of feature samples for each modality in the hyperbolic space, used in the subsequent model training process; In the case of representing a hyperbolic embedding function, the hyperbolic feature sequence is the feature of each modality in the hyperbolic space, which can be directly output to the cross-attention network.

[0193] Secondly, the intent label samples and cue labels used for cue enhancement are embedded into a hyperbolic space. That is, to enhance the modal awareness of label semantics, a learnable cue-based label encoding mechanism is introduced. Each intent label sample... All of these are connected to a series of learnable cue symbols to form input:

[0194]

[0195] The sequence is fed into a shared BERT encoder to obtain a cue-guided tag representation:

[0196]

[0197] Then it will be indicated Projected into hyperbolic space in the following manner:

[0198]

[0199] in As a label needle in hyperbolic space to guide cross-modal semantic alignment.

[0200] Furthermore, semantic alignment in the hyperbolic space is achieved through the first and second hyperbolic loss values. To achieve structural alignment between modalities and between modalities and intent label samples, a hyperbolic contrastive loss consisting of two components is introduced:

[0201] To enhance structural consistency, a hyperbolic contrast loss consisting of two parts is introduced.

[0202] The component used to obtain the first hyperbolic loss value is a hyperbolic alignment of modality to label, aiming to approximate the same semantics across different modalities in hyperbolic space; its expression is as follows:

[0203]

[0204] in denoted as the geodesic distance in hyperbolic space.

[0205] The component used to obtain the second hyperbolic loss value is cross-modal hyperbolic alignment, minimizing the distance between each modal-specific embedding and the corresponding tag needle:

[0206]

[0207] The overall comparison target is defined as:

[0208]

[0209] This formula explicitly reduces the semantic differences between multimodal causal embeddings and enhances discriminative consistency.

[0210] Furthermore, causal feature fusion is achieved through cross-attention. That is, after semantic alignment, modality-aware fusion is performed through cross-attention. Taking the text modality as the target modality semantics as an example, after the hyperbolic space transformation result of the hyperbolic embedding function enables semantic alignment of each modality in the hyperbolic space, the semantics of each feature in the hyperbolic space transformed by the hyperbolic embedding function are further fused. As a query, the connection features of the other two modalities are used as keys and values:

[0211]

[0212] A total of three cross-modal attention outputs are generated: Ftext, Fvideo, and They are then concatenated to form the final fused representation, which is then used by a classifier for prediction.

[0213]

[0214] The classification loss is calculated using standard cross-entropy:

[0215]

[0216] The overall training objectives consist of three parts:

[0217]

[0218] in and It is a weighting factor that balances alignment and causal modeling objectives. This fusion module completes the entire process from intramodal causal reasoning to cross-modal alignment and collaborative fusion.

[0219] In one embodiment, the superiority of C-HAP can be demonstrated through data. This embodiment evaluates the effectiveness and robustness of the proposed method on three representative multimodal intent recognition benchmarks: MIntRec, MELD-DA, and MIntRec2.0. MIntRec is a large dataset built for human-computer interaction scenarios, containing text, audio, and video modes. It is labeled with two coarse-grained (emotional expression and goal orientation) and 20 fine-grained intent categories, including 2224 samples, of which 1334 are used for training, 445 for validation, and 445 for testing. MELD-DA, derived from the MELD emotional dialogue corpus, focuses on intent recognition in multi-turn, multimodal dialogue settings. It contains 12 intent labels, including synchronized features from text, audio, and facial expressions, serving as a benchmark for evaluating context modeling and cross-modal alignment. MIntRec2.0 is a recently released large-scale benchmark for intent recognition in multi-party, multi-turn dialogues. It consists of 1,245 dialogues, 30 intent classes, and 15,040 utterances, including out-of-distribution samples, reflecting the diversity and unpredictability of user intents in open-domain scenarios.

[0220] This embodiment uses standard evaluation metrics, including overall accuracy (ACC), weighted F1 score (wF1), weighted precision (wP), and recall (R). These metrics comprehensively reflect model performance under class imbalance, fine-grained semantic discrimination, and sensitivity to low-frequency classes. All results are averaged over three independent runs to ensure stability and statistical reliability.

[0221] In terms of implementation details, this embodiment follows MVCL-DAF, using the same pre-trained feature extractor for all three modes to ensure fair comparison. This embodiment uses the AdamW optimizer to train all models. The batch size is set to 16 during training and to 8 during validation and testing. Hyperparameters are set to 1.0 and 1.0 respectively. All experiments were performed on a 32GB NVIDIA Tesla V100-SXM2 GPU.

[0222] Results on the MIntRec dataset. This embodiment evaluates the proposed C-HAP framework on the widely used MIntRec benchmark and compares it with several recent baselines. C-HAP achieves best performance on all four evaluation metrics: accuracy Weighted F1 Weighted Precision Recall rate Accuracy improved compared to a better baseline. The recall rate has increased. Compared to the previous main methods, the gains in weighted F1 and weighted precision are consistent. Table 1 shows a performance comparison of the MIntRec, MELD-DA, and MIntRec2.0 datasets. Metrics include accuracy (Acc), weighted F1 (wF1), weighted precision (wP), and recall (R).

[0223] Table 1

[0224]

[0225] This embodiment further evaluates the method of this embodiment on MELD-DA, a benchmark centered on emotion-aware intent recognition in multi-turn dialogues. As shown in Table 1, C-HAP again achieves high performance across all four metrics: accuracy Weighted F1 Weighted precision Recall rate Compared to strong baselines such as MVCL-DAF and SDIF-DA, this embodiment improves accuracy by at least [percentage missing]. The recall rate has increased. That concludes the discussion. It's worth noting that it also outperformed all previous methods in weighted F1, demonstrating the model's robustness in capturing intent-related cues under emotionally complex and multi-speaker interactions.

[0226] This embodiment further evaluates C-HAP on MIntRec2.0, a large-scale benchmark for real-world intent recognition. As shown in Table 1, the model in this embodiment achieves best performance on all four metrics: accuracy The weighted F1 score is 57.44%, and the weighted Precision score is... Recall rate Compared to the baseline MVCL-DAF, C-HAP improved accuracy. Weighted F1 improved The recall rate increased by nearly This demonstrates its robustness under more diverse and challenging conditions. The impact of the Modality-Specific Causal Intent Model (MCIM) on MintRec is shown in Table 2.

[0227] Table 2

[0228]

[0229] In one embodiment, a series of ablation studies were performed on the MIntRec dataset to evaluate the effectiveness of each key component in the proposed C-HAP framework. Modality-Specific Causal Intent Modeling: To evaluate the impact of modality-specific causal graphs, this embodiment removed them from the model and observed a significant performance degradation: a decrease in accuracy. The weight F1 decreases Recall rate decreased As shown in Table 2, these results highlight the importance of causal modeling in filtering noise and isolating critical signals.

[0230] In one embodiment, as shown in Table 3, the effect of replacing hyperbolic space with its Euclidean counterpart was investigated. This substitution resulted in a continuous decline in performance: a decrease in accuracy. F1 weighted decline Precision weighted decrease These results demonstrate the effectiveness of hyperbolic geometry in modeling hierarchical relationships across modalities. The exponential capacity of hyperbolic space allows for better separation of intent classes in heterogeneous modal representations, thereby enhancing overall discriminative power.

[0231] Table 3

[0232]

[0233] In one embodiment, two hyperbolic alignment strategies were designed: Modality-to-Label (MLHA) and Cross-Modality (CMHA). MLHA uses label cues to pin modality-specific causal features to class centers, while CMHA enforces cross-modality structural consistency in hyperbolic space.

[0234] As shown in Table 4, deleting any branch will lead to a performance decrease, with accuracy dropping to [a certain level]. The following shows that applying these two methods yielded better results than other methods across all metrics—achieving [the desired outcome]. accuracy and The weighted accuracy demonstrates the complementarity and effectiveness of the two calibration mechanisms.

[0235] Table 4

[0236]

[0237] In one embodiment, to evaluate the effectiveness of the prompt guidance mechanism, the prompt labels used for training were removed, and only the label sample embeddings were retained as semantic anchors. As shown in Table 5, this leads to a performance degradation: decreased accuracy. The weight F1 decreases These results demonstrate that cues, as soft semantic instructions, provide additional guidance during training to better constrain intent representations and enhance class classification structures in the embedding space.

[0238] Table 5

[0239]

[0240] In one embodiment, the text was compared. Visual Harmony The single-modal performance of the input is shown in Table 6. As shown in Table 6, the method in this embodiment consistently outperforms MVCL-DAF across all modalities, improving performance by +1.58 on T, +0.88 on V, and +2.03 on A. The highest accuracy is achieved in text modality recognition. This indicates that it carries the largest amount of intent recognition clue information. Acoustic modalities follow... Despite its noisy nature, it benefits from causal filtering. The visual modality performs the worst, at [value missing]. This indicates that it contains relatively little information related to intent.

[0241] Table 6

[0242]

[0243] In this embodiment, to evaluate the quality of the learned intent representation, t-SNE is used to visualize the top 10 most accurate classes. For example... Figure 8 As shown, Figure 8 Figure (a) in the diagram is related to TCL-MAP. Figure 8 Figure (b) is related to MVCL-DAF. Figure 8 Figure (c) in the diagram relates to C-HAP in this embodiment. Compared to TCL-MAP and MVCL-DAF, C-HAP exhibits more compact clusters, clearer separation, and significant overlap and dispersion. This demonstrates the superiority of the causal modeling and hyperbolic alignment in this embodiment in generating structured and discriminative intent embeddings.

[0244] Therefore, this embodiment proposes C-HAP, a causal awareness framework for multimodal intent recognition that collectively addresses modality-specific noise and cross-modal misalignment. By utilizing a modality-specific causal graph, C-HAP filters out spurious and redundant signals while preserving intent-related representations. Furthermore, a cue-guided hyperbolic alignment module enhances cross-modal coherence and intent-level discriminability through structured embedding. Extensive experiments on three common benchmarks (mintrec, MIntRec2.0, and melD-Da) demonstrate the superior performance of this embodiment. The results of this embodiment highlight the importance of causal inference and geometric alignment in modeling complex multimodal interactions, paving the way for more robust and interpretable intent understanding systems.

[0245] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0246] Based on the same inventive concept, this application also provides a multimodal intent recognition device for implementing the multimodal intent recognition method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the multimodal intent recognition device provided below can be found in the limitations of the multimodal intent recognition method described above, and will not be repeated here.

[0247] In one exemplary embodiment, such as Figure 9 As shown, a multimodal intent recognition device is provided, comprising:

[0248] The feature extraction module 902 is used to obtain the modal features of each modality;

[0249] The causal filtering module 904 is used to obtain the semantics of the modal features at each time step; determine the causal edge weights of the semantics and intent labels at each time step in the directed acyclic graph based on the adjacency matrix and the semantics of each time step; and filter the modal features of each modality based on the causal edge weights to obtain the key modal features of each modality.

[0250] Hyperbolic embedding module 906 is used to project the key modal features of each modality into the hyperbolic space through a hyperbolic embedding function to obtain each modal feature in the hyperbolic space;

[0251] Attention module 908 is used to perform cross-attention fusion on the semantics of multiple modal features in the hyperbolic space to obtain fused features;

[0252] The classification module 910 is used to classify the fused features to obtain intent labels.

[0253] In one embodiment, the apparatus further includes a training module for performing a training step on the adjacency matrix, the training module being used to:

[0254] Based on the initial adjacency matrix, the semantic samples of the modal features at each time step are adjusted to obtain the reconstructed values ​​of the semantic samples;

[0255] A first loss value is determined based on the difference between the reconstructed value of the semantic sample and the semantic sample at each time step;

[0256] A second loss value is determined based on the difference between the predicted value of the intent label corresponding to the semantic sample and the intent label sample.

[0257] Based on the first loss value and the second loss value, the initial adjacency matrix is ​​adjusted to obtain the trained adjacency matrix.

[0258] In one embodiment, the training module is used to:

[0259] An exponential operation is performed using the element-wise product of the initial adjacency matrix as the exponent, yielding the exponential operation result; a matrix tracking operation is performed on the exponential operation result, yielding the matrix tracking result; the difference between the matrix tracking result and the total number of time steps of the semantic samples is determined; the initial adjacency matrix is ​​adjusted based on the first loss value and the second loss value, constrained by the difference value, to obtain the trained adjacency matrix; and / or,

[0260] The absolute value of the vector transformed from the initial adjacency matrix is ​​determined, and the absolute value of the vector is adjusted by sparse coefficients to obtain a third loss value; based on the first loss value, the second loss value and the third loss value, the initial adjacency matrix is ​​adjusted to obtain the trained adjacency matrix.

[0261] In one embodiment, the apparatus further includes a training module for performing a training step of the hyperbolic embedding function, the training module being used to:

[0262] The key modal feature samples of each modality are projected into each modal feature sample in the hyperbolic space through the initial hyperbolic embedding function; the geodesic distance is calculated based on the common semantics between each modal feature sample in the hyperbolic space to obtain the first hyperbolic loss value;

[0263] Obtain intent label samples and cue symbols for classification, concatenate the intent label samples and the cue symbols and convert them into label pin samples in the hyperbolic space, calculate the geodesic distance based on the semantics between each modal feature sample in the hyperbolic space and the label pin, and obtain the second hyperbolic loss value;

[0264] Based on the first hyperbolic loss value and the second hyperbolic loss value, the initial hyperbolic embedding function is trained to obtain the hyperbolic embedding function.

[0265] In one embodiment, the semantics of the various modal features in the hyperbolic space includes each target modal semantic and non-target modal semantic; the non-target modal semantic is the semantic concatenation result of various modalities other than the target modal semantic among the semantics of the various modal features in the hyperbolic space;

[0266] The attention module 908 is used for:

[0267] Cross-attention calculation is performed based on the query vector corresponding to each target modality semantic, and the key vector and value vector corresponding to the non-target modality semantic, to obtain each feature to be fused;

[0268] Multiple features to be fused are spliced ​​together to obtain fused features.

[0269] In one embodiment, the feature extraction module 902 is used to:

[0270] Obtain the raw data for each modality;

[0271] The original data is split and features are extracted according to the unit length of each modality to obtain the embedding features of each modality at each unit length;

[0272] Temporal granularity alignment is performed based on the embedding features of each modality at each unit length to obtain the modal features of each modality.

[0273] Each module in the aforementioned multimodal intent recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0274] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal intent recognition method.

[0275] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0276] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0277] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0278] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0279] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0280] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0281] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0282] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-modal intent recognition method, characterized by, The method comprises: acquiring modal features of each modality; acquiring semantics of the modal features at each time step; determining causal edge weights of the semantics at each time step and intent labels in a directed acyclic graph according to an adjacency matrix and the semantics at each time step; and screening the modal features of each modality according to the causal edge weights to obtain key modal features of each modality; projecting the key modal features of each modality into hyperbolic space by a hyperbolic embedding function to obtain modal features in the hyperbolic space; cross-attention fusion is performed on the semantics of the modal features in the hyperbolic space to obtain fused features; classifying the fused features to obtain intent labels; wherein the training step of the adjacency matrix comprises: adjusting the semantics samples of the modal features at each time step according to an initial adjacency matrix to obtain reconstruction values of the semantics samples; determining a first loss value based on the difference value between the reconstruction values of the semantics samples and the semantics samples at each time step; determining a second loss value based on the difference value between the intent label prediction value corresponding to the semantics sample and the intent label sample; and adjusting the initial adjacency matrix based on the first loss value and the second loss value to obtain the trained adjacency matrix.

2. The method of claim 1, wherein, The adjustment of the initial adjacency matrix based on the first loss value and the second loss value to obtain the trained adjacency matrix comprises: performing exponential operation with the element product of the initial adjacency matrix as the index to obtain an exponential operation result; performing matrix tracking operation on the exponential operation result to obtain a matrix tracking result; determining the difference value between the matrix tracking result and the total number of time steps of the semantics sample; adjusting the initial adjacency matrix based on the first loss value and the second loss value with the difference value as a preset value as a constraint to obtain the trained adjacency matrix; and / or determining the vector absolute value converted from the initial adjacency matrix, adjusting the vector absolute value by a sparse coefficient to obtain a third loss value; and adjusting the initial adjacency matrix based on the first loss value, the second loss value and the third loss value to obtain the trained adjacency matrix.

3. The method of claim 1, wherein, The training method of the hyperbolic embedding function comprises: projecting the key modal feature samples of each modality as the modal feature samples in the hyperbolic space by an initial hyperbolic embedding function; calculating the geodesic distance between the modal feature samples in the hyperbolic space based on the same semantics to obtain a first hyperbolic loss value; acquiring intent label samples and prompt symbols for classification, connecting the intent label samples with the prompt symbols and converting them into label needle samples in the hyperbolic space, calculating the geodesic distance between each modal feature sample in the hyperbolic space and the label needle based on the semantics to obtain a second hyperbolic loss value; training the initial hyperbolic embedding function based on the first hyperbolic loss value and the second hyperbolic loss value to obtain the hyperbolic embedding function.

4. The method of claim 1, wherein, The semantics of the plurality of modal features in the hyperbolic space include each target modal semantic and a non-target modal semantic; the non-target modal semantic is a semantic splicing result of various modal semantics in the plurality of modal features in the hyperbolic space, except for the target modal semantic; The cross-attention fusion of the semantics of the plurality of modal features in the hyperbolic space obtains fused features, including: Based on the query vector corresponding to each target modal semantic and the key vector and the value vector corresponding to the non-target modal semantic, cross-attention calculation is performed to obtain each to-be-fused feature; Splicing is performed on the plurality of to-be-fused features to obtain fused features.

5. The method of claim 1, wherein, The modal feature of each modal is obtained, including: Original data of each modal is obtained; According to the unit length of each modal, the original data is split and feature extraction is performed to obtain embedded features of each modal at each unit length; Based on the embedded features of each modal at each unit length, time granularity alignment is performed to obtain the modal feature of each modal.

6. The method of claim 1, wherein, The semantics sample of the modal feature at each time step is adjusted according to the initial adjacency matrix to obtain a reconstructed value of the semantics sample, including: The semantics sample and the intent label sample of the modal feature at each time step are connected to obtain a sample of a causal graph node set; According to the initial adjacency matrix, the feature value of the sample of the causal graph node set at the corresponding time step is adjusted to obtain a reconstructed value of the node set sample; the reconstructed value of the node set sample is the reconstructed value of the semantics sample.

7. A multi-modal intent recognition apparatus, characterized by, The device includes: A feature extraction module is configured to obtain a modal feature of each modal; A causal screening module is configured to obtain semantics of the modal feature at each time step; determine a causal edge weight of the semantics and an intent label in a directed acyclic graph according to an adjacency matrix and the semantics at each time step; and screen the modal feature of each modal according to the causal edge weight to obtain a key modal feature of each modal; A hyperbolic embedding module is configured to project the key modal feature of each modal into a hyperbolic space by a hyperbolic embedding function to obtain each modal feature in the hyperbolic space; An attention module is configured to perform cross-attention fusion on the semantics of the plurality of modal features in the hyperbolic space to obtain fused features; A classification module is configured to classify the fused features to obtain an intent label; A training module is configured to adjust a semantics sample of the modal feature at each time step according to an initial adjacency matrix to obtain a reconstructed value of the semantics sample; determine a first loss value based on a difference value between the reconstructed value of the semantics sample and the semantics sample at each time step; determine a second loss value based on a difference value between an intent label prediction value corresponding to the semantics sample and an intent label sample; and adjust the initial adjacency matrix based on the first loss value and the second loss value to obtain a trained adjacency matrix.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, which when executed by a processor, implements the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program, which when executed by a processor, implements the steps of the method of any one of claims 1 to 6.

Images