A Multimodal Feature Fusion Method and System Based on Gated Attention

Abstract

This invention discloses a multimodal feature fusion method and system based on gated attention, belonging to the field of artificial intelligence and multimodal information processing technology. The method includes: S1, acquiring multimodal vectors to be fused, and performing a linear transformation on the multimodal vectors to obtain a multimodal transformation matrix; the multimodal vectors include: context vectors, personality feature vectors, and emotional state vectors; S2, calculating the gating weights of the multimodal vectors based on the multimodal vectors using an attention mechanism; S3, performing weighted fusion on the multimodal transformation matrix based on the gating weights to obtain a fused vector. The gating weights of this invention are learned from data through an attention mechanism, achieving dynamic fusion with content awareness; simultaneously, an emotion-priority dynamic adjustment mechanism ensures that the emotion weight is automatically increased during strong negative emotions.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and multimodal information processing technology, specifically relating to a multimodal feature fusion method and system based on gated attention. Background Technology

[0002] In artificial intelligence applications, personalized responses that comprehensively consider multiple user characteristics (such as contextual information, personality traits, and emotional states) are key to improving user experience. Existing multimodal feature fusion technologies mainly include the following categories: (1) Early Fusion: The feature vectors of different modalities are directly concatenated and then input into the model. This method is simple and direct, but it has the following drawbacks: it cannot handle the differences in the importance of features of different modalities; the feature dimension expansion leads to an increase in computational complexity; and it ignores the interaction between modalities.

[0003] (2) Late Fusion: Each modality is processed independently and then fused at the decision level. The drawbacks of this method are: it cannot capture the synergistic effect between modalities at the feature level; the fusion strategy (such as voting, weighted average) is too simple, and the fusion effect is limited by the preset parameters; it is difficult to achieve dynamic weight adjustment.

[0004] (3) Linear weighted fusion method: The features of each modality are weighted and summed using fixed or preset weights: , where Z represents the fused composite vector; , , These represent the feature vectors of the three modes, respectively. , , These represent pre-defined fixed weights (scalar values) used to control the contribution ratio of each modality to the fusion result. The formula is calculated by multiplying each modality vector by its corresponding weight and then summing them one by one to obtain a result vector with the same dimension as the input. The drawbacks of this method are: fixed weights cannot adaptively adjust according to the input content, potentially obscuring important information; it cannot capture non-linear interactions between features; and it lacks a response mechanism for special situations (such as emotional crises).

[0005] (4) Attention mechanism fusion method: Although existing attention-based multimodal fusion (such as Cross-Attention in the Transformer architecture) has implemented dynamic weights, it lacks a priority mechanism for specific modalities (such as emotions), and cannot respond in time at critical moments; it cannot quickly adjust the fusion strategy in emergency situations; and it has high computational complexity, making it unsuitable for real-time applications and difficult to meet the needs of real-time interaction. Summary of the Invention

[0006] The problem to be solved by this invention is: how to achieve adaptive deep fusion of multimodal features (context, personality, emotion), and automatically adjust the fusion strategy when an emotional crisis is detected, prioritizing the response to emotional factors.

[0007] To achieve the above objectives, the present invention provides the following solution: a multimodal feature fusion method based on gated attention, comprising the following steps: S1. Obtain the multimodal vectors to be fused, and perform a linear transformation on the multimodal vectors to obtain the multimodal transformation matrix; the multimodal vectors include: context vectors, personality feature vectors, and emotional state vectors; S2. Based on the multimodal vector, calculate the gating weights of the multimodal vector using an attention mechanism; S3. The multimodal transformation matrix is ​​weighted and fused based on the gating weights to obtain a fused vector.

[0008] More preferably, in S1, the emotional state vector includes: VAD dimension and emotional intensity. ∈[-1,1] and stability ; The VAD dimension includes: valence. ∈[-1,1], Arousal ∈[-1,1] and sense of control ∈[-1,1].

[0009] More preferably, in S2, the method for calculating the gating weight includes: ; in, ; In the formula, Represents contextual weights; Indicates the weight of personality traits; Indicates the weight of emotional state; This represents the normalized exponential function; This represents a joint feature vector formed by concatenating the context vector, personality vector, and emotion vector along the dimensional direction. Indicates the scaling factor; Represents the three-dimensional real number space; Represents a learnable query matrix; C represents vector concatenation; P represents the context vector; E represents the personality trait vector; and E represents the emotional state vector. Represents the dimension of the context vector; Represents the dimension of the personality trait vector; This represents the dimension of the emotional state vector.

[0010] More preferably, in S2, the calculation of the gating weights also includes an emotion-priority dynamic adjustment mechanism; The emotion-prioritized dynamic regulation mechanism includes: ; In the formula, Indicates the emotional state weight adjustment value; L represents the number of levels; Indicates the first Priority coefficient; Indicates an indicator function; Indicates the first Grade valence threshold; Indicates the first Level intensity threshold.

[0011] More preferably, after adjusting the emotional state weights according to the emotion-priority dynamic adjustment mechanism, the situation weights and personality trait weights are adjusted to obtain situation weight adjustment values ​​and personality trait weight adjustment values. The context weight adjustment values ​​include: ; The personality trait weight adjustment values ​​include: .

[0012] More preferably, the multimodal transformation matrix and the query matrix are jointly trained with downstream tasks in an end-to-end manner: ; In the formula, Represents the total loss function; This indicates losses in downstream tasks; Represents the regularization coefficient; This represents the regularization term.

[0013] The present invention also provides a multimodal feature fusion system based on gated attention, comprising: The transformation module is used to obtain the multimodal vectors to be fused, and to perform a linear transformation on the multimodal vectors to obtain a multimodal transformation matrix; the multimodal vectors include: context vectors, personality feature vectors, and emotional state vectors; The gating weight module is used to calculate the gating weights of the multimodal vectors based on the multimodal vectors through an attention mechanism; The fusion module is used to perform weighted fusion of the multimodal transformation matrix based on the gate weights to obtain a fusion vector.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: The gating weights of this invention learn from data through an attention mechanism, achieving dynamic fusion of content awareness; at the same time, the emotion-first dynamic adjustment mechanism ensures that the emotion weights can be automatically increased when there are strong negative emotions; requiring only a small number of matrix operations, the computational efficiency is high, making it suitable for real-time applications; the gating weights intuitively reflect the contribution of each modality vector, are highly interpretable, and can be applied to scenarios that require comprehensive consideration of context, personality, and emotional factors, such as intelligent customer service, psychological counseling, educational guidance, human-computer interaction, and affective computing. Attached Figure Description

[0015] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the overall framework of the multimodal feature fusion method based on gated attention according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the calculation process of the gating attention mechanism in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the triggering logic of the emotion-priority dynamic adjustment mechanism in an embodiment of the present invention. Figure 4 This is a schematic diagram of a multi-level emotion priority mechanism according to an embodiment of the present invention; Figure 5 This is an example diagram of the visualization of gating weights in an embodiment of the present invention; Figure 6 This is a comparison chart of gating weight distribution under different scenarios in embodiments of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] Example 1:

[0020] like Figure 1 As shown, this embodiment provides a multimodal feature fusion method based on gated attention, including the following steps: S1. Obtain the multimodal vectors to be fused, and perform a linear transformation on the multimodal vectors to obtain the multimodal transformation matrix; the multimodal vectors include: context vectors, personality feature vectors, and emotional state vectors.

[0021] Among them, the context vector Used to characterize the features of the current interaction scenario The context vector dimension includes: scenario type encoding (such as customer service consultation, learning tutoring, and psychological support), semantic embedding vector of triggering events, time features (such as time period, cycle, etc.), and compressed representation of historical interactions.

[0022] Personality trait vector Used to characterize a user's stable personality traits The dimensions of the personality trait vector include: the Big Five personality model (openness, conscientiousness, extraversion, agreeableness, and neuroticism), other personality scales (such as numerical representations of MBTI, 16PF, etc.), and behavioral preference characteristics (such as communication style and decision-making patterns).

[0023] Emotional state vector Used to represent the user's immediate emotional state. The dimensions of the emotional state vector include: VAD dimension and emotional intensity (such as happiness, sadness, anger, fear, and anxiety). ∈[0,1] and stability The VAD dimension includes: valence. ∈[-1,1], Arousal ∈[0,1] and sense of control ∈[0,1].

[0024] Perform a linear transformation on each modal vector to uniformly map it to... Dimensional space: , , ; In the formula, The context transformation matrix is ​​a OK, The learnable parameter matrix of the column; Representation matrix Matrix-vector multiplication with vector C: The context vector C of dimension is mapped as dimensional transformation vector ; Represents the personality transformation matrix, used to transform... The personality vector P of dimension is mapped as dimensional transformation vector ; This represents the emotion transformation matrix, used to... The emotion vector of dimension is mapped as dimensional transformation vector ; This represents the dimension of the fused vector, with preferred values ​​of 64, 128, or 256.

[0025] Since the original dimensions of the three types of input vectors are usually different, direct weighted fusion calculation is not possible. Instead, three independent linear transformation matrices are used to map the feature vectors of different dimensions to the same matrix. The 3D space makes subsequent gated weighted fusion mathematically feasible. The parameters of these transformation matrices are automatically learned during training, enabling the extraction of the most suitable feature representation for fusion for each modality.

[0026] S2. Based on multimodal vectors, the gating weights of multimodal vectors are calculated through an attention mechanism.

[0027] After concatenating the original multimodal vectors, the gating weights for each modality are obtained through an attention mechanism: ; ; In the formula, Represents contextual weights; Indicates the weight of personality traits; Indicates the weight of emotional state; This represents the normalization exponential function, used to transform any real vector into a probability distribution for an input vector. The softmax calculation method is as follows: The softmax function guarantees that all output values ​​are positive and strictly between 0 and 1, and that the sum of all output values ​​is 1. + + =1; X represents the joint feature vector formed by concatenating the context vector, personality vector, and emotion vector along the dimensional direction. This represents the scaling factor, used to stabilize the gradient; Represents a three-dimensional real number space, consisting of an ordered array of three real numbers; This represents a learnable query matrix with 3 rows and ( ) columns. The dimension of the joint feature vector corresponding to ). The result is a three-dimensional vector, which is the original attention score of each of the three modalities; This indicates vector concatenation.

[0028] For applications requiring finer control, gating weights can be extended to vector form. In this case, each dimension has independent gating weights, enabling more precise feature selection. ; In the formula, Indicates 3 lines The columns are real-valued matrices, meaning each modality is no longer just a scalar weight, but possesses... Each independent weight value is used to gate each dimension of the transformed feature vector; at this time... For the expanded query matrix, softmax is used to normalize along each modality dimension (in groups of three). Compared to scalar gating, vector gating allows the model to focus on different modalities across different feature dimensions. For example, it can focus more on emotional information in some dimensions and more on contextual information in others, enabling more refined feature selection and information integration.

[0029] Gating weights are learned through a query matrix. By automatically learning from data, this invention achieves "content-aware" adaptive weight allocation. Unlike fixed-weight methods, this invention can dynamically adjust the contribution of each modality based on the input content.

[0030] S3. The multimodal transformation matrix is ​​weighted and fused based on the gate weights to obtain the fused vector.

[0031] like Figure 2 As shown, the method for weighted fusion of the transformed features using gated weights to obtain the fused vector is as follows: ; In the formula, Z represents the fusion vector; This represents element-wise multiplication; This represents the weighted adjustment value for emotional state.

[0032] This embodiment also provides an emotion-priority dynamic adjustment mechanism. When a user is detected to be in a strong negative emotional state, the weight of the emotional modality in the fusion vector is automatically increased to ensure that the user's emotional needs are responded to first. The logic is as follows: The calculation method for the emotional state weight adjustment value includes: ; In the formula, The value represents the priority coefficient of emotion, with a preferred range of [0.1, 0.3], and is set to 0.2 in this embodiment; a negative value of emotion valence indicates negative emotion. The valence threshold is represented, with a preferred range of [-0.5, -0.2], and in this embodiment, it is set to -0.3; The intensity threshold is represented, with a preferred range of [0.5, 0.8], and in this embodiment, it is set to 0.6; This indicates an indicator function.

[0033] like Figure 3The diagram shown is a logic diagram for triggering the emotion-prioritized dynamic regulation mechanism; when When the emotional valence is negative, it indicates that the user is expressing negative emotions such as sadness, anger, and anxiety; when When both conditions are met, it indicates a high level of emotional intensity, with the user experiencing strong emotions rather than slight fluctuations. When both conditions are met, it indicates a strong negative emotional state, with the user in an emotional crisis requiring priority attention. The dynamic emotion-priority adjustment mechanism is triggered only when the user is simultaneously in a state of both negative and intense emotions.

[0034] like Figure 4 As shown, for applications requiring more refined emotional responses, a multi-level emotional dynamic priority regulation mechanism is adopted: ; In the formula, L represents the number of levels; Indicates the first Priority coefficient; Indicates the first Grade valence threshold; Indicates the first Level intensity threshold.

[0035] In this embodiment, the response level is divided into 4 levels. The valence threshold, intensity threshold, priority coefficient, and emotional state of level 1 are described as: -0.2, 0.5, 0.10, mild negative emotion; the valence threshold, intensity threshold, priority coefficient, and emotional state of level 2 are described as: -0.4, 0.6, 0.15, moderate negative emotion; the valence threshold, intensity threshold, priority coefficient, and emotional state of level 3 are described as: -0.6, 0.7, 0.25, severe negative emotion; and the valence threshold, intensity threshold, priority coefficient, and emotional state of level 4 are described as: -0.8, 0.8, 0.35, extreme negative emotion / crisis.

[0036] After adjusting the emotional state weights according to the emotion-priority dynamic adjustment mechanism, in order to keep the sum of the gating weights at 1, the situation weights and the personality trait weights are adjusted (re-normalized) to obtain the situation weight adjustment value and the personality trait weight adjustment value.

[0037] ; In the formula, normalize represents normalization; This represents the final sentiment weight after enhancement and normalization adjustments.

[0038] Alternatively, a soft adjustment method can be used to proportionally reduce the weights of other modalities. Specifically, the context weight adjustment values ​​include: ; The personality trait weight adjustment values ​​include: .

[0039] in, Can be replaced with The value is determined based on the calculation method of the aforementioned emotional adjustment state weight adjustment value.

[0040] By prioritizing emotional dynamic adjustment mechanism, the problem of "treating everyone the same" in existing integration methods is solved. In times of emotional crisis, the decision-making focus is automatically shifted to emotional processing, prioritizing emotional support and comfort, avoiding the execution of tasks that may increase negative emotions, and ensuring that users receive timely responses when they need them most.

[0041] In this embodiment, the multimodal transformation matrix and the query matrix are jointly trained with downstream tasks in an end-to-end manner: ; In the formula, Represents the total loss function; This indicates losses in downstream tasks; Represents the regularization coefficient; This represents a regularization term used to prevent overfitting.

[0042] Among them, the downstream task loss can be selected according to different loss functions based on the task type.

[0043] For classification tasks, such as emotion category recognition, cross-entropy loss can be used. ; In the formula, Indicates the total number of categories; The one-hot encoding representing the actual label; The model predicts the first Class probability.

[0044] For regression tasks, such as predicting emotion intensity, mean squared error loss can be used. ; In the formula, N represents the sample size; Indicates the first The true value of each sample; Indicates the first The predicted value for each sample.

[0045] Meanwhile, to increase interpretability, this embodiment imposes sparsity constraints on the gating weights: ; In the formula, Indicates the gating regularization term; This represents the normalized entropy, used to measure the uncertainty of the gating weight distribution.

[0046] At this point, the total loss function is: ; In the formula, This represents the gating regularization coefficient.

[0047] Example 2:

[0048] Combination Figure 4 , Figure 5 The diagrams and comparison diagrams of the gating weights under different scenarios of the present invention illustrate the effectiveness of the present invention.

[0049] This embodiment uses an intelligent customer service scenario as an example to illustrate the method proposed in this invention. Scenario description: A user contacts customer service due to an order issue and is in a relatively agitated state. Input information is as follows: Context vector C: Scenario type: after-sales complaint; Event embedding: semantic vector of [order delay, multiple reminders, unresolved]; Time feature: weekday evening.

[0050] Personality Vector P: Extraversion: 0.65 (high, tending to express directly); Neuroticism: 0.58 (moderately high, emotionally sensitive); Conscientiousness: 0.72 (high, demanding of service).

[0051] Emotion Vector E: Valence =-0.55 (negative); wakefulness =0.78 (highly agitated); Anger =0.72, Frustration =0.65; Emotional intensity =0.75.

[0052] Processing procedure: 1. Gating weight calculation: Initial gating: =[0.32, 0.28, 0.40].

[0053] 2. Prioritize Emotional Factors: =-0.55< =-0.3; =0.75> =0.6.

[0054] Triggering the emotion-prioritizing mechanism: =0.40+0.20=0.60.

[0055] 3. Weight normalization: After adjustment: =[0.24, 0.21, 0.55] (Emotion weight significantly increased).

[0056] 4. Fusion output: The fusion vector Z emphasizes emotional features; downstream systems should prioritize emotional reassurance.

[0057] Example 3:

[0058] This embodiment uses a psychological counseling scenario as an example to illustrate the method proposed in this invention. Scenario description: A user confides about work stress and low mood, but not to the point of crisis. Input information is as follows: Context vector C: Scenario type: stress venting; Event embedding: semantic vector of [busy at work, working overtime, tired].

[0059] Personality vector P: Extraversion: 0.35 (low, introverted); Neuroticism: 0.62 (high); Openness: 0.55 (moderate).

[0060] Emotion Vector E: Valence =-0.25 (mildly negative); Arousal level =0.35 (low, fatigue); Sadness =0.45, Fatigue =0.70; Emotional Intensity =0.52.

[0061] Processing procedure: 1. Gating weight calculation: Initial gating: =[0.30, 0.38, 0.32].

[0062] 2. Prioritize Emotional Factors: =-0.25> =-0.3 (valence not reached the threshold); emotion priority mechanism is not triggered.

[0063] 3. Fusion Output: The fusion vector Z maintains balanced fusion; the weight of personality traits is slightly higher, and the communication preferences of introverted users should be considered.

[0064] Example 4:

[0065] This embodiment uses an educational tutoring scenario as an example to illustrate the method proposed in this invention. Scenario description: A student is feeling down due to failing an exam and seeks help with their studies. Input information is as follows: Context vector C: Scenario type: learning tutoring; Event embedding: semantic vector of [unsatisfactory exam results, mathematics].

[0066] Personality Vector P: Conscientiousness: 0.75 (high, demanding of oneself); Neuroticism: 0.58 (moderately high); Openness: 0.62 (willing to try new methods).

[0067] Emotion Vector E: Valence =-0.48 (moderately negative); Sadness=0.52, Depression=0.68; Emotional Intensity =0.65.

[0068] Processing procedure: 1. Gating weight calculation: Initial gating: =[0.35, 0.30, 0.35].

[0069] 2. Prioritize Emotional Factors: =-0.48< =-0.3; =0.65> =0.6.

[0070] Triggering the emotion-prioritizing mechanism: =0.35+0.20=0.55.

[0071] 3. Fusion Output: The fusion vector Z emphasizes emotional factors; downstream systems should prioritize emotional support before providing learning suggestions.

[0072] like Figure 6 As shown, a grouped bar chart is used to compare the scenarios of Example 2, Example 3, and Example 4 before and after triggering. The comparison before and after the triggering intuitively shows the increase in emotion weight and the corresponding reduction in the weight of other modalities.

[0073] Example 5:

[0074] This embodiment provides a multimodal feature fusion system based on gated attention to implement the above-mentioned fusion method. It includes: a transformation module for acquiring the multimodal vectors to be fused and performing a linear transformation on the multimodal vectors to obtain a multimodal transformation matrix; the multimodal vectors include: a context vector, a personality feature vector, and an emotional state vector; a gate weight module for calculating the gate weights of the multimodal vectors based on the multimodal vectors using an attention mechanism; and a fusion module for performing weighted fusion on the multimodal transformation matrix based on the gate weights to obtain a fused vector.

[0075] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multimodal feature fusion method based on gated attention, characterized in that, Includes the following steps: S1. Obtain the multimodal vectors to be fused, and perform a linear transformation on the multimodal vectors to obtain the multimodal transformation matrix; the multimodal vectors include: context vectors, personality feature vectors, and emotional state vectors; S2. Based on the multimodal vector, calculate the gating weights of the multimodal vector using an attention mechanism; S3. The multimodal transformation matrix is ​​weighted and fused based on the gating weights to obtain a fused vector.

2. The multimodal feature fusion method based on gated attention according to claim 1, characterized in that, In S1, the emotional state vector includes: VAD dimension and emotional intensity. ∈[-1,1] and stability ; The VAD dimension includes: valence. ∈[-1,1], Arousal ∈[-1,1] and sense of control ∈[-1,1].

3. The multimodal feature fusion method based on gated attention according to claim 1, characterized in that, In S2, the method for calculating the gating weight includes: ； in, ； In the formula, Represents contextual weights; Indicates the weight of personality traits; Indicates the weight of emotional state; represents the normalized exponential function; X represents the joint feature vector formed by concatenating the context vector, personality vector, and emotion vector along the dimensional direction. Indicates the scaling factor; Represents the three-dimensional real number space; Represents a learnable query matrix; C represents vector concatenation; P represents the context vector; E represents the personality trait vector; and E represents the emotional state vector. Represents the dimension of the context vector; Represents the dimension of the personality trait vector; This represents the dimension of the emotional state vector.

4. The multimodal feature fusion method based on gated attention according to claim 3, characterized in that, In S2, the calculation of the gating weights also includes an emotion-prioritized dynamic adjustment mechanism; The emotion-prioritized dynamic regulation mechanism includes: ； In the formula, Indicates the emotional state weight adjustment value; L represents the number of levels; Indicates the first Priority coefficient; Indicates an indicator function; Indicates the first Grade valence threshold; Indicates the first Level intensity threshold.

5. The multimodal feature fusion method based on gated attention according to claim 4, characterized in that, After adjusting the weights of emotional states according to the aforementioned emotion-priority dynamic adjustment mechanism, the weights of the context and the weights of the personality traits are adjusted to obtain the adjusted values ​​of the context weights and the adjusted values ​​of the personality traits weights. The context weight adjustment values ​​include: ； The personality trait weight adjustment values ​​include: 。 6. The multimodal feature fusion method based on gated attention according to claim 3, characterized in that, The multimodal transformation matrix and the query matrix are jointly trained with downstream tasks in an end-to-end manner: ； In the formula, Represents the total loss function; This indicates losses in downstream tasks; Represents the regularization coefficient; This represents the regularization term.

7. A multimodal feature fusion system based on gated attention, used to implement the fusion method according to any one of claims 1-6, characterized in that, include: The transformation module is used to obtain the multimodal vector to be fused, and to perform a linear transformation on the multimodal vector to obtain the multimodal transformation matrix; The multimodal vectors include: context vectors, personality feature vectors, and emotional state vectors; The gating weight module is used to calculate the gating weights of the multimodal vectors based on the multimodal vectors through an attention mechanism; The fusion module is used to perform weighted fusion of the multimodal transformation matrix based on the gate weights to obtain a fusion vector.

Images