Adaptive multi-modal data fusion prediction method and system based on dynamic graph convolution

Abstract

A dynamic graph convolution-based adaptive multi-modal data fusion prediction method and system, the method comprising: extracting a text sentiment feature vector for representing sentiment semantics and a personality feature vector for representing individualized traits from the obtained text input, extracting an audio feature vector and a visual feature vector from the obtained audio and video input; after mapping the text sentiment feature vector and the personality feature vector to a shared space, performing semantic alignment by contrast learning constraint and personality weighted regression constraint to obtain an aligned text sentiment feature vector; taking the aligned text sentiment feature vector as a query vector, performing multi-head attention interaction on the audio feature vector and the visual feature vector respectively to obtain an audio enhanced feature vector and a visual enhanced feature vector; constructing a multi-node graph structure and realizing cross-modal feature fusion through dynamic graph convolution to obtain a fusion feature matrix; and generating a prediction result using the fusion feature matrix.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and multimodal data fusion technology, and in particular to an adaptive multimodal data fusion prediction method and system based on dynamic graph convolution. Background Technology

[0002] Multimodal Sentiment Analysis (MSA) is a cutting-edge field in artificial intelligence and multimodal information processing, aiming to accurately model individual emotional states by integrating information from multiple sensory modalities such as text, audio, and vision. With the widespread adoption of social media, distance education, smart healthcare, and human-computer interaction systems, extracting deep state features from massive amounts of unstructured multimodal data has become a key technological support. However, existing multimodal data analysis techniques still face many bottlenecks in practical applications, limiting their generalization ability and representation accuracy in complex scenarios.

[0003] From a technical implementation perspective, the main drawback of existing deep learning network architectures for multimodal data analysis (hereinafter referred to as "models") lies in the imbalance of contributions from multimodal data during feature fusion. Since textual modalities typically contain the most explicit semantic features, most models exhibit an over-reliance on textual features, weakening the complementary role of audio and visual modalities. When subtle deviations occur between textual semantics and audio-visual expression (such as changes in intonation or inconsistencies in visual expression), existing models often struggle to capture deep conflicts or synergistic relationships between modalities, leading to biases in the fused feature representation. Furthermore, existing technologies exhibit poor robustness in cross-data domain applications, especially for language data such as Chinese, which possesses complex acoustic parameters and rich modal particles, often failing to effectively align state expression features across different data distributions.

[0004] Specifically, existing solutions have the following shortcomings in feature extraction and fusion mechanisms: First, in the feature extraction stage, most solutions only focus on the representation of emotion itself (features extracted from the raw data to describe the emotional state), neglecting the inherent statistical correlation between the speaker's personality traits and emotional expression. Personality feature vectors, as the underlying background of emotional expression, have a significant impact on the strength and change patterns of emotional indications; the lack of a personality-emotional state expression alignment mechanism leads to insufficient feature extraction depth. Second, in the cross-modal fusion stage, the simple splicing or conventional attention mechanisms used in existing solutions often struggle to handle the loss problem in the transmission of long sequence features and lack effective loss constraints to balance the contributions of each single modality. Simultaneously, existing Graph Convolutional Networks (GCNs) applications are mostly statically constructed, unable to dynamically adjust the association weights between nodes based on the features of the data input, resulting in insufficient flexibility when processing heterogeneous data. Finally, existing models often only focus on the final fusion prediction, lacking explicit supervision of single-modal branches, making the model prone to getting trapped in local optima during training and failing to fully exploit discriminative information in the audio and visual spaces.

[0005] Therefore, how to construct a multimodal data fusion prediction method that can adaptively mine the distribution patterns and high-order interaction relationships of different modal features in the representation space based on the inherent statistical correlation of multimodal data, so as to improve the modeling accuracy and cross-scene generalization ability of individual emotional state representation, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this application aims to provide an adaptive multimodal data fusion prediction method based on dynamic graph convolution, which achieves objective information balance in the feature fusion process by dynamically adjusting the coupling strength between modalities.

[0007] To achieve the above objectives, this application provides an adaptive multimodal data fusion and prediction method based on dynamic graph convolution, comprising the following steps: From the acquired text input, extract text sentiment feature vectors for representing emotional semantics and personality feature vectors for representing individual characteristics. From the acquired audio and video input, extract audio feature vectors and visual feature vectors. After mapping the text sentiment feature vector and the personality feature vector to a shared space, semantic alignment is performed through contrastive learning constraints and personality weighted regression constraints to obtain the aligned text sentiment feature vector. Using the aligned text sentiment feature vector as the query vector, multi-head attention interaction is performed on the audio feature vector and the visual feature vector respectively to obtain the audio-enhanced feature vector and the visual-enhanced feature vector; A multi-node graph structure is constructed based on the feature vectors of each modality, and cross-modal feature fusion is achieved through dynamic graph convolution, thereby obtaining the fused feature matrix; The fused feature matrix is ​​sequentially passed through the ReLU activation function and a fully connected network, and prediction results are generated according to the task type.

[0008] Furthermore, the steps of extracting text sentiment feature vectors for representing emotional semantics and personality feature vectors for representing personalized traits from the acquired text input, and extracting audio feature vectors and visual feature vectors from the acquired audio and video input, further include: The acquired text input is encoded in parallel by a pre-trained sentiment encoder and a personality encoder to obtain text sentiment feature vectors and personality feature vectors; A bidirectional recurrent neural network is used to perform sequence modeling on the audio and visual features in the original video stream, and the output of the bidirectional recurrent neural network is obtained as the audio feature vector and the visual feature vector.

[0009] Furthermore, the step of mapping the text sentiment feature vector and the personality feature vector to the shared space, and then performing semantic alignment through contrastive learning constraints and personality-weighted regression constraints to obtain the aligned text sentiment feature vector further includes: using a contrastive learning loss function to bring the text sentiment feature vector and the personality feature vector of the same text sample closer together in the shared space, while pushing away the feature vectors of different text samples; and using a personality-weighted regression loss function, constraining the sentiment intensity regression of the text sentiment feature vector with the similarity between the text sentiment feature vector and the personality feature vector as the weight.

[0010] Further, in the step of using the aligned text sentiment feature vector as the query vector and performing multi-head attention interaction on the audio feature vector and the visual feature vector respectively to obtain the audio-enhanced feature vector and the visual-enhanced feature vector, the expression for the visual-enhanced feature vector is: The expression for the audio enhancement feature vector is: ;in Enhance visual feature vectors; The aligned text sentiment feature vector; For visual feature vectors; This represents the visual multi-head attention mechanism; Enhance the feature vector for audio; For audio feature vectors; This represents the audio multi-head attention mechanism.

[0011] Furthermore, after the step of using the aligned text sentiment feature vector as the query vector and performing multi-head attention interaction on the audio feature vector and the visual feature vector respectively to obtain the audio-enhanced feature vector and the visual-enhanced feature vector, the method further includes the step of applying independent single-modal prediction loss constraints to the aligned text sentiment feature vector, the audio-enhanced feature vector, and the visual-enhanced feature vector respectively: Using real labels as the supervision target, independent cross-entropy loss is applied to the aligned text sentiment feature vector, the audio enhancement feature vector, and the visual enhancement feature vector respectively, to ensure that each modality feature vector has independent state representation ability before entering the fusion step.

[0012] Furthermore, the step of constructing a multi-node graph structure based on the feature vectors of each modality, achieving cross-modal feature fusion through dynamic graph convolution, and thus obtaining a fused feature matrix, further includes: constructing a multi-node structure graph based on the relevant feature vectors of text, audio, and vision, encoding the correlation strength between nodes with a dynamically updated adjacency matrix, and using a Chebyshev network as a graph convolutional layer to perform high-order convolutional inference on the node features, and accumulating the feature matrices output by each layer of the graph convolutional layer to obtain a fused feature matrix; The multi-node graph structure includes two text nodes, two visual nodes, two audio nodes, and two interactive nodes.

[0013] Furthermore, it also includes: training an end-to-end deep learning network by jointly optimizing the total loss function, wherein the deep learning network is used in the method; the expression for the total loss function is: in, This is the total loss function; The multimodal prediction loss function; Let represent the single-modal prediction loss function, where t represents the text modality, a represents the audio modality, and v represents the visual modality; This is the composite alignment loss function; and Hyperparameters are used to balance the contributions of each loss term.

[0014] To achieve the above objectives, the present invention also provides an adaptive multimodal data fusion prediction system based on dynamic graph convolution, employing the adaptive multimodal data fusion prediction method based on dynamic graph convolution as described above. The system includes: The multi-path feature extraction module is used to extract text sentiment feature vectors for representing emotional semantics and personality feature vectors for representing individual characteristics from the acquired text input, and to extract audio feature vectors and visual feature vectors from the acquired audio and video input. The personality-emotion alignment module is used to map the text sentiment feature vector and the personality feature vector to a shared space, and then perform semantic alignment through contrastive learning constraints and personality weighted regression constraints to obtain the aligned text sentiment feature vector. The cross-modal attention interaction module is used to perform multi-head attention interaction on the audio feature vector and the visual feature vector respectively, using the aligned text sentiment feature vector as the query vector, to obtain the audio enhancement feature vector and the visual enhancement feature vector; The dynamic graph convolution fusion module is used to construct a multi-node graph structure based on the feature vectors of each modality, and to achieve cross-modal feature fusion through dynamic graph convolution, thereby obtaining a fused feature matrix; The joint optimization module is used to sequentially pass the fused feature matrix through the ReLU activation function and the fully connected network, and generate prediction results according to the task type.

[0015] To achieve the above objectives, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the computer program stored in the memory to implement the adaptive multimodal data fusion prediction method based on dynamic graph convolution as described above.

[0016] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program, which is loaded and executed by a processor to implement the adaptive multimodal data fusion prediction method based on dynamic graph convolution as described above.

[0017] The adaptive multimodal data fusion prediction method based on dynamic graph convolution provided in this application achieves the balance of objective information in the feature fusion process by dynamically adjusting the coupling strength between modalities, thereby realizing dynamic balance and higher-order inference between modalities, and improving the modeling accuracy and cross-scene generalization ability of individual emotional state representation.

[0018] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing this application. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the present application and form part of the specification. Together with the embodiments of the present application, they serve to explain the present application but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of an adaptive multimodal data fusion and prediction method based on dynamic graph convolution according to an embodiment of this application; Figure 2This is a schematic diagram of the structure of an adaptive multimodal data fusion prediction system based on dynamic graph convolution according to an embodiment of the invention; Figure 3 This is a schematic diagram of an electronic device structure according to an embodiment of the present invention. Detailed Implementation

[0020] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application.

[0021] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0022] The term "comprising" and its variations as used in this application are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments".

[0023] It should be noted that the terms "first" and "second" may be used in this application only to distinguish different devices, components or parts, and are not used to define the order of functions performed by these devices, components or parts or their interdependence.

[0024] It should be noted that the terms "one" and "more" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless explicitly stated otherwise in the context, they should be understood as "one or more". "More" should be understood as two or more.

[0025] It should be noted that, in this invention, the term "personal emotional state" refers to the set of objective information characterized by quantifiable feature parameters extracted from multimodal data. The method and system described in this invention achieve adaptive extraction and fusion of state indication features from multimodal signals through data processing. Its core processing object is the multimodal data itself and its statistical distribution in the feature space. The generated final predicted value y is used for an objective quantitative description of an individual's emotional state.

[0026] For ease of description, the meanings of some mathematical symbols in this invention may vary in different contexts. The specific meaning of each symbol is subject to its definition within the corresponding technical logic paragraph, and the meaning of the same symbol remains consistent within the same logic paragraph. Definitions of other terms will be provided below.

[0027] Example 1 In embodiments of the present invention, an adaptive multimodal data fusion prediction method based on dynamic graph convolution is provided. This method is implemented based on an end-to-end deep learning network, which includes a multi-path feature extraction module, a personality-sentiment alignment module, a cross-modal attention interaction module, a dynamic graph convolution fusion module, a prediction output module, and a joint optimization module. The method specifically includes the following steps: From the acquired text input, extract text sentiment feature vectors for representing emotional semantics and personality feature vectors for representing individual characteristics. From the acquired audio and video input, extract audio feature vectors and visual feature vectors. After mapping the text sentiment feature vector and the personality feature vector to a shared space, semantic alignment is performed through contrastive learning constraints and personality weighted regression constraints to obtain the aligned text sentiment feature vector. Using the aligned text sentiment feature vector as the query vector, multi-head attention interaction is performed on the audio feature vector and the visual feature vector respectively to obtain the audio-enhanced feature vector and the visual-enhanced feature vector; A multi-node graph structure is constructed based on the feature vectors of each modality, and cross-modal feature fusion is achieved through dynamic graph convolution, thereby obtaining the fused feature matrix; The fused feature matrix is ​​sequentially passed through the ReLU activation function and a fully connected network, and prediction results are generated according to the task type.

[0028] Figure 1 The flowchart below shows the adaptive multimodal data fusion prediction method based on dynamic graph convolution according to an embodiment of this application. Figure 1 The implementation method of this application will be described in further detail.

[0029] First, in step 101: multi-path feature extraction is performed on the multimodal input.

[0030] In embodiments of this invention, multimodal input includes text input and audio / video input (audio and visual). Features are extracted from the text input using a text path and from the audio / video input using an audio / video path. The text path comprises two independent pre-trained encoders (an emotion encoder and a personality encoder), which, through parallel dual-path encoding, uncover deep emotional semantics and personalized background traits within the text. The audio / video path utilizes a bidirectional recurrent neural network (Bi-RNN) to perform sequence modeling on the audio and visual features in the original video stream, respectively, to extract temporally dependent features (such as the visual feature vector and audio feature vector output after LSTM encoding, as described below).

[0031] In this embodiment, the given text input is First, CLS representations are obtained from the emotion encoder and the personality encoder, denoted as follows: and .in and for After extraction by independent sentiment encoders and personality encoders, initial sentiment feature vectors and initial personality feature vectors corresponding to the [CLS] marker bits are obtained. These two vectors serve as the original input features of the personality-sentiment alignment module, carrying a decoupled representation of emotional semantics and personalized traits in the text. Their ultimate mission is to be transformed into a shared feature space through linear projection, allowing subsequent contrastive learning and regression constraints to achieve cross-dimensional semantic alignment. The linear projection layer maps these two vectors to... Dimensional Shared Space: in, As the first weight parameter, This is a text sentiment feature vector. This is a vector of individual characteristics.

[0032] Step 102: Perform semantic alignment on the text sentiment feature vector and personality feature vector.

[0033] In embodiments of the present invention, in order to align text sentiment feature vectors and personality feature vectors at the semantic level, the present invention introduces a contrastive learning-based loss function. and personality-weighted regression loss function Composite alignment loss function Contrastive learning loss function Imposing dual constraints: In the D-dimensional shared space, sentiment and personality representations originating from the same text are brought closer together, while representations originating from different texts are pushed further apart, thereby enhancing the model's ability to perceive cross-domain semantic relationships. A personality-weighted loss function is introduced. This is to use individual feature vectors to guide sentiment intensity regression. Specifically, it involves contrastive learning of the loss function. The expression is as follows: In the formula, The cosine similarity function; For temperature parameters; and For positive sample pairs, they represent the text sentiment feature vector and the personality feature vector from the same i-th text sample, respectively; and (j≠i) represents a negative sample pair, which represents the text sentiment feature vector from the i-th text sample and the personality feature vector from the j-th text sample, respectively; N is the number of samples.

[0034] Personalized weighted regression loss function The expression is as follows: In the formula, This is the second weighting parameter; The true label of the i-th sample (its value format depends on the downstream task settings) is... middle, As The supervised objective is used to constrain the text sentiment feature vector. Retain sufficient information for emotional judgment.

[0035] Final composite alignment loss function As a personality-emotional alignment goal, its expression is as follows: By comparing the joint constraints of the learning loss function and the personality-weighted regression loss function, the personality feature vector is embedded as a semantic prior into the projection space of the text sentiment feature vector, thereby enhancing the discriminative power of the text sentiment feature vector.

[0036] In step 103: Multi-head attention is used to extract audiovisual sentiment features by using the aligned text sentiment feature vector, and single-modal supervision constraints are applied.

[0037] In embodiments of the present invention, the aligned text sentiment feature vector is used as the query vector, and the features highly correlated with emotion in the audiovisual modality, namely visual features and audio features, are extracted through a multi-head attention mechanism.

[0038] In this embodiment, visual feature vectors are obtained using LSTM (Long Short-Term Memory network). and audio feature vectors (The superscript (o) is only used to indicate that it is an output feature of the LSTM network, to distinguish it from the original audio and video features of the input.) The aligned text sentiment feature vector As a Query This represents the text sentiment feature vector after semantic alignment or output by the personality-sentiment alignment module, to distinguish it from the original, unaligned text after projection in step 101. Enhanced features are computed through cross-modal attention: Among them, visual enhancement feature vector The aligned text sentiment feature vector and visual feature vectors Obtained through interaction; Representing the visual multi-head attention mechanism; audio-enhanced feature vectors For text sentiment feature vectors and audio feature vectors Obtained through interaction; This represents the audio multi-head attention mechanism.

[0039] Furthermore, to ensure that each modality possesses independent state representation capabilities before fusion, this invention introduces a single-modal prediction loss function. .for (where a represents the audio modality and v represents the visual modality), its predicted value is calculated as follows: in, This represents the prediction result for mode m; The prediction weight parameters represent mode m; This represents the enhanced feature vector obtained by modality m through the previous round of cross-modal attention interaction.

[0040] For text modality t, the predicted value is calculated as follows: in, This represents the result of direct prediction of text modality t. The prediction weight parameters represent the text modality t. This is the aligned text sentiment feature vector.

[0041] The single-mode prediction loss function is in, , This represents the prediction result for mode m; The cross-entropy loss function; This represents the true label of a single sample.

[0042] This step involves aligning the text sentiment feature vectors. Audio enhancement feature vector With visually enhanced feature vectors Apply independent single-modal classification prediction loss functions to each. , and With real labels To ensure effective supervision, each unimodal feature vector is forced to possess independent emotional state discrimination capabilities before entering dynamic graph convolutional fusion. This constraint, acting as an auxiliary regularization term, effectively suppresses the model's over-reliance on the text modality, ensuring that emotionally relevant information in both audio and visual modalities is fully extracted and preserved.

[0043] In step 104: Construct a multi-node graph structure based on the feature vectors of each modality, and realize cross-modal feature fusion and logical reasoning through dynamic graph convolution.

[0044] In this step, a graph structure containing 8 heterogeneous nodes is constructed. Adaptively updated adjacency matrices and Chebyshev polynomials are used to achieve high-order feature fusion and logical reasoning between nodes. The 8 heterogeneous nodes include 2 text nodes, 2 visual nodes, 2 audio nodes, and 2 interactive nodes. The 2 text nodes are the extracted initial feature vectors. and aligned text sentiment feature vectors The two visual nodes represent the feature vectors before and after inputting the original visual features into the LSTM, respectively; the two audio nodes represent the feature vectors before and after inputting the original audio features into the LSTM, respectively; and the two interaction nodes represent the audio enhancement feature vectors. and visually enhanced feature vectors Organize the 8 heterogeneous nodes into a directed connected graph. V represents the set of nodes. This represents the set of edges connecting these nodes. Given a dynamic adjacency matrix (representing a set of global information), the element in the i-th row and j-th column of A... The importance of the connection between the i-th node and the j-th node was quantified. This step specifically includes: (1) Dynamic adjacency matrix construction: Unlike static graphs, this invention uses a Gaussian kernel function to dynamically calculate the association strength based on the Euclidean distance between nodes. : (9) in, and There are two fixed parameters: t is the scale scaling factor, which is used to control the basic amplitude and magnitude of the correlation strength; The standard deviation is used to control the sensitivity of the Euclidean distance between nodes to the decay of association strength. The larger the value, the more connections between distant nodes are preserved; The smaller the value, the stronger the association can be maintained only between very close nodes. Indicates the first The node and the first The Euclidean distance between nodes. This dynamic adjacency matrix A is adaptively updated with the model parameters during training, enabling it to flexibly learn the global positional relationships between modalities.

[0045] (2) Normalized Laplacian Operator: To achieve self-transmission of multimodal node features (the eigenvectors carried by each node in the graph structure) between graph nodes and suppress noise, this invention introduces a Laplacian matrix L=DA (where D is the degree matrix and A is the aforementioned constructed dynamic adjacency matrix). To satisfy the Chebyshev polynomial requirement for the domain of operator eigenvalues ​​([-1,1]), the Laplacian matrix L is normalized and scaled to obtain the scaled Laplacian operator. : in, Laplace matrix The largest eigenvalue, It is the identity matrix. This scaled operator This provides the foundation for subsequent spatial convolution operations.

[0046] (3) Chebyshev higher-order convolutional inference: The graph convolutional layer uses the ChebNet (Chebshev network, a classic spectral domain graph neural network model) module, utilizing Chebyshev polynomials. An approximate spectral domain graph convolution operator is used to achieve efficient graph convolution inference while avoiding high-complexity matrix eigenvalue decomposition. The graph convolution of the layer outputs the feature vector. The calculation formula is expressed as: in: Indicates the first The output feature matrix of the layer; For the output features of the previous layer (when When =1, That is, the matrix composed of the initial 8 multimodal node eigenvectors. ); K is the highest order of the Chebyshev polynomial, representing the receptive field size (i.e., aggregation) of the graph convolution. (K-order neighborhood information); The k-th order Chebyshev matrix polynomial operator is recursively defined as: It is the first Layered networks in the first layer A learnable weight matrix of order 1. It is a bias term.

[0047] By forward-cascading multiple graph convolutional layers (i.e., the output features of the previous layer are used as the input of the next layer), the network can capture global topological relationships and deep neighborhood interaction information between multimodal nodes from low to high order.

[0048] Finally, the outputs of each layer are accumulated and fused, and then passed through a ReLU activation function and a fully connected network to generate the final multimodal prediction value. and its corresponding multimodal prediction loss function The outputs of each layer mentioned above refer to the feature matrices output by each layer from the first to the last in the entire graph neural network after undergoing Chebyshev graphical convolution. For example, suppose the graph convolutional network contains a total of Layer, the feature matrix output by each layer is , … Then, by accumulating these features, the fused feature matrix is ​​obtained. The expression is as follows: in, For layer index, Indicates the first The node feature matrix output by layer graph convolution. The total number of nodes. For the first The dimension of the layer feature vector.

[0049] In embodiments of the present invention, the final predicted value and multimodal prediction loss function The complete calculation process is as follows: The accumulated fusion feature matrix The flattened feature vector is passed sequentially through a ReLU activation function and a multi-layer fully connected network (MLP), and finally the prediction result is output according to the downstream task type. For sentiment classification tasks, the Softmax function is used to map the output to a probability distribution: For the emotion regression task, a linear activation function is used. Directly output continuous values: Taking classification tasks as an example, the model's multimodal prediction loss function is optimized using the sentiment classification cross-entropy loss: in, The total number of emotion categories, One-hot encoding for real labels in the first... The value of dimension, The model predicts that the sample belongs to the first... The probability of each category. This loss function measures the difference between the predicted probability distribution and the true label distribution, and the optimization objective is to minimize... .

[0050] Overall objective function: By jointly optimizing the following total loss function To collaboratively train the aforementioned end-to-end deep learning network: in, The above-mentioned multimodal prediction loss functions guide the dynamic graph convolutional fusion module to output accurate multimodal sentiment predictions. The above single-mode prediction loss function is... The feature vectors of the text modality t, audio modality a, and visual modality v are constrained to have independent emotional expression capabilities before fusion, so as to suppress the model's over-reliance on the text modality; The aforementioned composite alignment loss function drives semantic alignment between the personality feature vector and the text sentiment feature vector in the shared projection space. The three are weighted and summed to form a unified optimization objective. and To balance the contributions of each loss term to the hyperparameters, the parameters of all modules in the network are updated synchronously during this joint optimization process, ultimately achieving a balance between the model's single-modal discriminative power, personality-sentiment alignment quality, and multimodal fusion accuracy.

[0051] The adaptive multimodal data fusion and prediction method based on dynamic graph convolution provided by this invention has the following advantages: First, by introducing a personality-emotion alignment mechanism, it is possible to identify the influence of the speaker's personalized traits on emotional expression, making the text emotion feature vector more discriminative and effectively overcoming the problem of insufficient representation caused by relying on a single emotion dimension for feature extraction in traditional methods.

[0052] Second, a dynamic graph convolutional fusion module is adopted, which combines prior topological structure with data-driven association learning. Compared with static fusion strategies, the dynamic adjacency matrix can adjust the intermodal coupling strength in real time according to the feature distribution of the input data. Its high-order spectral filtering capability based on Chebyshev polynomials enables the model to effectively capture complex nonlinear high-order interaction relationships between modes, significantly improving the inference depth of fused features.

[0053] Third, by applying independent single-modal prediction loss constraints to text, audio, and visual modalities respectively, the model is forced to fully explore discriminative information in non-text modalities, effectively alleviating the "text-dominated" bias problem common in multimodal models and achieving a balanced distribution of contributions from each modality.

[0054] Fourth, this method exhibits excellent cross-data domain generalization ability. When tested on Chinese datasets containing complex acoustic parameters and rich modal particles, the performance improvement of this method is more significant than that of traditional models due to its explicit mining of audio and video features and adaptive adjustment of dynamic graph structure, effectively adapting to the differences in emotional expression patterns under different language systems.

[0055] Example 2 In an embodiment of the present invention, an adaptive multimodal data fusion prediction system based on dynamic graph convolution is also provided, which adopts the steps of the adaptive multimodal data fusion prediction method based on dynamic graph convolution described in Embodiment 1. Figure 2 This is a schematic diagram of the structure of an adaptive multimodal data fusion prediction system based on dynamic graph convolution according to an embodiment of the invention, such as... Figure 2 As shown, this system includes: The multi-path feature extraction module 201 is used to extract text sentiment feature vectors for representing emotional semantics and personality feature vectors for representing personalized traits from the acquired text input, and to extract audio feature vectors and visual feature vectors from the acquired audio and video input.

[0056] The personality-emotion alignment module 202 is used to map the text emotion feature vector and personality feature vector output by the multi-path feature extraction module 201 to the shared space, and then perform semantic alignment through contrastive learning constraints and personality weighted regression constraints to obtain the aligned text emotion feature vector and pass it to the cross-modal attention interaction module 203.

[0057] The cross-modal attention interaction module 203 is used to perform multi-head attention interaction on the audio feature vector and visual feature vector extracted by the multi-path feature extraction module 201, respectively, using the aligned text sentiment feature vector as the query vector, to obtain the audio enhancement feature vector and the visual enhancement feature vector and pass them to the dynamic graph convolution fusion module 204. The dynamic graph convolution fusion module 204 is used to construct a multi-node graph structure based on the feature vectors of each modality, and to achieve cross-modal feature fusion through dynamic graph convolution, thereby obtaining a fused feature matrix.

[0058] The prediction output module 205 is used to sequentially pass the fused feature matrix obtained by the dynamic graph convolution fusion module 204 through the ReLU activation function and the fully connected network, and generate prediction results according to the task type.

[0059] In embodiments of the present invention, the system further includes a joint optimization module (not shown in the figure), used to train an end-to-end deep learning network through multiple loss functions to achieve a balance between single-modal discriminative power and multi-modal synergistic effect.

[0060] Example 3 In embodiments of the present invention, an electronic device is also provided. Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, such as... Figure 3 As shown, the electronic device of the present invention includes a processor 301 and a memory 302, wherein, The memory 302 stores a computer program, which, when read and executed by the processor 301, performs the steps described above in the embodiment of the adaptive multimodal data fusion prediction method based on dynamic graph convolution.

[0061] Example 4 In embodiments of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, wherein the computer program is configured to execute the steps described above in the embodiments of the adaptive multimodal data fusion prediction method based on dynamic graph convolution.

[0062] In this embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0063] It will be understood by those skilled in the art that the above descriptions are merely preferred embodiments of this application and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An adaptive multimodal data fusion and prediction method based on dynamic graph convolution, characterized in that, Includes the following steps: From the acquired text input, extract text sentiment feature vectors for representing emotional semantics and personality feature vectors for representing individual characteristics. From the acquired audio and video input, extract audio feature vectors and visual feature vectors. After mapping the text sentiment feature vector and the personality feature vector to a shared space, semantic alignment is performed through contrastive learning constraints and personality weighted regression constraints to obtain the aligned text sentiment feature vector. Using the aligned text sentiment feature vector as the query vector, multi-head attention interaction is performed on the audio feature vector and the visual feature vector respectively to obtain the audio-enhanced feature vector and the visual-enhanced feature vector; A multi-node graph structure is constructed based on the feature vectors of each modality, and cross-modal feature fusion is achieved through dynamic graph convolution, thereby obtaining the fused feature matrix; The fused feature matrix is ​​sequentially passed through the ReLU activation function and a fully connected network, and prediction results are generated according to the task type.

2. The adaptive multimodal data fusion and prediction method based on dynamic graph convolution according to claim 1, characterized in that, The steps of extracting text sentiment feature vectors for representing emotional semantics and personality feature vectors for representing personalized traits from the acquired text input, and extracting audio feature vectors and visual feature vectors from the acquired audio and video input, further include: The acquired text input is encoded in parallel by a pre-trained sentiment encoder and a personality encoder to obtain text sentiment feature vectors and personality feature vectors; A bidirectional recurrent neural network is used to perform sequence modeling on the audio and visual features in the original video stream, and the output of the bidirectional recurrent neural network is obtained as the audio feature vector and the visual feature vector.

3. The adaptive multimodal data fusion and prediction method based on dynamic graph convolution according to claim 1, characterized in that, The step of mapping the text sentiment feature vector and the personality feature vector to a shared space, and then performing semantic alignment through contrastive learning constraints and personality-weighted regression constraints to obtain the aligned text sentiment feature vector further includes: using a contrastive learning loss function to bring the text sentiment feature vector and the personality feature vector of the same text sample closer together in the shared space, while pushing away the feature vectors of different text samples; and using a personality-weighted regression loss function, constraining the sentiment intensity regression of the text sentiment feature vector with the similarity between the text sentiment feature vector and the personality feature vector as the weight.

4. The adaptive multimodal data fusion and prediction method based on dynamic graph convolution according to claim 1, characterized in that, In the step of using the aligned text sentiment feature vector as the query vector and performing multi-head attention interaction on the audio feature vector and the visual feature vector respectively to obtain the audio-enhanced feature vector and the visual-enhanced feature vector, the expression for the visual-enhanced feature vector is: The expression for the audio enhancement feature vector is: ;in Enhance visual feature vectors; The aligned text sentiment feature vector; For visual feature vectors; This represents the visual multi-head attention mechanism; Enhance the feature vector for audio; For audio feature vectors; This represents the audio multi-head attention mechanism.

5. The adaptive multimodal data fusion and prediction method based on dynamic graph convolution according to claim 1, characterized in that, After the step of using the aligned text sentiment feature vector as the query vector and performing multi-head attention interaction on the audio feature vector and the visual feature vector respectively to obtain the audio-enhanced feature vector and the visual-enhanced feature vector, the further step includes applying independent single-modal prediction loss constraints to the aligned text sentiment feature vector, the audio-enhanced feature vector, and the visual-enhanced feature vector respectively: Using real labels as the supervision target, independent cross-entropy loss is applied to the aligned text sentiment feature vector, the audio enhancement feature vector, and the visual enhancement feature vector respectively, to ensure that each modality feature vector has independent state representation ability before entering the fusion step.

6. The adaptive multimodal data fusion and prediction method based on dynamic graph convolution according to claim 1, characterized in that, The step of constructing a multi-node graph structure based on feature vectors of each modality, achieving cross-modal feature fusion through dynamic graph convolution, and obtaining a fused feature matrix further includes: constructing a multi-node structure graph based on relevant feature vectors of text, audio, and vision, encoding the correlation strength between nodes with a dynamically updated adjacency matrix, using a Chebyshev network as a graph convolutional layer to perform high-order convolutional inference on node features, and accumulating the feature matrices output by each layer of the graph convolutional layer to obtain a fused feature matrix; The multi-node graph structure includes two text nodes, two visual nodes, two audio nodes, and two interactive nodes.

7. The adaptive multimodal data fusion and prediction method based on dynamic graph convolution according to claim 1, characterized in that, Also includes: An end-to-end deep learning network is trained by jointly optimizing the total loss function, and the deep learning network is used in the method; the expression of the total loss function is: in, This is the total loss function; The multimodal prediction loss function; Let represent the single-modal prediction loss function, where t represents the text modality, a represents the audio modality, and v represents the visual modality; This is the composite alignment loss function; and Hyperparameters are used to balance the contributions of each loss term.

8. An adaptive multimodal data fusion and prediction system based on dynamic graph convolution, characterized in that, The system employs the adaptive multimodal data fusion prediction method based on dynamic graph convolution as described in any one of claims 1 to 7, the system comprising: The multi-path feature extraction module is used to extract text sentiment feature vectors for representing emotional semantics and personality feature vectors for representing individual characteristics from the acquired text input, and to extract audio feature vectors and visual feature vectors from the acquired audio and video input. The personality-emotion alignment module is used to map the text sentiment feature vector and the personality feature vector to a shared space, and then perform semantic alignment through contrastive learning constraints and personality weighted regression constraints to obtain the aligned text sentiment feature vector. The cross-modal attention interaction module is used to perform multi-head attention interaction on the audio feature vector and the visual feature vector respectively, using the aligned text sentiment feature vector as the query vector, to obtain the audio enhancement feature vector and the visual enhancement feature vector; The dynamic graph convolution fusion module is used to construct a multi-node graph structure based on the feature vectors of each modality, and to achieve cross-modal feature fusion through dynamic graph convolution, thereby obtaining a fused feature matrix; The joint optimization module is used to sequentially pass the fused feature matrix through the ReLU activation function and the fully connected network, and generate prediction results according to the task type.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor is configured to execute the computer program stored in the memory to implement the adaptive multimodal data fusion prediction method based on dynamic graph convolution as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is loaded and executed by a processor to implement the adaptive multimodal data fusion prediction method based on dynamic graph convolution as described in any one of claims 1 to 7.

Images