A multi-modal data fusion method and system for visual text interaction

Abstract

The application discloses a kind of multi-modal data fusion methods and systems for visual text interaction, wherein the method comprises: the original video data and original text data are preprocessed;Visual feature representation is obtained by embedding visual features based on the original video data using spatiotemporal graph convolution model;Text feature representation is obtained by processing original text data using hierarchical semantic aggregation based on BERT model;Aggregated appearance node representation and aggregated motion node representation are generated using visual feature representation and text feature representation based on interaction matrix to generate target visual representation;Visual text joint representation is obtained by fusing target visual representation and text feature representation, and visual text interaction task is performed based on visual text joint representation.The application solves the problem of shallow semantic fusion level and simple visual text interaction in existing multi-modal data fusion technology.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multimodal data fusion method and system for visual text interaction. Background Technology

[0002] In today's information age, data has become a key resource driving technological progress and social development. With the rapid development of sensor technology, communication technology, and computing power, the types and quantities of data that can be collected and processed have exploded. This data often spans multiple sensory modalities, such as vision, hearing, and touch, forming so-called multimodal data. Multimodal data fusion refers to combining data from different modalities to obtain richer and more comprehensive information than single-modal data. Different modalities of data have different characteristics and expressions, and the differences in format, scale, and expression make direct fusion difficult.

[0003] While current data fusion algorithms have achieved good results to some extent, they still have shortcomings. Traditional Transformer-based multimodal pre-trained models ignore some high-level multimodal semantic features. Furthermore, because the Transformer model only uses features extracted from the top layer for prediction and recognition, it ignores detailed information from other layers, making low-dimensional features unsuitable for model inference. Transformer models require multiple layers of neural networks, but deeper networks not only lead to vanishing gradients at lower levels but also blur positional encoding, reducing the effectiveness of feature fusion between multimodal data and sacrificing the generalization ability of the multimodal model. This necessitates a large amount of training data, increasing the training difficulty. As for fusion strategies based on graph neural networks, their overly simple text and visual embedding modules are unable to handle complex visual scenes and text inputs. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art. This invention provides a multimodal data fusion method and system for visual text interaction, which solves the problems of shallow semantic fusion level and simple visual text interaction in existing multimodal data fusion technologies.

[0005] To address the aforementioned technical problems, this invention provides a multimodal data fusion method for visual text interaction, the method comprising:

[0006] Acquire raw video data and raw text data, and preprocess the raw video data and raw text data to obtain preprocessed raw video data and raw text data;

[0007] Visual feature representations are obtained by embedding visual features into the preprocessed raw video data based on the spatiotemporal graph convolution model.

[0008] Based on the BERT model, hierarchical semantic aggregation processing is performed on the preprocessed original text data to obtain text feature representations;

[0009] Based on the interaction relationship matrix, aggregated appearance node representation and aggregated motion node representation are generated using the visual feature representation and text feature representation, and a target visual representation is generated based on the aggregated appearance node representation and aggregated motion node representation;

[0010] The visual representation of the target and the text feature representation are fused to obtain a joint visual-text representation, and a visual-text interaction task is performed based on the joint visual-text representation.

[0011] Optionally, the step of embedding visual features into the preprocessed original video data based on the spatiotemporal graph convolution model to obtain visual feature representations includes:

[0012] Motion and appearance features are extracted from pre-trained CNN models based on pre-processed raw video data;

[0013] A visual feature enhancement module based on spatiotemporal gating unit is used to combine the motion features and appearance features to generate motion feature representations and appearance feature representations;

[0014] Motion graphs and appearance graphs are constructed based on the motion feature representations and appearance feature representations, and visual feature representations are generated based on the motion graphs and appearance graphs.

[0015] Optionally, constructing the motion map and appearance map based on the motion feature representation and appearance feature representation includes:

[0016] Link scores are defined for the motion features and appearance features, and the node connection relationships between the motion graph and appearance graph are determined based on the link scores.

[0017] Calculate the first weight matrix representing the motion features and the second weight matrix representing the appearance features, and determine the connectivity based on the first weight matrix and the second weight matrix;

[0018] Motion graphs and appearance graphs are constructed based on the node connection relationships and connectivity.

[0019] Optionally, the expression for the link score is:

[0020]

[0021] Among them, s i,j Let be the link score between object i and object j, and ψ be the cosine similarity between object i and object j. Let t ∈ {1,2,…,l} be the appearance features of object i. v}, l v For frame length, Let λ be the appearance feature of object j, λ be the scaling factor, and IoU be the intersection-union ratio. For the motion characteristics of object i, The motion characteristics of object j;

[0022] The expression for the first weight matrix is:

[0023]

[0024] in, Let be the first weight matrix, exp be the exponential function, and λ be the scaling factor. Let T = γLK represent the appearance features of object i, where L is the length of the clip and K is the number of clips, and γ∈(0,1). This represents the appearance characteristics of object j;

[0025] The expression for the second weight matrix is:

[0026]

[0027] in, Let be the second weight matrix, exp be the exponential function, and λ be the scaling factor. Let T = γLK represent the motion characteristics of object i, where L is the length of the clip and K is the number of clips, and γ∈(0,1). The motion characteristics of object j are represented;

[0028] The expression for the connectivity is:

[0029]

[0030] Where σ is the activation function, R t For connectivity, This is the first weight matrix. This is the second weight matrix. This is a linear transformation of the first weight matrix. This is a linear transformation of the second weight matrix. Let T = γLK represent the appearance features of object i, where L is the length of the clip and K is the number of clips, and γ∈(0,1). Let the appearance features of object j be represented. Let i be the motion feature representation of object i. Let j be the motion characteristic representation of object j.

[0031] Optionally, the hierarchical semantic aggregation processing of the preprocessed original text data based on the BERT model to obtain text feature representation includes:

[0032] Entity recognition technology is used to extract entities from preprocessed raw text data to obtain target text entities, and the target text entities are converted into a first embedding vector.

[0033] Based on the BERT model, the first embedding vector is semantically encoded to obtain entity semantic information;

[0034] The entity semantic information is decomposed into semantic units at different levels, including entity level, atomic action level, action level, and event level.

[0035] Optionally, the decomposition of the entity semantic information into semantic units at different levels includes:

[0036] The entity semantic information is used as the entity level of the semantic unit;

[0037] Based on syntactic analysis technology, verb phrase extraction is performed on the preprocessed raw text data to obtain target verb phrases, and the target verb phrases are used as atomic action representations;

[0038] The atomic action representation is refined based on semantic role labeling technology to obtain a refined atomic action representation;

[0039] The refined atomic action representation is converted into a second embedding vector. Based on the edge aggregation mechanism of the graph neural network, the second embedding vector is combined with the entity semantic information to integrate information and obtain a high-level atomic action representation. The high-level atomic action representation is used as the atomic action level.

[0040] The atomic action level is captured by a long short-term network to obtain the target action sequence and temporal relationship, and the action level semantic representation is generated by using the target action sequence and temporal relationship based on an attention mechanism.

[0041] Based on event extraction technology, an event semantic template is constructed using the action-level semantic representation, and an event-level embedding representation is generated using the event semantic template based on the hierarchical aggregation mechanism of graph neural networks.

[0042] Optionally, the step of generating aggregated appearance node representations and aggregated motion node representations based on the interaction relationship matrix using the visual feature representations and text feature representations, and generating a target visual representation based on the aggregated appearance node representations and aggregated motion node representations, includes:

[0043] Based on the visual feature representation and text feature representation, an interaction relationship matrix is ​​generated using the softmax function;

[0044] Based on the interaction relationship matrix, aggregated appearance node representation and aggregated motion node representation are generated using a fully connected layer and problem nodes;

[0045] Perform continuous spatiotemporal graph convolution on the aggregated appearance node representation and aggregated motion node representation to obtain the aggregated appearance node representation and aggregated motion node representation after continuous spatiotemporal graph convolution;

[0046] Average pooling is performed on the aggregated appearance node representation and aggregated motion node representation after convolution of the continuous spatiotemporal graph to obtain the vectorized representation of the aggregated appearance node and the vectorized representation of the aggregated motion node.

[0047] The vectorized representations of the aggregated appearance nodes and the vectorized representations of the aggregated motion nodes are concatenated to obtain an integrated visual representation, and the target visual representation is generated based on the heterogeneous Transformer encoder using the integrated visual representation.

[0048] Optionally, the expression for the interaction relationship matrix is:

[0049]

[0050] Among them, S v Let λ be the interaction matrix, and λ be the scaling factor. Represented by visual features, For text feature representation, T = γLK, where L is the length of the clip, K is the number of clips, and γ∈(0,1);

[0051] The expression represented by the aggregated appearance node is:

[0052]

[0053] Where σ is the activation function, This represents the aggregated appearance node. This is the weight matrix of the fully connected layer. Let K represent the appearance features, and K be the total number of problem nodes. Let i be the interaction value in the interaction relationship matrix, j be the index of the visual node, and u be the index of the question node. j Let b be the bias term, and let b be the problem graph representation.

[0054] The expression representing the aggregated motion node is:

[0055]

[0056] Where σ is the activation function, Represented as aggregated motion nodes. This is the weight matrix of the fully connected layer. The motion characteristics are represented by K, where K is the total number of problem nodes. Let i be the interaction value in the interaction relationship matrix, j be the index of the visual node, and u be the index of the question node. j Let b be the problem diagram, and b be the bias term.

[0057] Optionally, fusing the target visual representation and text feature representation to obtain a joint visual-text representation includes:

[0058] The target visual representation and text feature representation are linearly transformed to obtain the linearly transformed target visual representation and text feature representation;

[0059] The linearly transformed visual representation of the target and the text feature representation are concatenated to obtain the concatenated visual representation of the target and the text feature representation.

[0060] Heterogeneous mapping is performed on the concatenated visual representation of the target and the text feature representation to obtain a joint visual-text representation. The expression for obtaining the joint visual-text mapping is as follows:

[0061]

[0062] y′=σ(W y y+b),

[0063] s = softmax(W y′ y′+b),

[0064] Where y represents the concatenated visual representation and text feature representation of the target, σ is the activation function, and W1, W2, W... y W y′ For different mapping functions, For the visual representation of the target, Let be the text feature representation, b be the bias term, y′ be the heterogeneous mapping result, and s be the visual-text joint mapping.

[0065] In addition, the present invention also provides a multimodal data fusion system for visual text interaction, the system comprising:

[0066] Data acquisition and preprocessing module: used to acquire raw video data and raw text data, and preprocess the raw video data and raw text data to obtain preprocessed raw video data and raw text data;

[0067] Visual feature representation acquisition module: used to embed visual features into the preprocessed raw video data based on the spatiotemporal graph convolution model to obtain visual feature representations;

[0068] Text Feature Representation Acquisition Module: Used to perform hierarchical semantic aggregation processing on preprocessed raw text data based on the BERT model to obtain text feature representations;

[0069] Target visual representation acquisition module: used to generate aggregated appearance node representation and aggregated motion node representation based on the interaction relationship matrix using the visual feature representation and text feature representation, and to generate target visual representation based on the aggregated appearance node representation and aggregated motion node representation;

[0070] Visual-text joint representation acquisition module: used to fuse the target visual representation and text feature representation to obtain a visual-text joint representation, and to perform a visual-text interaction task based on the visual-text joint representation.

[0071] In this embodiment of the invention, visual features are embedded into preprocessed raw video data based on a spatiotemporal graph convolutional model to obtain visual feature representations. This achieves multi-level extraction and association of visual features. The spatiotemporal graph convolutional model can robustly encode dynamic and static elements in the video, greatly enhancing the model's ability to handle complex video inference tasks. Based on the BERT model, hierarchical semantic aggregation is performed on preprocessed raw text data to obtain text feature representations, achieving multi-level feature extraction of text data and solving the feature loss problem caused by long text inputs. Based on the interaction relation matrix, aggregated appearance node representations and aggregated motion node representations are generated using the visual feature representations and text feature representations to generate a target visual representation. This effectively aggregates node information, integrates data from various nodes, and ensures a comprehensive representation of interactions within the network. The target visual representation and text feature representation are fused to obtain a joint visual-text representation to perform visual-text interaction tasks, achieving multi-level fusion of visual and semantic information. This solves the problems of shallow semantic fusion and simple visual-text interaction in existing multimodal data fusion technologies. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a flowchart illustrating the multimodal data fusion method for visual text interaction in an embodiment of the present invention.

[0074] Figure 2 This is a schematic diagram of the structural composition of a multimodal data fusion system for visual text interaction according to an embodiment of the present invention. Detailed Implementation

[0075] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0076] Example 1

[0077] Please see Figure 1 , Figure 1 This is a flowchart illustrating the multimodal data fusion method for visual text interaction in an embodiment of the present invention.

[0078] like Figure 1 As shown, a multimodal data fusion method for visual-text interaction is described, the method comprising:

[0079] S11: Acquire raw video data and raw text data, and preprocess the raw video data and raw text data to obtain preprocessed raw video data and raw text data;

[0080] In the specific implementation of this invention, raw video data and raw text data are acquired. The raw text data may include text data obtained through speech recognition. The raw video data and raw text data are preprocessed. The raw video data undergoes denoising processing, i.e., noise removal from the video, resulting in denoised raw video data. The raw text data undergoes denoising processing, which involves cleaning the text data, removing web page tags, non-semantic characters, and garbled text, resulting in denoised raw text data. The denoised raw video data and raw text data undergo redundancy removal processing. Since multimodal data contains some data with repeated dimensions, which are meaningless for subsequent feature extraction and analysis, this redundant data with repeated dimensions is removed. The multimodal data is divided into several data blocks, and the values ​​in each data block are calculated. The hash value is used to calculate the code distance between data. If the code distance is less than a preset threshold, it is considered redundant data. The redundant data is then removed to obtain the deredundant original video data and original text data. The deredundant original video data and original text data are then normalized to eliminate the order-of-magnitude differences between each data point, avoiding excessive differences in data levels that could lead to large prediction errors in subsequent data. This yields normalized original video data and normalized original text data. Keyframes are extracted from the normalized original video data to obtain preprocessed original video data. After the original text data undergoes normalization, the preprocessing is complete, yielding preprocessed original text data.

[0081] S12: Visual feature embedding is performed on the preprocessed raw video data based on the spatiotemporal graph convolution model to obtain visual feature representation;

[0082] In the specific implementation of this invention, the step of embedding visual features into the preprocessed original video data based on the spatiotemporal graph convolution model to obtain visual feature representations includes: extracting motion features and appearance features from the preprocessed original video data based on a pre-trained CNN model; using a visual feature enhancement module based on spatiotemporal gating units to combine the motion features and appearance features to generate motion feature representations and appearance feature representations; constructing motion graphs and appearance graphs based on the motion feature representations and appearance feature representations; and generating visual feature representations based on the motion graphs and appearance graphs.

[0083] Furthermore, the step of constructing a motion graph and an appearance graph based on the motion feature representation and the appearance feature representation includes: defining link scores for the motion features and appearance features, determining the node connection relationships of the motion graph and the appearance graph based on the link scores; calculating a first weight matrix of the motion feature representation and a second weight matrix of the appearance feature representation, determining connectivity based on the first weight matrix and the second weight matrix; and constructing the motion graph and the appearance graph based on the node connection relationships and connectivity.

[0084] Furthermore, the expression for the link score is:

[0085]

[0086] Among them, s i,j Let be the link score between object i and object j, and ψ be the cosine similarity between object i and object j. Let t ∈ {1,2,…,l} be the appearance features of object i. v}, l v For frame length, Let λ be the appearance feature of object j, λ be the scaling factor, and IoU be the intersection-union ratio. For the motion characteristics of object i, The motion characteristics of object j;

[0087] The expression for the first weight matrix is:

[0088]

[0089] in, Let be the first weight matrix, exp be the exponential function, and λ be the scaling factor. Let T = γLK represent the appearance features of object i, where L is the length of the clip and K is the number of clips, and γ∈(0,1). This represents the appearance characteristics of object j;

[0090] The expression for the second weight matrix is:

[0091]

[0092] in, Let be the second weight matrix, exp be the exponential function, and λ be the scaling factor. Let T = γLK represent the motion characteristics of object i, where L is the length of the clip and K is the number of clips, and γ∈(0,1). The motion characteristics of object j are represented;

[0093] The expression for the connectivity is:

[0094]

[0095] Where σ is the activation function, R t For connectivity, This is the first weight matrix. This is the second weight matrix. This is a linear transformation of the first weight matrix. This is a linear transformation of the second weight matrix. Let T = γLK represent the appearance features of object i, where L is the length of the clip and K is the number of clips, and γ∈(0,1). Let the appearance features of object j be represented. Let i be the motion feature representation of object i. Let j be the motion characteristic representation of object j.

[0096] Specifically, constructing node representations of visual features is a key step in the visual feature embedding module based on graph neural networks. It involves converting visual information in images or videos into nodes in a graph structure for further analysis and processing. Preprocessed raw video data is extracted according to a preset speed and divided into several clips of preset length. For each clip, a dense stream of several frames is maintained to obtain clip-level motion features, and a sparse stream of several frames is maintained to obtain region and frame appearance features. This involves extracting motion and appearance features from the preprocessed raw video data based on a pre-trained Convolutional Neural Network (CNN) model, specifically the 3D version of the ResNeXt-101 model for motion and the ResNet-101 model for frame appearance. A visual feature enhancement module based on a spatiotemporal gating unit is used to combine motion features and appearance features to generate motion feature representations and appearance feature representations. Several appearance features and their bounding boxes are extracted from each frame of the sparse stream. The appearance features and motion features are mapped to a feature space of a preset dimension. The visual feature enhancement module of the spatiotemporal gating unit then forms the feature representation output, thus obtaining the motion feature representation and appearance feature representation. To find identical objects between different frames within a segment, a link score is defined by considering their appearance and spatial location. That is, a link score is defined for the motion features and appearance features. The expression for the link score is:

[0097]

[0098] Among them, s i,j Let be the link score between object i and object j, and ψ be the cosine similarity between object i and object j. Let t ∈ {1,2,…,l} be the appearance features of object i. v}, l v For frame length, Let λ be the appearance feature of object j, λ be the scaling factor, and IoU be the intersection-union ratio. For the motion characteristics of object i, Let λ represent the motion feature of object j, where object i and object j are two objects detected in adjacent frames. Here, λ can be set to 1. Several detected objects in the first frame of each segment are designated as anchor objects. Then, by greedily maximizing the link score frame by frame, the detected objects in consecutive frames are connected to the anchor objects. By aligning objects within a segment, we ensure the consistency of the node and edge representations of the graphs constructed in different frames. In the graph representation used in the model, each node is closely connected to every other node through edges. These edges represent the relationships between nodes and are quantized using vectors that encapsulate the relevant features of each node. That is, the node connection relationships of the motion graph and appearance graph are determined based on the link score. A first weight matrix of the motion feature representation and a second weight matrix of the appearance feature representation are calculated. The weight matrices represent the connections between nodes and their corresponding edge weights, which can be calculated based on the affinity between the motion feature representation and the appearance feature representation. Connectivity is determined based on the first weight matrix and the second weight matrix. The expression for the first weight matrix is:

[0099]

[0100] in, Let be the first weight matrix, exp be the exponential function, and λ be the scaling factor. Let T = γLK represent the appearance features of object i, where L is the length of the clip and K is the number of clips, and γ∈(0,1). This represents the appearance characteristics of object j;

[0101] The expression for the second weight matrix is:

[0102]

[0103] in, Let be the second weight matrix, exp be the exponential function, and λ be the scaling factor. Let T = γLK represent the motion characteristics of object i, where L is the length of the clip and K is the number of clips, and γ∈(0,1). The motion characteristics of object j are represented;

[0104] The expression for the connectivity is:

[0105]

[0106] Where σ is the activation function, R t For connectivity, This is the first weight matrix. This is the second weight matrix. This is a linear transformation of the first weight matrix. This is a linear transformation of the second weight matrix. Let T = γLK represent the appearance features of object i, where L is the length of the clip and K is the number of clips, and γ∈(0,1). Let the appearance features of object j be represented. Let i be the motion feature representation of object i. The motion characteristics of object j are represented;

[0107] The linear transformation employs different weighting mechanisms: the linear transformation of the first weight matrix and the linear transformation of the second weight matrix. These different transformations reflect the asymmetric nature of the interaction between subjective and objective elements in the real world. Visual feature representations are generated based on the motion graph and appearance graph. The motion graph and appearance graph are then merged through matrix leverage, integrating different graph structures into a unified matrix. Subsequently, this concatenated matrix is ​​processed by a self-attention layer, designed to enhance the spatiotemporally interwoven representation. The self-attention mechanism plays a crucial role in highlighting important features and interactions in the appearance and motion dimensions. In this way, the model can capture a more comprehensive and detailed understanding of the video content, thereby gaining deeper insights into the dynamic interactions displayed in the video.

[0108] S13: Based on the BERT model, hierarchical semantic aggregation processing is performed on the preprocessed original text data to obtain text feature representations;

[0109] In the specific implementation of this invention, the hierarchical semantic aggregation processing of the preprocessed original text data based on the BERT model to obtain text feature representation includes: extracting entities from the preprocessed original text data based on entity recognition technology to obtain target text entities, and converting the target text entities into first embedding vectors; encoding semantic information into the first embedding vectors based on the BERT model to obtain entity semantic information; and decomposing the entity semantic information into semantic units at different levels, including entity level, atomic action level, action level, and event level.

[0110] Furthermore, the step of decomposing the entity semantic information into semantic units at different levels includes: using the entity semantic information as the entity level of the semantic unit; performing verb phrase extraction processing on the preprocessed original text data based on syntactic analysis technology to obtain target verb phrases, and using the target verb phrases as atomic action representations; refining the atomic action representations based on semantic role labeling technology to obtain refined atomic action representations; converting the refined atomic action representations into second embedding vectors, and using the edge aggregation mechanism of graph neural networks to integrate the second embedding vectors with the entity semantic information to obtain high-level atomic action representations, and using the high-level atomic action representations as atomic action levels; capturing action sequences and temporal relationships of the atomic action levels based on long short-term networks to obtain target action sequences and temporal relationships, and generating action-level semantic representations based on attention mechanisms using the target action sequences and temporal relationships; constructing event semantic templates using the action-level semantic representations based on event extraction technology, and generating event-level embedding representations using the event semantic templates based on the hierarchical aggregation mechanism of graph neural networks.

[0111] Specifically, the text embedding module based on hierarchical semantic aggregation is a key technology in multimodal data fusion. Its core lies in hierarchical processing of text features to achieve refined semantic representation and deep information fusion. This module decomposes text content into semantic units at different levels, including entity level, atomic action level, action level, and event level. At the entity level, the module first identifies named entities in the text, such as person names, place names, and organization names. These entities, as basic semantic units of the text, are crucial for understanding the text content. Based on entity recognition technology, entity extraction is performed on the preprocessed raw text data. Entity recognition techniques, such as conditional random fields or deep learning-based named entity recognition models, obtain target text entities and convert them into first embedding vectors. The first embedding vector is then encoded with semantic information using a Bidirectional Encoder Representations from Transformers (BERT) model. A word embedding model is constructed using the BERT model, further encoding the semantic information of the entities, laying the foundation for subsequent semantic aggregation. Alternatively, a global word frequency statistics-based word representation model can be used to construct the word embedding model to obtain entity semantic information, i.e., to obtain the entity-level semantic units. Atomic action-level features involve basic action or event units in text, typically composed of verbs and their associated objects. Based on syntactic analysis techniques, verb phrase extraction is performed on the preprocessed raw text data. Syntactic analysis techniques, such as dependency parsing, obtain target verb phrases, which are then used as atomic action representations. Semantic role labeling techniques are used to refine these atomic action representations, clarifying the roles of the agent, patient, etc., in the action. The refined atomic action representations are then converted into second embedding vectors. Using a graph neural network edge aggregation mechanism, these second embedding vectors are combined with the entity semantic information for information integration. This integration of the semantic information of the action and its participants forms a high-level representation of the atomic action, which is then used as the atomic action level. Action-level features are a further abstraction and generalization of atomic action-level features, involving the combination of multiple related atomic actions into more complex action sequences or patterns. Based on a Long Short-Term Network (LST), the atomic action level is used to capture action sequences and temporal relationships to obtain target action sequences and temporal relationships. Then, based on an attention mechanism, action-level semantic representations are generated using these target action sequences and temporal relationships. The target action sequences and temporal relationships are encoded into higher-dimensional embedding vectors, and the attention mechanism emphasizes key actions and transitions within the target action sequences and temporal relationships, thus forming action-level semantic representations, i.e., obtaining the action level of semantic units. Event-level features are the highest level in the text embedding module, integrating related action-level features into a complete event framework.Event semantic templates are constructed using the action-level semantic representation based on event extraction techniques, such as template matching or graph neural network-based event extraction models. Key events are identified, and event semantic templates are constructed using the identified key events. These event semantic templates include attributes such as event type, participants, time, and location. Event-level embedding representations are generated using the event semantic templates based on the hierarchical aggregation mechanism of graph neural networks. Semantic information from different levels is integrated through the hierarchical aggregation mechanism of graph neural networks to form event-level embedding representations.

[0112] S14: Based on the interaction relationship matrix, generate aggregated appearance node representation and aggregated motion node representation using the visual feature representation and text feature representation, and generate target visual representation based on the aggregated appearance node representation and aggregated motion node representation;

[0113] In a specific implementation of this invention, the step of generating aggregated appearance node representations and aggregated motion node representations based on the interaction relationship matrix using the visual feature representations and text feature representations, and generating a target visual representation based on the aggregated appearance node representations and aggregated motion node representations, includes: generating an interaction relationship matrix using the softmax function based on the visual feature representations and text feature representations; generating aggregated appearance node representations and aggregated motion node representations based on the interaction relationship matrix using fully connected layers and question nodes; performing continuous spatiotemporal graph convolution on the aggregated appearance node representations and aggregated motion node representations to obtain aggregated appearance node representations and aggregated motion node representations after continuous spatiotemporal graph convolution; performing average pooling on the aggregated appearance node representations and aggregated motion node representations after continuous spatiotemporal graph convolution to obtain vectorized representations of aggregated appearance nodes and aggregated motion nodes; concatenating the vectorized representations of aggregated appearance nodes and aggregated motion nodes to obtain an integrated visual representation, and generating a target visual representation based on the integrated visual representation using a heterogeneous Transformer encoder.

[0114] Furthermore, the expression for the interaction relationship matrix is:

[0115]

[0116] Among them, S v Let λ be the interaction matrix, and λ be the scaling factor. Represented by visual features, For text feature representation, T = γLK, where L is the length of the clip, K is the number of clips, and γ∈(0,1);

[0117] The expression represented by the aggregated appearance node is:

[0118]

[0119] Where σ is the activation function, This represents the aggregated appearance node. This is the weight matrix of the fully connected layer. Let K represent the appearance features, and K be the total number of problem nodes. Let i be the interaction value in the interaction relationship matrix, j be the index of the visual node, and u be the index of the question node. j Let b be the bias term, and let b be the problem graph representation.

[0120] The expression representing the aggregated motion node is:

[0121]

[0122] Where σ is the activation function, Represented as aggregated motion nodes. This is the weight matrix of the fully connected layer. The motion characteristics are represented by K, where K is the total number of problem nodes. Let i be the interaction value in the interaction relationship matrix, j be the index of the visual node, and u be the index of the question node. j Let b be the problem diagram, and b be the bias term.

[0123] Specifically, the goal of the visual-to-text interaction is to learn visual representations under problem conditions. This process involves associating problem nodes with corresponding visual nodes and propagating the problem representation along visual edges through a spatial-temporal graph convolutional layer. We employ a dynamic graph Transformer method to effectively aggregate node information. This method utilizes a multi-head attention mechanism to synthesize data from various nodes, ensuring a comprehensive representation of interactions within the network. Furthermore, edge information is also aggregated in this way and processed using an edge Transformer method. Based on the visual and text feature representations, an interaction matrix is ​​generated using a softmax function. The softmax function is applied to the affinity matrix between the visual and text feature representations on the problem edges to generate the interaction matrix, the expression of which is:

[0124]

[0125] Among them, S v Let λ be the interaction matrix, and λ be the scaling factor. Represented by visual features, For text feature representation, T = γLK, where L is the length of the clip, K is the number of clips, and γ ∈ (0,1). Based on the interaction relation matrix, aggregated appearance node representations and aggregated motion node representations are generated using fully connected layers and question nodes. A scaling factor is used to adjust the influence of affinity values. Each interaction value in the interaction relation matrix quantifies the degree of association between visual nodes and question nodes. All question nodes are aggregated to their corresponding appearance nodes through the interaction relation matrix, and the generated vectors are passed through a fully connected layer to form an aggregated appearance node representation. Similarly, motion features can also form an aggregated motion node representation. The expression for the aggregated appearance node representation is:

[0126]

[0127] Where σ is the activation function, This represents the aggregated appearance node. This is the weight matrix of the fully connected layer. Let K represent the appearance features, and K be the total number of problem nodes. Let i be the interaction value in the interaction relationship matrix, j be the index of the visual node, and u be the index of the question node. j Let b be the bias term, and let b be the problem graph representation.

[0128] The expression representing the aggregated motion node is:

[0129]

[0130] Where σ is the activation function, Represented as aggregated motion nodes. This is the weight matrix of the fully connected layer. The motion characteristics are represented by K, where K is the total number of problem nodes. Let i be the interaction value in the interaction relationship matrix, j be the index of the visual node, and u be the index of the question node. j Let be the problem graph representation, and b be the bias term. Perform continuous spatiotemporal graph convolution on the aggregated appearance node representation and the aggregated motion node representation to obtain the convolved aggregated appearance node representation and the aggregated motion node representation. The expressions for the convolved aggregated appearance node representation and the aggregated motion node representation are:

[0131]

[0132] in, This represents the aggregated appearance node representation after convolution of a continuous spatiotemporal graph. This represents the aggregated motion nodes after convolution of a continuous spatiotemporal graph. and Here, F represents the parameters of the spatiotemporal graph convolutional layer, and W represents the feedforward processing of the spatiotemporal graph convolutional layer.v W is the first weight matrix. m This is the second weight matrix. This represents the aggregated appearance node. The aggregation of motion node representations is performed. Average pooling is applied along the time axis to both the aggregated appearance node representations and the aggregated motion node representations after convolution of the continuous spatiotemporal graph, effectively vectorizing these representations to obtain vectorized representations of aggregated appearance nodes and aggregated motion nodes. This vectorization process simplifies complex spatiotemporal data, making it easier to process. The vectorized representations of aggregated appearance nodes and aggregated motion nodes are then concatenated to obtain an integrated visual representation, the expression of which is:

[0133]

[0134] Where O represents integrated visual representation. Vectorization representation of aggregated appearance nodes. The integrated visual representation of the aggregated motion nodes is used to generate the target visual representation based on the heterogeneous Transformer encoder. The integrated visual representation is then transmitted through the heterogeneous Transformer encoder to generate the final visual representation, i.e., to obtain the target visual representation.

[0135] S15: The target visual representation and text feature representation are fused to obtain a visual-text joint representation, and a visual-text interaction task is performed based on the visual-text joint representation.

[0136] In a specific implementation of this invention, the fusion of the target visual representation and text feature representation to obtain a joint visual-text representation includes: performing a linear transformation on the target visual representation and text feature representation to obtain linearly transformed target visual representation and text feature representation; concatenating the linearly transformed target visual representation and text feature representation to obtain concatenated target visual representation and text feature representation; and performing heterogeneous mapping on the concatenated target visual representation and text feature representation to obtain a joint visual-text representation. The expression for obtaining the joint visual-text mapping is as follows:

[0137]

[0138] y′=σ(W y y+b),

[0139] s = softmax(W y′ y′+b),

[0140] Where y represents the concatenated visual representation and text feature representation of the target, σ is the activation function, and W1, W2, W... y W y′ For different mapping functions, For the visual representation of the target, Let be the text feature representation, b be the bias term, y′ be the heterogeneous mapping result, and s be the visual-text joint mapping.

[0141] Specifically, the target visual representation and text feature representation are linearly transformed to obtain linearly transformed target visual representation and text feature representation; the linearly transformed target visual representation and text feature representation are concatenated to obtain concatenated target visual representation and text feature representation; the concatenated target visual representation and text feature representation are heterogeneously mapped to obtain a visual-text joint representation. The expression for obtaining the visual-text joint mapping process is as follows:

[0142]

[0143] y′=σ(W y y+b),

[0144] s = softmax(W y′ y′+b),

[0145] Where y represents the concatenated visual representation and text feature representation of the target, σ is the activation function, and W1, W2, W... y W y′ For different mapping functions, For the visual representation of the target, Let be the text feature representation, b be the bias term, y′ be the heterogeneous mapping result, and s be the visual-text joint mapping. The target visual representation can be object-level, frame-level, or clip-level visual features, while the text feature representation can be word-level, sentence-level, or paragraph-level text features. For the output visual-text joint representation, we found that it can fuse visual and text features at different levels, effectively solving the difficulty of multi-granularity fusion representation in multimodal information fusion. We then applied the fused visual-text features to a visual-text interaction task. This task involves utilizing the bidirectional encoding context capabilities of the BERT model to establish contextual dependencies within the text. Subsequently, CoreNLP is used to identify semantic dependencies between questions and candidate answers. CoreNLP is a suite of natural language processing tools developed by the Stanford University Natural Language Processing Group to help users perform text analysis and understanding. This analysis involves parsing the part-of-speech tags of various types of words in the sentence. Based on this language analysis, each word is assigned a node, and a graphical representation of the question is constructed. Weights are assigned based on part-of-speech tags, and the similarity matrix reflecting these relationships can be described as follows:

[0146]

[0147] Among them, e ij Let λ represent the affinity between the i-th and j-th problem nodes, λ be the scaling factor, exp be the exponential function, and K be the total number of problem nodes. Let be the text feature representation of the j-th question node. Let T = γLK, where L is the length of the clip, K is the number of clips, γ∈(0,1), and λ is a scaling factor to adjust the influence between question nodes. Then, a weight matrix is ​​calculated using the adjacency matrix, which represents the connectivity of the question graph. This is then used for question-and-answer processing between the question and candidate answers. This comprehensive integration process ultimately produces a well-researched answer prediction. By effectively integrating insights from both visual and textual domains, the model can efficiently answer the questions posed, demonstrating robust cross-modal reasoning capabilities.

[0148] In this embodiment of the invention, visual features are embedded into preprocessed raw video data based on a spatiotemporal graph convolutional model to obtain visual feature representations. This achieves multi-level extraction and association of visual features. The spatiotemporal graph convolutional model can robustly encode dynamic and static elements in the video, greatly enhancing the model's ability to handle complex video inference tasks. Based on the BERT model, hierarchical semantic aggregation is performed on preprocessed raw text data to obtain text feature representations, achieving multi-level feature extraction of text data and solving the feature loss problem caused by long text inputs. Based on the interaction relation matrix, aggregated appearance node representations and aggregated motion node representations are generated using the visual feature representations and text feature representations to generate a target visual representation. This effectively aggregates node information, integrates data from various nodes, and ensures a comprehensive representation of interactions within the network. The target visual representation and text feature representation are fused to obtain a joint visual-text representation to perform visual-text interaction tasks, achieving multi-level fusion of visual and semantic information. This solves the problems of shallow semantic fusion and simple visual-text interaction in existing multimodal data fusion technologies.

[0149] Example 2

[0150] Please see Figure 2 , Figure 2 This is a schematic diagram of the structural composition of a multimodal data fusion system for visual text interaction according to an embodiment of the present invention.

[0151] like Figure 2 As shown, a multimodal data fusion system for visual-text interaction is disclosed, the system comprising:

[0152] Data acquisition and preprocessing module 21: used to acquire raw video data and raw text data, and preprocess the raw video data and raw text data to obtain preprocessed raw video data and raw text data;

[0153] Visual feature representation acquisition module 22: used to embed visual features into the preprocessed raw video data based on the spatiotemporal graph convolution model to obtain visual feature representation;

[0154] Text feature representation acquisition module 23: used to perform hierarchical semantic aggregation processing on the preprocessed original text data based on the BERT model to obtain text feature representation;

[0155] Target visual representation acquisition module 24: used to generate aggregated appearance node representation and aggregated motion node representation based on the interaction relationship matrix using the visual feature representation and text feature representation, and generate target visual representation based on the aggregated appearance node representation and aggregated motion node representation;

[0156] Visual-text joint representation acquisition module 25: used to fuse the target visual representation and text feature representation to obtain a visual-text joint representation, and to perform a visual-text interaction task based on the visual-text joint representation.

[0157] In the specific implementation of this invention, the specific implementation methods of the system items can be referred to the implementation methods of the above-mentioned method items, and will not be repeated here.

[0158] In this embodiment of the invention, visual features are embedded into preprocessed raw video data based on a spatiotemporal graph convolutional model to obtain visual feature representations. This achieves multi-level extraction and association of visual features. The spatiotemporal graph convolutional model can robustly encode dynamic and static elements in the video, greatly enhancing the model's ability to handle complex video inference tasks. Based on the BERT model, hierarchical semantic aggregation is performed on preprocessed raw text data to obtain text feature representations, achieving multi-level feature extraction of text data and solving the feature loss problem caused by long text inputs. Based on the interaction relation matrix, aggregated appearance node representations and aggregated motion node representations are generated using the visual feature representations and text feature representations to generate a target visual representation. This effectively aggregates node information, integrates data from various nodes, and ensures a comprehensive representation of interactions within the network. The target visual representation and text feature representation are fused to obtain a joint visual-text representation to perform visual-text interaction tasks, achieving multi-level fusion of visual and semantic information. This solves the problems of shallow semantic fusion and simple visual-text interaction in existing multimodal data fusion technologies.

[0159] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0160] Furthermore, the above provides a detailed description of a multimodal data fusion method and system for visual text interaction provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multimodal data fusion method for visual-text interaction, characterized in that, The method includes: Acquire raw video data and raw text data, and preprocess the raw video data and raw text data to obtain preprocessed raw video data and raw text data; Visual feature representations are obtained by embedding visual features into the preprocessed raw video data based on the spatiotemporal graph convolution model. Based on the BERT model, hierarchical semantic aggregation processing is performed on the preprocessed original text data to obtain text feature representations; Based on the interaction relationship matrix, aggregated appearance node representation and aggregated motion node representation are generated using the visual feature representation and text feature representation, and a target visual representation is generated based on the aggregated appearance node representation and aggregated motion node representation; The visual representation of the target and the text feature representation are fused to obtain a joint visual-text representation, and a visual-text interaction task is performed based on the joint visual-text representation; The step of generating aggregated appearance node representations and aggregated motion node representations based on the interaction relationship matrix using the visual feature representations and text feature representations, and generating a target visual representation based on the aggregated appearance node representations and aggregated motion node representations, includes: generating an interaction relationship matrix using the softmax function based on the visual feature representations and text feature representations; generating aggregated appearance node representations and aggregated motion node representations based on the interaction relationship matrix using a fully connected layer and question nodes; performing continuous spatiotemporal graph convolution on the aggregated appearance node representations and aggregated motion node representations to obtain aggregated appearance node representations and aggregated motion node representations after continuous spatiotemporal graph convolution; performing average pooling on the aggregated appearance node representations and aggregated motion node representations after continuous spatiotemporal graph convolution to obtain vectorized representations of aggregated appearance nodes and aggregated motion nodes; concatenating the vectorized representations of aggregated appearance nodes and aggregated motion nodes to obtain an integrated visual representation, and generating a target visual representation based on the integrated visual representation using a heterogeneous Transformer encoder; The expression for the interaction relationship matrix is: ， in, This is an interaction relationship matrix. Scaling factor Represented by visual features, Text feature representation, L is the length of the clip, and K is the number of clips. ; The expression represented by the aggregated appearance node is: ， in, For activation function, This represents the aggregated appearance node. This is the weight matrix of the fully connected layer. Let K represent the appearance features, and K be the total number of problem nodes. Let be the interaction value in the interaction relationship matrix, where i is the index of the visual node and j is the index of the question node. Let b be the bias term, and let b be the problem graph representation. The expression representing the aggregated motion node is: ， in, For activation function, Represented as aggregated motion nodes. This is the weight matrix of the fully connected layer. Let K be the total number of problem nodes, representing the motion characteristics. Let be the interaction value in the interaction relationship matrix, where i is the index of the visual node and j is the index of the question node. Let b be the problem diagram, and b be the bias term.

2. The multimodal data fusion method for visual-text interaction according to claim 1, characterized in that, The method of embedding visual features into the preprocessed raw video data based on the spatiotemporal graph convolution model to obtain visual feature representations includes: Motion and appearance features are extracted from pre-trained CNN models based on pre-processed raw video data; A visual feature enhancement module based on spatiotemporal gating unit is used to combine the motion features and appearance features to generate motion feature representations and appearance feature representations; Motion graphs and appearance graphs are constructed based on the motion feature representations and appearance feature representations, and visual feature representations are generated based on the motion graphs and appearance graphs.

3. The multimodal data fusion method for visual-text interaction according to claim 2, characterized in that, The construction of motion graphs and appearance graphs based on the motion feature representations and appearance feature representations includes: Link scores are defined for the motion features and appearance features, and the node connection relationships between the motion graph and appearance graph are determined based on the link scores. Calculate the first weight matrix representing the motion features and the second weight matrix representing the appearance features, and determine the connectivity based on the first weight matrix and the second weight matrix; Motion graphs and appearance graphs are constructed based on the node connection relationships and connectivity.

4. The multimodal data fusion method for visual-text interaction according to claim 3, characterized in that, The expression for the link score is: ， in, The link score between object i and object j. Let cosine similarity be the similarity between object i and object j. For the appearance characteristics of object i, , For frame length, For the appearance characteristics of object j, The scaling factor is IoU, where IoU is the intersection-union ratio. For the motion characteristics of object i, The motion characteristics of object j; The expression for the first weight matrix is: ， in, Let be the first weight matrix, and exp be the exponential function. Scaling factor Let i represent the appearance features of object i. L is the length of the clip, and K is the number of clips. , This represents the appearance characteristics of object j; The expression for the second weight matrix is: ， in, Let exp be the second weight matrix, and let exp be the exponential function. Scaling factor Let i be the motion feature representation of object i. L is the length of the clip, and K is the number of clips. , The motion characteristics of object j are represented; The expression for the connectivity is: ， in, For activation function, For connectivity, This is the first weight matrix. This is the second weight matrix. This is a linear transformation of the first weight matrix. This is a linear transformation of the second weight matrix. Let i represent the appearance features of object i. L is the length of the clip, and K is the number of clips. , The appearance features of object j are represented. Let i be the motion feature representation of object i. Let j be the motion characteristic representation of object j.

5. The multimodal data fusion method for visual text interaction according to claim 1, characterized in that, The BERT-based model performs hierarchical semantic aggregation on the preprocessed original text data to obtain text feature representations, including: Entity recognition technology is used to extract entities from preprocessed raw text data to obtain target text entities, and the target text entities are converted into a first embedding vector. Based on the BERT model, the first embedding vector is semantically encoded to obtain entity semantic information; The entity semantic information is decomposed into semantic units at different levels, including entity level, atomic action level, action level, and event level.

6. The multimodal data fusion method for visual-text interaction according to claim 5, characterized in that, The decomposition of the entity semantic information into semantic units at different levels includes: The entity semantic information is used as the entity level of the semantic unit; Based on syntactic analysis technology, verb phrase extraction is performed on the preprocessed raw text data to obtain target verb phrases, and the target verb phrases are used as atomic action representations; The atomic action representation is refined based on semantic role labeling technology to obtain a refined atomic action representation; The refined atomic action representation is converted into a second embedding vector. Based on the edge aggregation mechanism of the graph neural network, the second embedding vector is combined with the entity semantic information to integrate information and obtain a high-level atomic action representation. The high-level atomic action representation is used as the atomic action level. The atomic action level is captured by a long short-term network to obtain the target action sequence and temporal relationship, and the action level semantic representation is generated by using the target action sequence and temporal relationship based on an attention mechanism. Based on event extraction technology, an event semantic template is constructed using the action-level semantic representation, and an event-level embedding representation is generated using the event semantic template based on the hierarchical aggregation mechanism of graph neural networks.

7. The multimodal data fusion method for visual text interaction according to claim 1, characterized in that, The process of fusing the visual representation and text feature representation of the target to obtain a joint visual-text representation includes: The target visual representation and text feature representation are linearly transformed to obtain the linearly transformed target visual representation and text feature representation; The linearly transformed visual representation of the target and the text feature representation are concatenated to obtain the concatenated visual representation of the target and the text feature representation. Heterogeneous mapping is performed on the concatenated visual representation of the target and the text feature representation to obtain a joint visual-text representation. The expression for obtaining the joint visual-text mapping is as follows: ， ， ， Where y represents the concatenated visual representation of the target and the text feature representation. For activation function, For different mapping functions, For the visual representation of the target, Here, b represents the text features, and b is the bias term. is the heterogeneous mapping result, and s is the visual-text joint mapping.

8. A multimodal data fusion system for visual-text interaction, characterized in that, The system includes: Data acquisition and preprocessing module: used to acquire raw video data and raw text data, and preprocess the raw video data and raw text data to obtain preprocessed raw video data and raw text data; Visual feature representation acquisition module: used to embed visual features into the preprocessed raw video data based on the spatiotemporal graph convolution model to obtain visual feature representations; Text Feature Representation Acquisition Module: Used to perform hierarchical semantic aggregation processing on preprocessed raw text data based on the BERT model to obtain text feature representations; Target visual representation acquisition module: used to generate aggregated appearance node representation and aggregated motion node representation based on the interaction relationship matrix using the visual feature representation and text feature representation, and to generate target visual representation based on the aggregated appearance node representation and aggregated motion node representation; Visual-text joint representation acquisition module: used to fuse the target visual representation and text feature representation to obtain a visual-text joint representation, and to perform a visual-text interaction task based on the visual-text joint representation; The step of generating aggregated appearance node representations and aggregated motion node representations based on the interaction relationship matrix using the visual feature representations and text feature representations, and generating a target visual representation based on the aggregated appearance node representations and aggregated motion node representations, includes: generating an interaction relationship matrix using the softmax function based on the visual feature representations and text feature representations; generating aggregated appearance node representations and aggregated motion node representations based on the interaction relationship matrix using a fully connected layer and question nodes; performing continuous spatiotemporal graph convolution on the aggregated appearance node representations and aggregated motion node representations to obtain aggregated appearance node representations and aggregated motion node representations after continuous spatiotemporal graph convolution; performing average pooling on the aggregated appearance node representations and aggregated motion node representations after continuous spatiotemporal graph convolution to obtain vectorized representations of aggregated appearance nodes and aggregated motion nodes; concatenating the vectorized representations of aggregated appearance nodes and aggregated motion nodes to obtain an integrated visual representation, and generating a target visual representation based on the integrated visual representation using a heterogeneous Transformer encoder; The expression for the interaction relationship matrix is: ， in, This is an interaction relationship matrix. Scaling factor Represented by visual features, Text feature representation, L is the length of the clip, and K is the number of clips. ; The expression represented by the aggregated appearance node is: ， in, For activation function, This represents the aggregated appearance node. This is the weight matrix of the fully connected layer. Let K represent the appearance features, and K be the total number of problem nodes. Let be the interaction value in the interaction relationship matrix, where i is the index of the visual node and j is the index of the question node. Let b be the bias term, and let b be the problem graph representation. The expression representing the aggregated motion node is: ， in, For activation function, Represented as aggregated motion nodes. This is the weight matrix of the fully connected layer. Let K be the total number of problem nodes, representing the motion characteristics. Let be the interaction value in the interaction relationship matrix, where i is the index of the visual node and j is the index of the question node. Let b be the problem diagram, and b be the bias term.

Images