Visual text interaction-oriented multi-modal data fusion method and system

Abstract

The invention discloses a multi-modal data fusion method and system for visual text interaction, and the method comprises the steps: carrying out the preprocessing of original video data and original text data; performing visual feature embedding on the original video data based on a space-time diagram convolution model to obtain visual feature representation; performing hierarchical semantic aggregation processing on the original text data based on a BERT model to obtain text feature representation; generating an aggregated appearance node representation and an aggregated motion node representation using the visual feature representation and the textual feature representation based on the interaction relationship matrix to generate a target visual representation; and fusing the target visual representation and the text feature representation to obtain a visual text joint representation, and executing a visual text interaction task based on the visual text joint representation. According to the method, the problems of shallow semantic fusion level and simple visual text interaction in the existing multi-modal data fusion technology are solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal data fusion method and system for visual-text interaction. Background Art

[0002] In today's information age, data has become a key resource to promote scientific and technological progress and social development. With the rapid development of sensor technology, communication technology and computing power, the types and quantities of data that can be collected and processed have exploded. These data often span multiple sensory modalities such as vision, hearing, and touch, forming the so-called multimodal data. Multimodal data fusion refers to combining data from different modalities to obtain richer and more comprehensive information than single-modal data. Data from different modalities have different characteristics and expressions. The differences in format, scale and expression of these data make direct fusion difficult.

[0003] Although the currently used data fusion algorithms have achieved good results to a certain extent, they still have shortcomings. The traditional Transformer-based multimodal pre-training model ignores some high-level multimodal semantic features. Moreover, since the Transformer model only uses the features extracted from the top layer for prediction and recognition, it will ignore the details of other layers, making low-dimensional features unable to serve as the basis for model reasoning. The Transformer model needs to be superimposed with multiple layers of neural networks, but when the network is deep, not only will the underlying gradient disappear, but the position encoding will also be blurred, which will reduce the feature fusion effect between multimodal data and lose the generalization of the multimodal model. This requires a large amount of training data for such models, increasing the difficulty of training. As for the fusion strategy based on graph neural networks, due to the overly simple text and visual embedding modules, it cannot handle complex visual scenes and text inputs. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art. The present invention provides a multimodal data fusion method and system for visual-text interaction, which solves the problems of shallow semantic fusion level and simple visual-text interaction in the existing multimodal data fusion technology.

[0005] In order to solve the above technical problems, the present invention provides a multimodal data fusion method for visual text interaction, the method comprising:

[0006] Acquire original video data and original text data, and preprocess the original video data and original text data to obtain preprocessed original video data and original text data;

[0007] Based on the spatiotemporal graph convolution model, visual features are embedded into the preprocessed raw video data to obtain visual feature representation;

[0008] Based on the BERT model, the preprocessed raw text data is subjected to hierarchical semantic aggregation to obtain text feature representation;

[0009] Generate an aggregated appearance node representation and an aggregated motion node representation based on the interaction relationship matrix using the visual feature representation and the text feature representation, and generate a target visual representation based on the aggregated appearance node representation and the aggregated motion node representation;

[0010] The target visual representation and the text feature representation are fused to obtain a visual-text joint representation, and a visual-text interaction task is performed based on the visual-text joint representation.

[0011] Optionally, embedding visual features of the preprocessed original video data based on the spatiotemporal graph convolution model to obtain visual feature representation includes:

[0012] Extract motion features and appearance features from the preprocessed raw video data based on the pre-trained CNN model;

[0013] Using a visual feature enhancement module based on a spatiotemporal gating unit to combine the motion feature and the appearance feature to generate a motion feature representation and an appearance feature representation;

[0014] A motion map and an appearance map are constructed based on the motion feature representation and the appearance feature representation, and a visual feature representation is generated based on the motion map and the appearance map.

[0015] Optionally, constructing a motion graph and an appearance graph based on the motion feature representation and the appearance feature representation includes:

[0016] defining a link score for the motion features and the appearance features, and determining a node connection relationship between a motion graph and an appearance graph based on the link score;

[0017] Calculating a first weight matrix for the motion feature representation and a second weight matrix for the appearance feature representation, and determining connectivity based on the first weight matrix and the second weight matrix;

[0018] A motion graph and an appearance graph are constructed based on the node connection relationship and connectivity.

[0019] Optionally, the link score is expressed as:

[0020]

[0021] Among them, s i,j is the link score between object i and object j, ψ is the cosine similarity between object i and object j, is the appearance feature of object i, t∈{1,2,…,l v}, l v is the frame length, is the appearance feature of object j, λ is the scaling factor, IoU is the intersection over union ratio, is the motion feature of object i, is the motion feature of object j;

[0022] The expression of the first weight matrix is:

[0023]

[0024] in, is the first weight matrix, exp is the exponential function, λ is the scaling factor, is the appearance feature representation of object i, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1), is the appearance feature representation of object j;

[0025] The expression of the second weight matrix is:

[0026]

[0027] in, is the second weight matrix, exp is the exponential function, λ is the scaling factor, is the motion feature representation of object i, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1), is the motion feature representation of object j;

[0028] The expression of the connectivity is:

[0029]

[0030] Among them, σ is the activation function, R t For connectivity, is the first weight matrix, is the second weight matrix, is the linear transformation of the first weight matrix, is the linear transformation of the second weight matrix, is the appearance feature representation of object i, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1), is the appearance feature representation of object j, is the motion feature representation of object i, is the motion feature representation of object j.

[0031] Optionally, performing hierarchical semantic aggregation processing on the preprocessed original text data based on the BERT model to obtain text feature representation includes:

[0032] Perform entity extraction on the preprocessed original text data based on entity recognition technology to obtain a target text entity, and convert the target text entity into a first embedding vector;

[0033] Encoding semantic information of the first embedding vector based on the BERT model to obtain entity semantic information;

[0034] The entity semantic information is decomposed into semantic units at different levels, and the semantic units include entity level, atomic action level, action level and event level.

[0035] Optionally, the decomposing into semantic units of different levels based on the entity semantic information includes:

[0036] Taking the entity semantic information as the entity level of the semantic unit;

[0037] Based on the syntactic analysis technology, the preprocessed original text data is subjected to verb phrase extraction processing to obtain a target verb phrase, and the target verb phrase is used as an atomic action representation;

[0038] Refining the atomic action representation based on semantic role labeling technology to obtain a refined atomic action representation;

[0039] The refined atomic action representation is converted into a second embedding vector, and the second embedding vector is used in combination with the entity semantic information for information integration based on an edge aggregation mechanism of a graph neural network to obtain a high-level atomic action representation, and the high-level atomic action representation is used as the atomic action level;

[0040] Based on the long-short time network, the action sequence and the temporal relationship of the atomic action level are captured to obtain the target action sequence and the temporal relationship, and the target action sequence and the temporal relationship are used to generate the action-level semantic representation based on the attention mechanism;

[0041] Based on the event extraction technology, the action-level semantic representation is used to construct an event semantic template, and based on the hierarchical aggregation mechanism of the graph neural network, the event semantic template is used to generate an event-level embedded representation.

[0042] Optionally, the generating an aggregated appearance node representation and an aggregated motion node representation based on the interaction relationship matrix using the visual feature representation and the text feature representation, and generating a target visual representation based on the aggregated appearance node representation and the aggregated motion node representation, includes:

[0043] Generate an interaction relationship matrix based on the visual feature representation and the text feature representation using a softmax function;

[0044] Generate an aggregated appearance node representation and an aggregated motion node representation based on the interaction relationship matrix using a fully connected layer and a problem node;

[0045] Performing continuous spatiotemporal graph convolution on the aggregated appearance node representation and the aggregated motion node representation to obtain the aggregated appearance node representation and the aggregated motion node representation after continuous spatiotemporal graph convolution;

[0046] Based on the aggregated appearance node representation and the aggregated motion node representation after the continuous spatiotemporal graph convolution, average pooling is performed to obtain the aggregated appearance node vectorized representation and the aggregated motion node vectorized representation;

[0047] The aggregated appearance node vectorized representation and the aggregated motion node vectorized representation are concatenated to obtain an integrated visual representation, and the integrated visual representation is used to generate a target visual representation based on a heterogeneous Transformer encoder.

[0048] Optionally, the expression of the interaction relationship matrix is:

[0049]

[0050] Among them, S v is the interaction matrix, λ is the scaling factor, is the visual feature representation, is the text feature representation, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1);

[0051] The expression represented by the aggregate appearance node is:

[0052]

[0053] Among them, σ is the activation function, is the aggregate appearance node representation, is the weight matrix of the fully connected layer, is the appearance feature representation, K is the total number of problem nodes, is the interaction value in the interaction relationship matrix, i is the sequence number of the visual node, j is the sequence number of the problem node, and u j is the problem graph representation, b is the bias term;

[0054] The expression of the aggregate motion node is:

[0055]

[0056] Among them, σ is the activation function, is the aggregate motion node representation, is the weight matrix of the fully connected layer, is the motion feature representation, K is the total number of problem nodes, is the interaction value in the interaction relationship matrix, i is the sequence number of the visual node, j is the sequence number of the problem node, and u j is the problem graph representation, and b is the bias term.

[0057] Optionally, fusing the target visual representation and the text feature representation to obtain a visual-text joint representation includes:

[0058] Performing a linear change on the target visual representation and the text feature representation to obtain a linearly changed target visual representation and text feature representation;

[0059] splicing the linearly changed target visual representation and text feature representation to obtain a spliced ​​target visual representation and text feature representation;

[0060] The spliced ​​target visual representation and text feature representation are heterogeneously mapped to obtain a visual-text joint representation. The expression for obtaining the visual-text joint mapping is:

[0061]

[0062] y′=σ(W y y+b),

[0063] s=softmax(W y′ y′+b),

[0064] Among them, y is the concatenated target visual representation and text feature representation, σ is the activation function, W1, W2, W y , W y′ For different mapping functions, is the target visual representation, is the text feature representation, b is the bias term, y′ is the heterogeneous mapping result, and s is the visual-text joint mapping.

[0065] In addition, the present invention also provides a multimodal data fusion system for visual text interaction, the system comprising:

[0066] Data acquisition and preprocessing module: used to acquire original video data and original text data, and preprocess the original video data and original text data to obtain preprocessed original video data and original text data;

[0067] Visual feature representation acquisition module: used to embed visual features of pre-processed raw video data based on the spatiotemporal graph convolution model to obtain visual feature representation;

[0068] Text feature representation acquisition module: used to perform hierarchical semantic aggregation processing on the pre-processed raw text data based on the BERT model to obtain text feature representation;

[0069] A target visual representation acquisition module: used to generate an aggregated appearance node representation and an aggregated motion node representation based on the interaction relationship matrix using the visual feature representation and the text feature representation, and to generate a target visual representation based on the aggregated appearance node representation and the aggregated motion node representation;

[0070] Visual-text joint representation acquisition module: used to fuse the target visual representation and text feature representation to obtain a visual-text joint representation, and perform a visual-text interaction task based on the visual-text joint representation.

[0071] In an embodiment of the present invention, visual feature embedding is performed on the preprocessed raw video data based on the spatiotemporal graph convolution model to obtain a visual feature representation, thereby realizing the extraction and association of multi-level visual features. The spatiotemporal graph convolution model can robustly encode dynamic and static elements in the video, greatly enhancing the model's ability to handle complex video reasoning tasks. Based on the BERT model, hierarchical semantic aggregation is performed on the preprocessed raw text data to obtain a text feature representation, thereby realizing multi-level feature extraction of text data, thereby solving the problem of feature loss caused by long text input. Based on the interactive relationship matrix, the visual feature representation and the text feature representation are used to generate an aggregated appearance node representation and an aggregated motion node representation to generate a target visual representation, thereby effectively aggregating node information, integrating data from each node, and ensuring a comprehensive representation of the interaction within the network. The target visual representation and the text feature representation are fused to obtain a visual-text joint representation to perform visual-text interaction tasks, thereby realizing multi-level fusion of visual and semantic information, thereby solving the problems of shallow semantic fusion and simple visual-text interaction in existing multimodal data fusion technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 is a flow chart of a multimodal data fusion method for visual-text interaction in an embodiment of the present invention;

[0074] Figure 2 It is a schematic diagram of the structural composition of a multimodal data fusion system for visual-text interaction in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] Embodiment 1

[0077] See also Figure 1 , Figure 1 It is a flowchart of a multimodal data fusion method for visual-text interaction in an embodiment of the present invention.

[0078] like Figure 1 As shown, a multimodal data fusion method for visual text interaction, the method comprising:

[0079] S11: acquiring original video data and original text data, and preprocessing the original video data and original text data to obtain preprocessed original video data and original text data;

[0080] In the specific implementation process of the present invention, original video data and original text data are obtained, and the original text data may include text data through voice recognition. The original video data and original text data are preprocessed, and the original video data is denoised, that is, the noise in the video is removed from the original video data to obtain the original video data after denoising. The original text data is denoised. Text denoising is to clean the text data, remove web page tags, non-semantic characters and garbled characters, etc., to obtain the original text data after denoising. The original video data and original text data after denoising are de-redundant. Since the multimodal data contains data of some repeated dimensions, it is meaningless for the subsequent feature extraction and analysis. The redundant data of this part of the repeated dimensions is removed, and the multimodal data is divided into several data blocks, and the number of data blocks in the data blocks is calculated. The hash value of the data is calculated, and the code distance between the data is calculated according to the hash value. If the code distance is less than a preset threshold, it is redundant data. The redundant data is cleared to obtain the original video data and original text data after redundancy removal. The original video data and original text data after redundancy removal are normalized to eliminate the order of magnitude difference between each bit of the data to avoid excessive data level difference, which may cause excessive subsequent prediction errors. The normalized original video data and normalized original text data are obtained. The key frames of the normalized original video data are extracted to obtain the preprocessed original video data. After the original text data is normalized, the preprocessing is completed to obtain the preprocessed original text data.

[0081] S12: embedding visual features of the preprocessed raw video data based on the spatiotemporal graph convolution model to obtain visual feature representation;

[0082] In the specific implementation process of the present invention, the visual feature embedding of the preprocessed original video data based on the spatiotemporal graph convolution model is performed to obtain a visual feature representation, including: extracting motion features and appearance features of the preprocessed original video data based on a pretrained CNN model; using a visual feature enhancement module based on a spatiotemporal gating unit to combine the motion features and appearance features to generate a motion feature representation and an appearance feature representation; constructing a motion graph and an appearance graph based on the motion feature representation and the appearance feature representation, and generating a visual feature representation based on the motion graph and the appearance graph.

[0083] Furthermore, the construction of a motion graph and an appearance graph based on the motion feature representation and the appearance feature representation includes: defining link scores for the motion features and the appearance features, and determining the node connection relationship of the motion graph and the appearance graph based on the link scores; calculating a first weight matrix for the motion feature representation and a second weight matrix for the appearance feature representation, and determining connectivity based on the first weight matrix and the second weight matrix; and constructing a motion graph and an appearance graph based on the node connection relationship and connectivity.

[0084] Furthermore, the expression of the link score is:

[0085]

[0086] Among them, s i,j is the link score between object i and object j, ψ is the cosine similarity between object i and object j, is the appearance feature of object i, t∈{1,2,…,l v}, l v is the frame length, is the appearance feature of object j, λ is the scaling factor, IoU is the intersection over union ratio, is the motion feature of object i, is the motion feature of object j;

[0087] The expression of the first weight matrix is:

[0088]

[0089] in, is the first weight matrix, exp is the exponential function, λ is the scaling factor, is the appearance feature representation of object i, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1), is the appearance feature representation of object j;

[0090] The expression of the second weight matrix is:

[0091]

[0092] in, is the second weight matrix, exp is the exponential function, λ is the scaling factor, is the motion feature representation of object i, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1), is the motion feature representation of object j;

[0093] The expression of the connectivity is:

[0094]

[0095] Among them, σ is the activation function, R t For connectivity, is the first weight matrix, is the second weight matrix, is the linear transformation of the first weight matrix, is the linear transformation of the second weight matrix, is the appearance feature representation of object i, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1), is the appearance feature representation of object j, is the motion feature representation of object i, is the motion feature representation of object j.

[0096] Specifically, the construction of node representation of visual features is a key step of the visual feature embedding module based on graph neural network, which involves converting the visual information in the image or video into nodes in the graph structure for further analysis and processing. The preprocessed raw video data is extracted according to the preset speed, and the preprocessed raw video data is divided into several clips of preset length. For each clip, a dense stream of several frames is maintained to obtain clip-level motion features, and a sparse stream of several frames is maintained to obtain regional and frame appearance features, that is, the motion features and appearance features are extracted from the preprocessed raw video data based on the pretrained Convolutional Neural Network (CNN) model, especially the 3D version ResNeXt-101 model for motion and the ResNet-101 model for frame appearance. The motion feature representation and appearance feature representation are generated by combining the motion feature and appearance feature with the visual feature enhancement module based on the spatiotemporal gating unit, extracting several appearance features and their bounding boxes from each frame of the sparse stream, mapping the appearance feature and the motion feature to a feature space of a preset dimension, and forming a feature representation output through the visual feature enhancement module of the spatiotemporal gating unit, that is, obtaining the motion feature representation and the appearance feature representation. In order to find the same object between different frames in a clip, a link score is defined by considering their appearance and spatial position, that is, defining a link score for the motion feature and the appearance feature, and the expression of the link score is:

[0097]

[0098] Among them, s i,j is the link score between object i and object j, ψ is the cosine similarity between object i and object j, is the appearance feature of object i, t∈{1,2,…,l v}, l v is the frame length, is the appearance feature of object j, λ is the scaling factor, IoU is the intersection over union ratio, is the motion feature of object i, is the motion feature of object j, object i and object j are two objects detected in adjacent frames, where λ can be set to 1. Several detected objects in the first frame of each segment are designated as anchor objects, and then the detected objects in consecutive frames are connected to the anchor objects by greedily maximizing the link score frame by frame. By aligning the objects within the segment, we ensure the consistency of the representation of the nodes and edges of the graphs constructed in different frames. In the graph representation used in the model, each node is closely connected to every other node through edges. These edges represent the relationship between the nodes and are quantized using a vector that encapsulates the relevant features of each node, that is, the node connection relationship of the motion graph and the appearance graph is determined based on the link score. Calculate the first weight matrix of the motion feature representation and the second weight matrix of the appearance feature representation. The weight matrix represents the connection between the nodes and their corresponding edge weights, which can be calculated based on the affinity between the motion feature representation and the appearance feature representation. The connectivity is determined based on the first weight matrix and the second weight matrix. The expression of the first weight matrix is:

[0099]

[0100] in, is the first weight matrix, exp is the exponential function, λ is the scaling factor, is the appearance feature representation of object i, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1), is the appearance feature representation of object j;

[0101] The expression of the second weight matrix is:

[0102]

[0103] in, is the second weight matrix, exp is the exponential function, λ is the scaling factor, is the motion feature representation of object i, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1), is the motion feature representation of object j;

[0104] The expression of the connectivity is:

[0105]

[0106] Among them, σ is the activation function, R t For connectivity, is the first weight matrix, is the second weight matrix, is the linear transformation of the first weight matrix, is the linear transformation of the second weight matrix, is the appearance feature representation of object i, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1), is the appearance feature representation of object j, is the motion feature representation of object i, is the motion feature representation of object j;

[0107] The linear transformation adopts different weighting mechanisms. The linear transformation of the first weight matrix and the linear transformation of the second weight matrix are different ways of change to reflect the asymmetric nature of subjective and objective interactions in the real world. Visual feature representation is generated based on the motion map and appearance map. The motion map and appearance map are merged through the matrix. This process integrates different graph structures into a unified matrix. Subsequently, this concatenated matrix is ​​processed by the self-attention layer to enhance the spatiotemporal interwoven representation. The self-attention mechanism plays a key role in highlighting the important features and interactions of the appearance and motion dimensions. In this way, the model can capture a more comprehensive and detailed understanding of the video content, thereby gaining a deep insight into the dynamic interactions shown in the video.

[0108] S13: Perform hierarchical semantic aggregation on the preprocessed raw text data based on the BERT model to obtain text feature representation;

[0109] In the specific implementation process of the present invention, the pre-processed original text data is subjected to hierarchical semantic aggregation processing based on the BERT model to obtain text feature representation, including: performing entity extraction on the pre-processed original text data based on entity recognition technology to obtain target text entities, and converting the target text entities into a first embedding vector; encoding semantic information of the first embedding vector based on the BERT model to obtain entity semantic information; decomposing into semantic units of different levels based on the entity semantic information, and the semantic units include entity level, atomic action level, action level and event level.

[0110] Furthermore, the decomposition into semantic units of different levels based on the entity semantic information includes: taking the entity semantic information as the entity level of the semantic unit; performing verb phrase extraction processing on the preprocessed original text data based on syntactic analysis technology to obtain the target verb phrase, and taking the target verb phrase as the atomic action representation; refining the atomic action representation based on semantic role labeling technology to obtain the refined atomic action representation; converting the refined atomic action representation into a second embedding vector, using the second embedding vector based on the edge aggregation mechanism of the graph neural network to combine the entity semantic information for information integration to obtain a high-level atomic action representation, and taking the high-level atomic action representation as the atomic action level; capturing the action sequence and temporal relationship of the atomic action level based on the long-short time network to obtain the target action sequence and temporal relationship, and generating the action level semantic representation based on the attention mechanism using the target action sequence and temporal relationship; constructing an event semantic template based on the action level semantic representation based on the event extraction technology, and generating an event level embedding representation based on the hierarchical aggregation mechanism of the graph neural network using the event semantic template.

[0111] Specifically, the text embedding module based on hierarchical semantic aggregation is a key technology in multimodal data fusion. Its core lies in hierarchical processing of text features to achieve refined semantic representation and deep information fusion. The module decomposes the text content into semantic units at different levels, including entity level, atomic action level, action level and event level. At the entity level, the module first identifies named entities in the text, such as names of people, places, and institutions. These entities, as basic semantic units of the text, are crucial for understanding the text content. Based on entity recognition technology, the pre-processed raw text data is subjected to entity extraction. Entity recognition technology such as conditional random fields or named entity recognition models based on deep learning are used to obtain target text entities, and the target text entities are converted into a first embedding vector. Semantic information is encoded on the first embedding vector based on the bidirectional encoder representation transformer (Bidirectional Encoder Representations from Transformers, BERT) model, and the BERT model is used to construct a word embedding model. The semantic information of the entity is further encoded through the word embedding model, laying the foundation for subsequent semantic aggregation. At the same time, a word representation model based on global word frequency statistics can also be used to construct a word embedding model to obtain entity semantic information, that is, to obtain the entity level of the semantic unit. Atomic action level features involve basic action or event units in the text, which are usually composed of verbs and their related objects. The preprocessed raw text data is subjected to verb phrase extraction based on syntactic analysis technology. Syntactic analysis technology such as dependency syntactic analysis is used to obtain the target verb phrase, and the target verb phrase is used as the atomic action representation. The atomic action representation is refined based on semantic role labeling technology to obtain a refined atomic action representation. The role of refinement is to clarify the role relationship of the action agent, the object, etc. The refined atomic action representation is converted into a second embedding vector. The edge aggregation mechanism of the graph neural network uses the second embedding vector combined with the entity semantic information to perform information integration, integrate the semantic information of the action and its participants, and form a high-level representation of the atomic action, that is, to obtain a high-level atomic action representation, and the high-level atomic action representation is used as the atomic action level. The action level feature is a further abstraction and generalization of the atomic action level feature, which involves combining multiple related atomic actions into a more complex action sequence or pattern. Based on the long-short time network, the action sequence and temporal relationship of the atomic action level are captured to obtain the target action sequence and temporal relationship, and the target action sequence and temporal relationship are used to generate action-level semantic representation based on the attention mechanism, and the target action sequence and temporal relationship are encoded into a higher-dimensional embedding vector. The key actions and transitions in the target action sequence and temporal relationship are emphasized through the attention mechanism, thereby forming an action-level semantic representation, that is, obtaining the action level of the semantic unit. The event-level feature is the highest level in the text embedding module, which integrates the relevant action-level features into a complete event framework.The action-level semantic representation is used to construct an event semantic template based on event extraction technology. Event extraction technology such as template matching or event extraction model based on graph neural network identifies key events, and constructs event semantic templates through the identified key events. These event semantic templates include attributes such as event type, participants, time and place, and generate event-level embedded representation based on the hierarchical aggregation mechanism of graph neural network using the event semantic template. The semantic information from different levels is integrated through the hierarchical aggregation mechanism of graph neural network to form an event-level embedded representation.

[0112] S14: generating an aggregated appearance node representation and an aggregated motion node representation based on the interaction relationship matrix using the visual feature representation and the text feature representation, and generating a target visual representation based on the aggregated appearance node representation and the aggregated motion node representation;

[0113] In the specific implementation process of the present invention, the method of generating an aggregated appearance node representation and an aggregated motion node representation based on the visual feature representation and the text feature representation based on the interaction relationship matrix, and generating a target visual representation based on the aggregated appearance node representation and the aggregated motion node representation, includes: generating an interaction relationship matrix based on the visual feature representation and the text feature representation using a softmax function; generating an aggregated appearance node representation and an aggregated motion node representation based on the interaction relationship matrix using a fully connected layer and a problem node; performing continuous spatiotemporal graph convolution on the aggregated appearance node representation and the aggregated motion node representation to obtain an aggregated appearance node representation and an aggregated motion node representation after continuous spatiotemporal graph convolution; performing average pooling on the aggregated appearance node representation and the aggregated motion node representation after continuous spatiotemporal graph convolution to obtain an aggregated appearance node vectorized representation and an aggregated motion node vectorized representation; concatenating the aggregated appearance node vectorized representation and the aggregated motion node vectorized representation to obtain an integrated visual representation, and generating a target visual representation based on a heterogeneous Transformer encoder using the integrated visual representation.

[0114] Furthermore, the expression of the interaction relationship matrix is:

[0115]

[0116] Among them, S v is the interaction matrix, λ is the scaling factor, is the visual feature representation, is the text feature representation, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1);

[0117] The expression represented by the aggregate appearance node is:

[0118]

[0119] Among them, σ is the activation function, is the aggregate appearance node representation, is the weight matrix of the fully connected layer, is the appearance feature representation, K is the total number of problem nodes, is the interaction value in the interaction relationship matrix, i is the sequence number of the visual node, j is the sequence number of the problem node, and u j is the problem graph representation, b is the bias term;

[0120] The expression of the aggregate motion node is:

[0121]

[0122] Among them, σ is the activation function, is the aggregate motion node representation, is the weight matrix of the fully connected layer, is the motion feature representation, K is the total number of problem nodes, is the interaction value in the interaction relationship matrix, i is the sequence number of the visual node, j is the sequence number of the problem node, and u j is the problem graph representation, and b is the bias term.

[0123] Specifically, the goal of visual-to-text interaction is to learn visual representations under question conditions. This process involves associating question nodes with corresponding visual nodes and propagating question representations along visual edges through spatial-temporal graph convolutional layers. We adopted a dynamic graph Transformer method to effectively aggregate node information. This method uses a multi-head attention mechanism to integrate data from each node to ensure a comprehensive representation of interactions within the network. In addition, the edges also aggregate information in this way and are processed using the edge Transformer method. Based on the visual feature representation and the text feature representation, the softmax function is used to generate an interaction relationship matrix. The softmax function is applied to the affinity matrix between the visual feature representation and the text feature representation on the question edge to generate an interaction relationship matrix. The expression of the interaction relationship matrix is:

[0124]

[0125] Among them, S v is the interaction matrix, λ is the scaling factor, is the visual feature representation, is the text feature representation, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1). Based on the interaction relationship matrix, the fully connected layer and the problem node are used to generate the aggregated appearance node representation and the aggregated motion node representation. The scaling factor is used to adjust the influence of the affinity value. Each interaction value in the interaction relationship matrix quantifies the degree of association between the visual node and the problem node. All problem nodes are aggregated to the corresponding appearance nodes through the interaction relationship matrix, and the generated vector is formed into an aggregated appearance node representation through the fully connected layer. Similarly, the motion feature can also form an aggregated motion node representation. The expression of the aggregated appearance node representation is:

[0126]

[0127] Among them, σ is the activation function, is the aggregate appearance node representation, is the weight matrix of the fully connected layer, is the appearance feature representation, K is the total number of problem nodes, is the interaction value in the interaction relationship matrix, i is the sequence number of the visual node, j is the sequence number of the problem node, and u j is the problem graph representation, b is the bias term;

[0128] The expression of the aggregate motion node is:

[0129]

[0130] Among them, σ is the activation function, is the aggregate motion node representation, is the weight matrix of the fully connected layer, is the motion feature representation, K is the total number of problem nodes, is the interaction value in the interaction relationship matrix, i is the sequence number of the visual node, j is the sequence number of the problem node, and u j is the problem graph representation, and b is the bias term. The aggregated appearance node representation and the aggregated motion node representation are subjected to continuous spatiotemporal graph convolution to obtain the aggregated appearance node representation and the aggregated motion node representation after continuous spatiotemporal graph convolution. The expressions of the aggregated appearance node representation and the aggregated motion node representation after continuous spatiotemporal graph convolution are:

[0131]

[0132] in, is the aggregated appearance node representation after the convolution of the continuous spatiotemporal graph, is the aggregated motion node representation after the convolution of the continuous spatiotemporal graph, and is the parameter of the spatiotemporal graph convolutional layer, F represents the feedforward processing of the spatiotemporal graph convolutional layer, and Wv is the first weight matrix, W m is the second weight matrix, is the aggregate appearance node representation, is the aggregated motion node representation. Based on the aggregated appearance node representation and the aggregated motion node representation after the continuous spatiotemporal graph convolution, average pooling is performed on the aggregated appearance node representation and the aggregated motion node representation after the continuous spatiotemporal graph convolution along the time axis to effectively vectorize these representations to obtain the aggregated appearance node vectorized representation and the aggregated motion node vectorized representation. This vectorization process simplifies the complex spatiotemporal data and makes it easier to handle. The aggregated appearance node vectorized representation and the aggregated motion node vectorized representation are spliced ​​to obtain an integrated visual representation. The expression of the integrated visual representation is:

[0133]

[0134] Where O is the integrated visual representation, is the vectorized representation of the aggregated appearance nodes. The motion nodes are aggregated into vectorized representations, and the target visual representation is generated based on the heterogeneous Transformer encoder using the integrated visual representation, and the integrated visual representation is transmitted through the heterogeneous Transformer encoder to generate a final visual representation, that is, to obtain the target visual representation.

[0135] S15: Fusing the target visual representation and the text feature representation to obtain a visual-text joint representation, and performing a visual-text interaction task based on the visual-text joint representation.

[0136] In the specific implementation process of the present invention, the fusion of the target visual representation and the text feature representation to obtain a visual-text joint representation includes: linearly changing the target visual representation and the text feature representation to obtain the linearly changed target visual representation and the text feature representation; splicing the linearly changed target visual representation and the text feature representation to obtain the spliced ​​target visual representation and the text feature representation; performing heterogeneous mapping on the spliced ​​target visual representation and the text feature representation to obtain a visual-text joint representation, and the expression for obtaining the visual-text joint mapping is:

[0137]

[0138] y′=σ(W y y+b),

[0139] s=softmax(W y′ y′+b),

[0140] Among them, y is the concatenated target visual representation and text feature representation, σ is the activation function, W1, W2, W y , W y′ For different mapping functions, is the target visual representation, is the text feature representation, b is the bias term, y′ is the heterogeneous mapping result, and s is the visual-text joint mapping.

[0141] Specifically, the target visual representation and the text feature representation are linearly changed to obtain the target visual representation and the text feature representation after the linear change; the target visual representation and the text feature representation after the linear change are spliced ​​to obtain the spliced ​​target visual representation and the text feature representation; the spliced ​​target visual representation and the text feature representation are heterogeneously mapped to obtain the visual text joint representation, and the expression of the acquisition process of the visual text joint mapping is:

[0142]

[0143] y′=σ(W y y+b),

[0144] s=softmax(W y′ y′+b),

[0145] Among them, y is the concatenated target visual representation and text feature representation, σ is the activation function, W1, W2, W y , W y′ For different mapping functions, is the target visual representation, is the text feature representation, b is the bias term, y′ is the heterogeneous mapping result, s is the visual text joint mapping, the target visual representation can be the visual features at the object level, frame level, and clip level, respectively, and the text feature representation can be the text features at the word level, sentence level, and paragraph level. For the output visual question joint representation, we found that it can fuse the visual and text features at different levels, effectively solving the difficulty of multi-granular fusion representation in multimodal information fusion. For the fused visual text features, we apply them to the visual text interaction task. The process of the visual text interaction task may include using the BERT model's ability to encode context bidirectionally, establishing contextual dependencies within the text, and then using CoreNLP to identify the semantic dependencies between questions and candidate answers. CoreNLP is a set of natural language processing tools developed by the Natural Language Processing Group of Stanford University to help users analyze and understand text. This analysis involves parsing the parts of speech of various types of words in the sentence. Based on this language analysis, each word is assigned as a node, and a graphical representation of the question is constructed. The weight is based on the part-of-speech assignment, and the similarity matrix of the question representation reflecting these relationships can be described as follows:

[0146]

[0147] Among them, e ij represents the affinity between the i-th and j-th problem nodes, λ is the scaling factor, exp is the exponential function, K is the total number of problem nodes, is the text feature representation of the j-th question node, is the text feature representation of the i-th question node, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1), and λ is used as a scaling factor to adjust the influence between question nodes. The weight matrix is ​​then calculated in conjunction with the adjacency matrix, which represents the connectivity of the question graph, and then the question and candidate answer are asked. This comprehensive integration process ultimately produces a fully informed answer prediction. By effectively integrating insights from the visual and textual fields, the model can effectively answer the questions raised, demonstrating robust cross-modal reasoning capabilities.

[0148] In an embodiment of the present invention, visual feature embedding is performed on the preprocessed raw video data based on the spatiotemporal graph convolution model to obtain a visual feature representation, thereby realizing the extraction and association of multi-level visual features. The spatiotemporal graph convolution model can robustly encode dynamic and static elements in the video, greatly enhancing the model's ability to handle complex video reasoning tasks. Based on the BERT model, hierarchical semantic aggregation is performed on the preprocessed raw text data to obtain a text feature representation, thereby realizing multi-level feature extraction of text data, thereby solving the problem of feature loss caused by long text input. Based on the interactive relationship matrix, the visual feature representation and the text feature representation are used to generate an aggregated appearance node representation and an aggregated motion node representation to generate a target visual representation, thereby effectively aggregating node information, integrating data from each node, and ensuring a comprehensive representation of the interaction within the network. The target visual representation and the text feature representation are fused to obtain a visual-text joint representation to perform visual-text interaction tasks, thereby realizing multi-level fusion of visual and semantic information, thereby solving the problems of shallow semantic fusion and simple visual-text interaction in existing multimodal data fusion technologies.

[0149] Embodiment 2

[0150] See also Figure 2 , Figure 2 It is a schematic diagram of the structural composition of a multimodal data fusion system for visual-text interaction in an embodiment of the present invention.

[0151] like Figure 2 As shown, a multimodal data fusion system for visual text interaction, the system includes:

[0152] Data acquisition and preprocessing module 21: used to acquire original video data and original text data, and preprocess the original video data and original text data to obtain preprocessed original video data and original text data;

[0153] Visual feature representation acquisition module 22: used to embed visual features of the pre-processed original video data based on the spatiotemporal graph convolution model to obtain visual feature representation;

[0154] The text feature representation acquisition module 23 is used to perform hierarchical semantic aggregation processing on the pre-processed original text data based on the BERT model to obtain text feature representation;

[0155] A target visual representation acquisition module 24 is configured to generate an aggregate appearance node representation and an aggregate motion node representation based on the interaction relationship matrix using the visual feature representation and the text feature representation, and to generate a target visual representation based on the aggregate appearance node representation and the aggregate motion node representation;

[0156] The visual-text joint representation acquisition module 25 is used to fuse the target visual representation and the text feature representation to obtain the visual-text joint representation, and perform the visual-text interaction task based on the visual-text joint representation.

[0157] In the specific implementation process of the present invention, the specific implementation method of the system item can refer to the implementation method of the above-mentioned method item, which will not be repeated here.

[0158] In an embodiment of the present invention, visual feature embedding is performed on the preprocessed raw video data based on the spatiotemporal graph convolution model to obtain a visual feature representation, thereby realizing the extraction and association of multi-level visual features. The spatiotemporal graph convolution model can robustly encode dynamic and static elements in the video, greatly enhancing the model's ability to handle complex video reasoning tasks. Based on the BERT model, hierarchical semantic aggregation is performed on the preprocessed raw text data to obtain a text feature representation, thereby realizing multi-level feature extraction of text data, thereby solving the problem of feature loss caused by long text input. Based on the interactive relationship matrix, the visual feature representation and the text feature representation are used to generate an aggregated appearance node representation and an aggregated motion node representation to generate a target visual representation, thereby effectively aggregating node information, integrating data from each node, and ensuring a comprehensive representation of the interaction within the network. The target visual representation and the text feature representation are fused to obtain a visual-text joint representation to perform visual-text interaction tasks, thereby realizing multi-level fusion of visual and semantic information, thereby solving the problems of shallow semantic fusion and simple visual-text interaction in existing multimodal data fusion technologies.

[0159] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0160] In addition, the above is a detailed introduction to a multimodal data fusion method and system for visual text interaction provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A multimodal data fusion method for visual-text interaction, characterized in that: The method comprises: Acquire original video data and original text data, and preprocess the original video data and original text data to obtain preprocessed original video data and original text data; Based on the spatiotemporal graph convolution model, visual feature embedding is performed on the preprocessed raw video data to obtain visual feature representation; Based on the BERT model, the preprocessed raw text data is subjected to hierarchical semantic aggregation to obtain text feature representation; Generate an aggregated appearance node representation and an aggregated motion node representation based on the interaction relationship matrix using the visual feature representation and the text feature representation, and generate a target visual representation based on the aggregated appearance node representation and the aggregated motion node representation; The target visual representation and the text feature representation are fused to obtain a visual-text joint representation, and a visual-text interaction task is performed based on the visual-text joint representation.

2. The multimodal data fusion method for visual-text interaction according to claim 1 is characterized in that: The method of embedding visual features of the preprocessed original video data based on the spatiotemporal graph convolution model to obtain visual feature representation includes: Extract motion features and appearance features from the preprocessed raw video data based on the pre-trained CNN model; Using a visual feature enhancement module based on a spatiotemporal gating unit to combine the motion feature and the appearance feature to generate a motion feature representation and an appearance feature representation; A motion map and an appearance map are constructed based on the motion feature representation and the appearance feature representation, and a visual feature representation is generated based on the motion map and the appearance map.

3. The multimodal data fusion method for visual-text interaction according to claim 2 is characterized in that: The step of constructing a motion graph and an appearance graph based on the motion feature representation and the appearance feature representation comprises: defining a link score for the motion features and the appearance features, and determining a node connection relationship between a motion graph and an appearance graph based on the link score; Calculating a first weight matrix for the motion feature representation and a second weight matrix for the appearance feature representation, and determining connectivity based on the first weight matrix and the second weight matrix; A motion graph and an appearance graph are constructed based on the node connection relationship and connectivity.

4. The multimodal data fusion method for visual-text interaction according to claim 3 is characterized in that: The expression of the link score is: Among them, s i, is the link score between object i and object j, ψ is the cosine similarity between object i and object j, is the appearance feature of object i, t∈{1,2,…,l v }, l v is the frame length, is the appearance feature of object j, λ is the scaling factor, IoU is the intersection over union ratio, is the motion feature of object i, is the motion feature of object j; The expression of the first weight matrix is: in, is the first weight matrix, exp is the exponential function, λ is the scaling factor, is the appearance feature representation of object i, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1), is the appearance feature representation of object j; The expression of the second weight matrix is: in, is the second weight matrix, exp is the exponential function, λ is the scaling factor, is the motion feature representation of object i, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1), is the motion feature representation of object j; The expression of the connectivity is: Among them, σ is the activation function, R t For connectivity, is the first weight matrix, is the second weight matrix, is the linear transformation of the first weight matrix, is the linear transformation of the second weight matrix, is the appearance feature representation of object i, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1), is the appearance feature representation of object j, is the motion feature representation of object i, is the motion feature representation of object j.

5. The multimodal data fusion method for visual-text interaction according to claim 1 is characterized in that: The BERT model is used to perform hierarchical semantic aggregation processing on the preprocessed original text data to obtain text feature representation, including: Perform entity extraction on the preprocessed original text data based on entity recognition technology to obtain a target text entity, and convert the target text entity into a first embedding vector; Based on the BERT model, semantic information is encoded on the first embedding vector to obtain entity semantic information; The entity semantic information is decomposed into semantic units at different levels, and the semantic units include entity level, atomic action level, action level and event level.

6. The multimodal data fusion method for visual-text interaction according to claim 5 is characterized in that: The decomposition into semantic units of different levels based on the entity semantic information includes: Taking the entity semantic information as the entity level of the semantic unit; Based on the syntactic analysis technology, the preprocessed original text data is subjected to verb phrase extraction processing to obtain a target verb phrase, and the target verb phrase is used as an atomic action representation; Refining the atomic action representation based on semantic role labeling technology to obtain a refined atomic action representation; The refined atomic action representation is converted into a second embedding vector, and the second embedding vector is used in combination with the entity semantic information for information integration based on an edge aggregation mechanism of a graph neural network to obtain a high-level atomic action representation, and the high-level atomic action representation is used as the atomic action level; Based on the long-short time network, the action sequence and the temporal relationship of the atomic action level are captured to obtain the target action sequence and the temporal relationship, and the target action sequence and the temporal relationship are used to generate the action-level semantic representation based on the attention mechanism; Based on the event extraction technology, the action-level semantic representation is used to construct an event semantic template, and based on the hierarchical aggregation mechanism of the graph neural network, the event semantic template is used to generate an event-level embedded representation.

7. The multimodal data fusion method for visual-text interaction according to claim 1 is characterized in that: The generating of an aggregated appearance node representation and an aggregated motion node representation based on the interaction relationship matrix using the visual feature representation and the text feature representation, and generating a target visual representation based on the aggregated appearance node representation and the aggregated motion node representation, comprises: Generate an interaction relationship matrix based on the visual feature representation and the text feature representation using a softmax function; Generate an aggregated appearance node representation and an aggregated motion node representation based on the interaction relationship matrix using a fully connected layer and a problem node; Performing continuous spatiotemporal graph convolution on the aggregated appearance node representation and the aggregated motion node representation to obtain the aggregated appearance node representation and the aggregated motion node representation after continuous spatiotemporal graph convolution; Based on the aggregated appearance node representation and the aggregated motion node representation after the continuous spatiotemporal graph convolution, average pooling is performed to obtain the aggregated appearance node vectorized representation and the aggregated motion node vectorized representation; The aggregated appearance node vectorized representation and the aggregated motion node vectorized representation are concatenated to obtain an integrated visual representation, and the integrated visual representation is used to generate a target visual representation based on a heterogeneous Transformer encoder.

8. The multimodal data fusion method for visual-text interaction according to claim 7 is characterized in that: The expression of the interaction relationship matrix is: Among them, S v is the interaction matrix, λ is the scaling factor, is the visual feature representation, is the text feature representation, T = γLK, L is the length of the clip, K is the number of clips, γ∈(0,1); The expression represented by the aggregate appearance node is: Among them, σ is the activation function, is the aggregate appearance node representation, is the weight matrix of the fully connected layer, is the appearance feature representation, K is the total number of problem nodes, is the interaction value in the interaction relationship matrix, i is the sequence number of the visual node, j is the sequence number of the problem node, and u j is the problem graph representation, b is the bias term; The expression of the aggregate motion node is: Among them, σ is the activation function, is the aggregate motion node representation, is the weight matrix of the fully connected layer, is the motion feature representation, K is the total number of problem nodes, is the interaction value in the interaction relationship matrix, i is the sequence number of the visual node, j is the sequence number of the problem node, and u j is the problem graph representation, and b is the bias term.

9. The multimodal data fusion method for visual-text interaction according to claim 1, characterized in that: The step of fusing the target visual representation and the text feature representation to obtain a visual-text joint representation includes: Performing a linear change on the target visual representation and the text feature representation to obtain a linearly changed target visual representation and text feature representation; splicing the linearly changed target visual representation and text feature representation to obtain a spliced ​​target visual representation and text feature representation; The spliced ​​target visual representation and text feature representation are heterogeneously mapped to obtain a visual-text joint representation. The expression for obtaining the visual-text joint mapping is: y′=σ(W y y+b), s=softmax(W y′ y′+b), Among them, y is the concatenated target visual representation and text feature representation, σ is the activation function, W1, W2, W y , W y′ For different mapping functions, is the target visual representation, is the text feature representation, b is the bias term, y′ is the heterogeneous mapping result, and s is the visual-text joint mapping.

10. A multimodal data fusion system for visual-text interaction, characterized in that: The system comprises: Data acquisition and preprocessing module: used to acquire original video data and original text data, and preprocess the original video data and original text data to obtain preprocessed original video data and original text data; Visual feature representation acquisition module: used to embed visual features of pre-processed raw video data based on the spatiotemporal graph convolution model to obtain visual feature representation; Text feature representation acquisition module: used to perform hierarchical semantic aggregation processing on the pre-processed raw text data based on the BERT model to obtain text feature representation; A target visual representation acquisition module: used to generate an aggregated appearance node representation and an aggregated motion node representation based on the interaction relationship matrix using the visual feature representation and the text feature representation, and to generate a target visual representation based on the aggregated appearance node representation and the aggregated motion node representation; Visual-text joint representation acquisition module: used to fuse the target visual representation and text feature representation to obtain a visual-text joint representation, and perform a visual-text interaction task based on the visual-text joint representation.

Images