A decoupling method, representation method and representation system for cross-modal data

Abstract

The present invention relates to the field of cross-modal data processing and graph convolution technology, and discloses a decoupling method, a characterization method and a characterization system for cross-modal data. The decoupling method comprises the following steps: S1, feature extraction: extracting the features of each modal data and synchronizing the features of different modal data in the time dimension; S2, feature decoupling: constructing a shared mapping module based on an attention mechanism, performing feature decoupling of different modal data, and outputting decoupled modal features. The present invention solves the problems existing in the prior art that it is difficult to accurately understand and process cross-modal information.

Description

Technical Field

[0001] The present invention relates to the field of cross-modal data processing and graph convolution technology, and specifically to a decoupling method, a characterization method and a characterization system for cross-modal data. Background Art

[0002] With the rapid development of information technology, the generation and dissemination of a large amount of cross-modal data on social media platforms has forced us to have more efficient cross-modal information processing capabilities. The cross-modal characteristics of information, including text, images, videos and other forms, make the decoupling and representation of cross-modal information very complicated. In this process, how to effectively decouple cross-modal data and clearly characterize the relationship between various modalities has become the key to understanding and managing cyberspace content. Especially under the high degree of interweaving of information dissemination, accurately separating the independent features of each modality and integrating them in a reasonable way can significantly improve the understanding of cross-modal content.

[0003] Traditional cross-modal information processing technology has limitations in the two core links of cross-modal data decoupling and representation. Most traditional methods rely on simple inter-modal fusion, ignoring the inherent differences between modalities, resulting in serious data interweaving between different modalities. For example, in terms of cross-modal decoupling, traditional technologies have failed to deeply explore how to separate effective independent features from cross-modal data, resulting in the failure to effectively eliminate the interference between different modalities, affecting the understanding and processing of cross-modal information. On the other hand, when representing cross-modal information, traditional methods fail to fully consider the deep fusion of network graph structural features and cross-modal content features, ignoring the complexity of social networks and communication behaviors, causing the model to perform insufficiently in the face of complex scenarios and unable to effectively capture the diversity and complexity of cross-modal information dissemination. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the present invention provides a decoupling method, a characterization method and a characterization system for cross-modal data, so as to solve the problems existing in the prior art such as difficulty in accurately understanding and processing cross-modal information.

[0005] The technical solution adopted by the present invention to solve the above problems is:

[0006] A method for decoupling cross-modal data includes the following steps:

[0007] S1, feature extraction: extract the features of each modality data and synchronize the features of different modal data in the time dimension;

[0008] S2, feature decoupling: Based on the attention mechanism, a shared mapping module is built to decouple the features of different modal data and output the decoupled modal features.

[0009] As a preferred technical solution, in step S1, when extracting features of modal data, if the modal data includes video data, the image and video frames are segmented into a series of non-overlapping visual markers, and then a video encoder is used in combination with local time modeling and self-attention mechanism to extract time series features from the video data, and functional modular modeling is performed. The formula of functional modular modeling is expressed as follows:

[0010]

[0011]

[0012]

[0013] Where n represents the number of layers of the neural network. represents the output of the nth layer of neural network after local time modeling, Represents the output of the nth layer network after the self-attention mechanism, Represents the final output features of the nth layer network, LN represents the normalization operation, represents the output features of the n-1th layer network, represents the local time modeling function, represents the feature extraction function of the self-attention layer, Represents the feedforward layer network calculation function;

[0014] The calculation process is:

[0015]

[0016] in, Indicates the number in Indicates the nth layer Group feature output, The nth layer The weight matrix of the group, Represents a linear rectification operation is the linear transformation function, represents the connection operation, T represents the number of time series frames, and C represents the dimension of the features in each time series frame. represents the feature output of different groups, and G represents the number of groups.

[0017] As a preferred technical solution, in step S1, a dynamic time warping alignment method is used to achieve synchronization of audio data features and video data in the time dimension:

[0018] Features after decoding the input video encoder And the audio feature A after short-time Fourier transform, and then calculate the distance between the video and audio at different time steps to get the cumulative cost matrix:

[0019]

[0020]

[0021] in, represents the distance between the i-th time step of the video and the j-th time step of the audio, represents the features of the video at the i-th time step, express The i-th element in represents the features of the audio at the jth time step, represents the i-th element in A, represents the Euclidean distance, represents the cumulative cost matrix, From the starting point arrive The minimum cumulative cost of

[0022] Finally pass the end point of the cumulative cost matrix Trace back and find the best path , and calculate the average of the final distances:

[0023]

[0024] The tracking rules are:

[0025]

[0026] The final distance average is:

[0027]

[0028] in, indicates that the last time step of video and audio is aligned, represents the total number of time steps of video features, represents the total number of time steps of audio features, is a point on the path, Represents the video time step and the audio time step The alignment relationship, Indicates choosing the path with the smallest cumulative cost. Indicates the path number, K indicates the path The total number of steps in, K represents the number of time steps after alignment, represents the average alignment distance, Indicates the path The sum of the distances on .

[0029] As a preferred technical solution, in step S1, after extracting the information of each single mode, the features extracted from all modes are integrated by linear weighted summation to generate a weighted global feature combination. , the formula is:

[0030]

[0031] in, Represents the global features after weighted combination, represents the feature vector after decoding of the i-th modality, i represents the modality type, m represents the number of models of cross-modal features, Representation characteristics The weight vector of , and satisfy the constraints ;

[0032] By splicing , retaining global and fine-grained features and forming the final cross-modal feature vector representation:

[0033]

[0034] in, Represents the index of the mode, and [·] is the concatenation operation.

[0035] As a preferred technical solution, in step S2, after extracting the features of each modality, a shared mapping module is constructed, including the steps of:

[0036] S21, the input feature-extracted data X is linearly transformed to obtain the total query vector Q, key vector K and value vector V:

[0037]

[0038]

[0039]

[0040]

[0041]

[0042]

[0043] Where i represents the number of the head, and n represents the number of heads. represents the i-th element in Q, represents the i-th element in K, represents the i-th element in V, express The transformation matrix, express The transformation matrix, express The transformation matrix, Represents a splicing operation;

[0044] S22, use the attention mechanism to calculate the attention weight matrix:

[0045]

[0046] in, represents the calculation result of attention weight, represents the normalization function, represents the dimension of the key vector K, T represents the transpose calculation, represents the dot product computation of the transposed query and key;

[0047] S23, through the multi-head attention mechanism parallel calculation, independently calculates the weighted output of each attention head:

[0048]

[0049] in, represents the attention output of the i-th head;

[0050] S24, concatenates the outputs of multiple heads and obtains the final multi-head output through linear transformation:

[0051] Multi-Head Output Concat head head

[0052] Among them, Multi-Head Output represents the final multi-head output, The weight matrix representing the linear transformation;

[0053] S25, the final output of the shared mapping module is obtained through multi-head output and linear transformation:

[0054] X Output Multi-Head Output

[0055] Among them, X Output represents the output of the shared mapping module, X Output is the modal feature after decoupling.

[0056] A method for characterizing cross-modal data, including the above-mentioned method for decoupling cross-modal data, further comprising the following steps:

[0057] S3, graph representation: construct a propagation network graph based on the decoupled modal features, and perform convolution operations on cross-modal features in the propagation network graph through a graph convolutional network to generate a graph representation of cross-modal features.

[0058] As a preferred technical solution, step S3 includes the following steps:

[0059] S31, use cross entropy to optimize the internal node distribution of the subgraph in the propagation network graph and calculate the i-th modal subgraph node The normalized probability distribution of and the distribution loss within the subgraph are calculated as:

[0060]

[0061] in, represents the jth node in the i-th modal subgraph, represents the normalized probability distribution of the jth node in the i-th modal subgraph, n i represents the total number of nodes in the ith modal subgraph, Representation Node The eigenvector of , k represents the node number in the i-th modal subgraph, represents the exponential map, It means calculating the sum of the eigenvalues ​​of all nodes in the i-th modal subgraph after exponential mapping. represents the total distribution loss within the subgraph, and m represents the total number of modal types;

[0062] S32, use KL divergence to optimize the node similarity between modes, calculate the KL divergence between nodes j and K of mode i1 and mode i2, and construct a similarity matrix. The calculation formula of KL loss is:

[0063]

[0064] Among them, i1 and i2 represent the mode numbers. represents the KL divergence between node j in mode i1 and node k in mode i2, Normalized probability distribution of node j in mode i1;

[0065] S33, construct similarity matrix of nodes between modalities , and calculate the similarity loss between modalities :

[0066]

[0067]

[0068] in, represents the similarity matrix of nodes between modalities, Used to store the KL divergence between node pairs, represents the KL divergence of the node pair (j, k) between modalities, Represents the similarity loss between modalities.

[0069] As a preferred technical solution, the total loss function calculation formula of the graph convolutional network operation is:

[0070]

[0071] in, represents the total loss, Represents the balance coefficient.

[0072] As a preferred technical solution, a joint subgraph is constructed, and the graph convolution forward propagation formula of the graph convolution network operation is used to realize graph node fusion:

[0073] based on Construct a joint subgraph, the formula is:

[0074] ;

[0075] In the formula, ,

[0076] ,

[0077] ,

[0078] ;

[0079] in, is the set of nodes of the joint subgraph, represents the adjacency matrix, i represents the node number of the modal subgraph, m represents the number of nodes in the modal subgraph, represents the node numbered i of the modal subgraph, represents a sparse matrix, represents the KL divergence of the node pair (j, k) between modalities, represents the threshold parameter, Represents the layer number of the graph convolutional network, Indicates The node feature matrix of the layer, represents a nonlinear activation function, represents the normalized adjacency matrix, Indicates The node feature matrix of the layer, Indicates The learnable weight matrix of the layer, D represents the normalized degree matrix.

[0080] A cross-modal data characterization system, used to implement the cross-modal data characterization method, includes the following modules connected in sequence:

[0081] Feature extraction module: used to execute step S1;

[0082] Feature decoupling module: used to execute step S2;

[0083] Graph characterization module: used to execute step S3.

[0084] Compared with the prior art, the present invention has the following beneficial effects:

[0085] (1) The present invention is based on the method of cross-modal decoupling + fusion graph representation: the features of each modality are separated by independent encoders, and the deep fusion of cross-modal data is performed using a mutual graph convolutional network, thus achieving accurate processing and integration of cross-modal information;

[0086] (2) The present invention introduces a dynamic time-warping alignment and mutual learning mechanism: This not only ensures the time synchronization of temporal modalities such as audio and video, but also optimizes the learning of node relationships between different modalities, thereby improving the robustness and prediction performance of the model;

[0087] (3) The present invention can be used in some social networks to help deepen the understanding of social networks and user behavior: by graphically representing cross-modal data, users' interest preferences, interaction patterns, and information dissemination paths can be more accurately captured, which not only significantly improves the effects of personalized recommendations and service customization, but also helps the platform identify trends, predict popular content, and optimize resource allocation in advance. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 is a flow chart of the present invention;

[0089] Figure 2 It is a schematic diagram of the overall process of the present invention;

[0090] Figure 3 yes Figure 2 One of the partial enlarged pictures;

[0091] Figure 4 yes Figure 2 The second partial enlarged picture;

[0092] Figure 5 yes Figure 2 The third partial enlarged picture;

[0093] Figure 6 It is a schematic diagram of the decoupling architecture in the present invention;

[0094] Figure 7 yes Figure 6 One of the partial enlarged pictures;

[0095] Figure 8 yes Figure 6 The second partial enlarged picture;

[0096] Fig. 9 yes Figure 6 The third partial enlarged picture;

[0097] Fig.10 It is a schematic diagram of the representation architecture of the present invention. DETAILED DESCRIPTION

[0098] The present invention will be further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0099] Terminology explanation:

[0100] Graph: A graph is a data structure used to represent objects and the relationships between them. A graph consists of nodes (V) and edges (E), where nodes represent objects and edges represent connections or relationships between objects. Graphs are widely used in social networks, logistics routes, knowledge graphs and other fields.

[0101] Graph Convolutional Network (GCN): Graph Convolutional Network is an implementation of Graph Neural Network (GNN) that focuses on feature learning on graph data. It updates the representation of each node by recursively aggregating the features of the node's neighbors, thereby capturing the structure of the graph and the relationship between nodes. The core idea of ​​GCN is that the representation of each node depends not only on its own features, but also on the features of its neighbors. Compared with traditional neural networks, GCN can process non-Euclidean data more efficiently, such as user interaction graphs in social networks.

[0102] Attention mechanism: Attention mechanism is a computational method that simulates humans focusing on key information. When processing large amounts of data, the attention mechanism can assign higher weights to important information, thereby improving the performance of the model. It works by scoring different parts of the input data and dynamically selecting more critical features based on the scoring results. For example, in cross-modal tasks, the attention mechanism can allow the model to pay more attention to an important time segment in speech or a key area in an image, making information collaboration between different modalities more efficient.

[0103] Self-supervised learning: Self-supervised learning is a learning method that does not require manual annotation. It automatically generates pseudo labels from data by constructing tasks (such as contrastive learning or data reconstruction) to learn feature representations. In cross-modal learning, self-supervised learning can extract features and align semantics of data from different modalities, making effective use of unlabeled data.

[0104] Example 1

[0105] like Figures 1 to 10 As shown, the present invention belongs to the fields of multimodal learning, self-supervised learning and graph representation learning in deep learning, and proposes a graph representation method based on cross-modal decoupling and fusion. The method aims to solve the complexity of decoupling and fusion of cross-modal data features, especially the heterogeneity problem when different modal data such as audio, video, image and text are processed in a collaborative manner. In response to the challenges of heterogeneity and inconsistent information structure of cross-modal data, the present invention proposes to perform feature modeling through an independent encoder, and combine the dynamic time warping (DTW) alignment method to achieve accurate alignment of audio and video modalities with time features. Secondly, the subgraphs of each modality are represented and learned through a mutual graph convolutional network (GCN), and the node relationship between each modality is optimized through a mutual learning mechanism and KL divergence constraints to achieve cross-modal information fusion in the graph space. This method effectively enhances the information complementarity between cross-modalities, improves the fusion ability and prediction performance of cross-modal data, and is particularly suitable for complex cross-modal information analysis and prediction tasks.

[0106] The present invention designs a shared mapping module and an adaptive combination mechanism mainly from the perspective of cross-modal information decoupling and fusion graph representation, encourages collaboration among modalities through shared functional modules, and improves the flexibility and effectiveness of content understanding and perception.

[0107] The present invention aims to solve the problem of traditional methods relying on a single modality or network structure in the process of cross-modal decoupling, which leads to cross-modal information interference and difficulty in decoupling. To this end, a cross-modal decoupling method combined with a shared mapping module is proposed, which can effectively solve the problem of information entanglement by promoting collaboration between modalities, thereby improving the understanding of cross-modal content. At the same time, in response to the challenges of cross-platform content perception and representation in high-difference environments, a fusion graph representation method driven by mutual graph convolution is proposed. This method uses a graph element learning framework, combined with the network graph structure characteristics of social media communication and cross-modal content characteristics, to achieve the fusion of data decoupling and unified graph element representation, significantly improving the understanding of cross-modal content. These methods effectively overcome the performance bottlenecks of traditional technologies in data decoupling and graph representation, and enhance the accuracy and consistency of cross-modal content fusion.

[0108] The general idea of ​​the present invention is: starting from the key problem of modal information interweaving and representation difficulty in cross-modal content processing, through the idea of ​​decoupling, the whole process is divided into three core stages: feature extraction, feature decoupling and graph representation. First, the temporal modalities are aligned through dynamic time warping, and the independent information and shared information of the cross-modal data are decoupled in combination with the shared function module, and the unique features and shared information of the modalities are extracted to reduce the interference between the modalities and lay the foundation for the subsequent unified representation. The independent information refers to the unique features or content information of each modality (such as image, text, video, audio), for example: in audio features, such as tone, rhythm, etc. Shared information refers to the parts that are related or complementary between different modalities. This information reflects the common semantics or correlation between modalities. For example: when the image and text describe the same content, there may be a certain correlation between the semantic information of the text and the visual content of the image. Subsequently, by constructing a unified cross-modal spatial graph, using the cross-modal decoupled features as graph nodes, a subgraph is constructed for each modality, and the mutual learning mechanism and graph convolutional network operations are used to quantify the node features between different modalities and form a global feature graph. This invention significantly improves the understanding of cross-modal content, and at the same time has higher representation accuracy and management efficiency in complex network environments, and has strong adaptability and scalability.

[0109] Based on the above invention ideas, the method for cross-modal content decoupling and fusion graph representation provided by the present invention is as follows, and attached Figures 2 to 10 To further illustrate:

[0110] S1, cross-modal data preprocessing and feature extraction: Perform cross-modal processing on cross-modal network content. The audio encoder extracts core features such as spectrograms from audio, the video encoder extracts key frames and time series features from video, the text encoder completes word segmentation and semantic embedding from text, and the image encoder extracts visual features such as color and texture from images. At the same time, dynamic time warping is used to align audio and video to ensure synchronization of the time dimension and the uniqueness of information between modalities.

[0111] S2, decoupling of cross-modal content features: After feature extraction, joint modeling is performed using the attention mechanism through a shared mapping module. Through the "Key-Query-Value" mechanism, shared information between modalities is collaboratively represented while keeping modality-specific information independent. Dynamic weighting mechanisms (such as F(Q,K) weighted calculation and normalization) effectively separate unique and shared features and reduce modal interference. The adaptive selection mechanism dynamically selects modal feature outputs according to task requirements to improve the flexibility and robustness of the model.

[0112] S3, fused graph representation: cross-modal features are globally integrated through a unified graph representation method. Each modal content is represented as a unified graph node, and the relationship between nodes is modeled as the edge of the graph through modal similarity and the correlation between nodes. In this process, through the mutual learning mechanism, the cross-modal features are further deeply integrated with the multimodal semantic association to generate a graph representation of the cross-modal content.

[0113] The purpose of step S1 in the above-mentioned cross-modal content decoupling and fusion graph representation method is to perform sub-modal processing on the cross-modal network content to extract the core features of each modality and ensure the synchronization of different modalities in the time dimension. Text data is segmented and semantically embedded through a text encoder; for image data, an image encoder is used to extract color, texture and other features; for audio data, an audio encoder is used to extract core features such as spectrograms. Among them, short-time Fourier transform can be used, which captures the frequency components of the signal at different time points by dividing the signal into multiple short segments and applying Fourier transform to each segment. The core formula is as follows:

[0114]

[0115] Represents the short-time Fourier transform of the input signal x(t), x(t) is the input audio signal, w(t) is a window function (such as Hanning window, rectangular window, etc.) used to localize the time range, τ is the time offset, τ indicates the position of the currently processed signal segment, ω is the angular frequency, and j is the imaginary unit. These features can capture the time and frequency information of the audio, thereby providing the model with rich audio representation.

[0116] At the same time, in order to extract time series features from video data, the video is segmented into a series of non-overlapping visually labeled frames, and then a video encoder is used in combination with local time modeling and self-attention mechanism to capture the temporal dependency between frames. The time series features are enhanced by functional modular processing modeling, where functional modular processing includes temporal dependency modeling, self-attention mechanism processing, and nonlinear feature conversion. The formula for functional modular processing is expressed as:

[0117]

[0118]

[0119]

[0120] Where n represents the number of layers of the neural network. represents the output of the nth layer of neural network after local time modeling, Represents the output of the nth layer network after the self-attention mechanism, Represents the final output features of the nth layer network, LN represents the normalization operation, represents the output features of the n-1th layer network, represents the local time modeling function, represents the feature extraction function of the self-attention layer, Represents the feedforward layer network calculation function;

[0121] in the formula The calculation process can be expressed as:

[0122]

[0123] in, Indicates the number in Indicates the nth layer Group feature output, The nth layer The weight matrix of the group is a learnable temporal relationship parameter. Represents a linear rectification operation is a linear transformation function, which is used to further extract or transform the concatenated global features. represents the connection operation, T represents the number of frames in the time series, that is, the division unit of the input feature in the time dimension, and C represents the dimension size of the feature in each time frame. Represents the feature outputs of different groups, G represents the number of groups, and the grouping criterion is to divide the input features into G different subspaces (i.e., a low-dimensional feature set divided from a high-dimensional feature space) by combining the time and feature dimensions, so that each group focuses on a specific time segment or feature subspace.

[0124] Finally, the DTW dynamic time warping alignment method is used to achieve the time synchronization of audio and video. The DTW algorithm calculates the similarity between two time series by nonlinearly stretching or compressing the time axis, and inputs the features decoded by the video encoder. Assuming that the audio feature after short-time Fourier transform is A, the distance between the video and audio at different time steps is calculated to obtain the cumulative cost matrix:

[0125]

[0126]

[0127] represents the distance between the i-th time step of the video and the j-th time step of the audio, Represents the feature of the i-th time step, the final feature from the video , represents the features at the jth time step, from the final feature extraction of the audio, represents the Euclidean distance, Represents the cumulative cost matrix, that is, from the starting point arrive The minimum cumulative cost, d , , They represent the three paths above, to the left, and diagonally. Finally, the path with the lowest cost will be selected.

[0128] Finally pass the end point of the cumulative cost matrix Trace back and find the best path , and calculate the average of the final distances:

[0129]

[0130] Tracking rules:

[0131]

[0132] Final distance average:

[0133]

[0134] indicates that the last time step of video and audio is aligned, The total number of time steps of video features, The total number of time steps of the audio features, is a point on the path, representing the video time step and the audio time step The alignment relationship, It means to select the path with the smallest cumulative cost from the top, left and diagonal paths. K is the optimal path. The total number of steps in represents the number of time steps after alignment, represents the average alignment distance, Indicates the path The sum of the distances on .

[0135] After obtaining the information of each single mode, the features extracted from all modes are integrated through linear weighted summation to generate a weighted global feature combination. , the formula is:

[0136]

[0137] in, Represents the global features after weighted combination, represents the feature vector after decoding of the i-th modality, i represents the modality type, such as video, text, etc., and m represents the number of models of cross-modal features. Representation characteristics The weight vector of , and satisfy the constraints ;

[0138] By splicing , retaining global and fine-grained features and forming the final cross-modal feature vector representation:

[0139]

[0140] in, Indicates the selection or index of the corresponding mode, for example: Indicates the index of the first mode, represents the final cross-modal feature vector representation and [·] is the concatenation operation.

[0141] The purpose of step S2 in the above-mentioned cross-modal content decoupling and fusion graph representation method is to achieve decoupling of cross-modal features through a shared mapping module (by building a shared mapping module to encourage collaboration among modalities, thereby achieving decoupling of cross-modal features) and joint modeling using the attention mechanism. The mapping between modalities is completed through attention calculation and linear transformation between features, and the conversion of information from the modal space to the shared space is achieved, thereby achieving effective decoupling of information from different modalities.

[0142] The cross-modal decoupled features are linearly transformed through a unified weight matrix. These weight matrices , , Shared by all modalities, rather than designed separately for a particular modality, the following are the main steps to establish a shared mapping module:

[0143] 1. Input the feature-extracted data X (i.e. ) The total query vector Q, key vector K and value vector V are obtained through linear transformation:

[0144]

[0145]

[0146]

[0147]

[0148]

[0149] Where i represents the number of the head, and n represents the number of heads. represents the i-th element in Q (the query of the i-th head of multimodal data), represents the i-th element in K (the key of the multimodal data in the i-th head), represents the i-th element in V (the value of the multimodal data in the i-th head), express The transformation matrix, express The transformation matrix, express The transformation matrix, Represents a splicing operation;

[0150] 2. Use the attention mechanism to calculate the attention weight matrix:

[0151]

[0152] in, represents the calculation result of attention weight, Represents the normalization function, which is used to normalize the weights to the interval from 0 to 1. Indicates the dimension of the key vector K, used for normalization to prevent the value from being too large, T represents the transpose calculation, The dot product calculation representing the transposed query and key is used to calculate the similarity;

[0153] 3. Perform multi-head processing on the attention mechanism, and calculate the attention weight independently through each head, so as to better capture the dependencies between different positions:

[0154]

[0155] in, represents the attention output of the i-th head;

[0156] 4. Concatenate the outputs of multiple heads and obtain the final multi-head output through linear transformation:

[0157] Multi-Head Output Concat head head

[0158] Among them, Multi-Head Output represents the final multi-head output, The weight matrix representing the linear transformation is used to integrate the concatenated multi-head outputs to further extract cross-modal features.

[0159] 5. The final output of the shared mapping module is obtained through multi-head output and linear transformation:

[0160] X Output Multi-Head Output

[0161] Among them, X Output represents the output of the shared mapping module, i.e., the final modal features after cross-modal decoupling. Represents the linear transformation weight matrix. The final multi-head output undergoes another linear transformation to generate the final output of the shared mapping module.

[0162] The purpose of step S3 in the above-mentioned cross-modal content decoupling and fusion graph representation method is to construct a unified cross-modal graph representation, deeply integrate cross-modal content and propagation paths, and generate high-quality structured representations. By designing a mutual learning mechanism, optimizing the node representation within the subgraph network, and introducing KL divergence as a loss constraint between subgraph networks, quantifying the similarity of node distribution between modalities, and finally using graph convolutional network operations to achieve graph node fusion, the following are the specific implementation steps:

[0163] 1. Construct a cross-modal content dissemination network diagram

[0164] First, the cross-modal decoupled features , which is defined as:

[0165]

[0166] Initialize the subgraph constructed for each modality i

[0167] Among them, the point set Represents the node and edge set in mode i Represents the connection relationship between nodes within the modality, initially defined as node similarity:

[0168]

[0169] represents the eigenvector of the i-th mode after cross-modal decoupling, m represents the total number of modes, is the cosine similarity, represents the eigenvector of the i-th mode of the j-th node after cross-modal decoupling, Represents the cosine similarity between nodes j and k in the i-th mode.

[0170] 2. Mutual learning mechanism design

[0171] 2.1 Using cross entropy to optimize node representation within subgraph networks

[0172] In order to optimize the node representation within the subgraph network, cross entropy is used to measure the uncertainty of node representation. k , calculate the i-th modal subgraph node The normalized probability distribution of and the distribution loss within the subgraph are defined as:

[0173]

[0174] represents the jth node in the i-th modal subgraph, represents the normalized probability distribution of the jth node in the i-th modal subgraph, which is used to represent the node The importance or weight in the subgraph, n i represents the total number of nodes in the ith modal subgraph, Representation Node The feature vector contains the information after cross-modal decoupling, k represents the node number, It means calculating the sum of the eigenvalues ​​of all nodes in the i-th modal subgraph after exponential mapping. represents the total distribution loss within the subgraph, and m represents the total number of modal types;

[0175] 2.2 Introducing KL divergence as a loss constraint between subgraph networks

[0176] After optimizing the node representation within the subgraph, use KL divergence to optimize the node similarity between modalities, calculate the KL divergence between nodes j and K of modal i1 and modal i2, and construct a similarity matrix to calculate its KL loss:

[0177]

[0178] represents the KL divergence between node j in mode i1 and node k in mode i2, The normalized probability distribution of node j in mode i1 can be obtained from 2.1;

[0179] 2.3 Constructing the Similarity Matrix of Nodes between Modalities

[0180] In order to more intuitively display the similarity between nodes, we can construct the similarity matrix of nodes between modalities based on the calculation results of 2.2. , and calculate the similarity loss between its modalities :

[0181]

[0182]

[0183] in, represents the similarity matrix of nodes between modalities, Store the KL divergence between pairs of nodes; represents the KL divergence of the node pair (j, k) between modalities; Represents the similarity loss between modalities (i.e., KL divergence loss), which is used to optimize the node association between modalities.

[0184] 2.4 Calculate the total loss

[0185] By calculating the loss within the subgraph and the loss between modes, it can be used to guide the entire graph representation process to ensure the optimization of the relationship between nodes and between modes. The total loss formula is:

[0186]

[0187] L represents the total loss, is the balance coefficient.

[0188] 2.5 Inter-modality Correlation Fusion

[0189] Based on the similarity matrix Construct a joint subgraph, ,

[0190] in , the adjacency matrix Expressed as:

[0191]

[0192] And use the graph convolution forward propagation formula of the graph convolution network operation to realize graph node fusion, that is, global feature update:

[0193]

[0194]

[0195] in, It is the node set of the joint subgraph, including nodes of all modes. Through the set union operation, the nodes of each modal subgraph are merged into the joint subgraph, is a sparse matrix, in which the non-zero elements represent the connection relationship between nodes. The KL divergence between node j and node k can be calculated based on get, is a threshold parameter, Indicates the layer number of the graph convolutional network (GCN), Indicates The node feature matrix of the layer, Represents a nonlinear activation function, which is used to introduce nonlinearity and improve the expressiveness of graph convolutional networks. Represents the adjacency matrix Perform normalization to obtain the normalized adjacency matrix. Indicates The node feature matrix of the layer, Indicates The learnable weight matrix of the layer, D represents the normalized degree matrix, and the degree matrix is ​​used to normalize the sparse matrix to ensure that the graph node features can be aggregated fairly during feature propagation.

[0196] So far, we have implemented a method based on cross-modal decoupling and fusion graph representation.

[0197] Example 2

[0198] like Figures 1 to 4 As shown, based on Example 1, this example provides a more detailed implementation method.

[0199] As 1 and Figure 2 As shown in the figure, in the first step, we first extract the information features of the input content, which will use the dynamic time warping alignment mentioned in S1, and then decouple the cross-modal content in the shared mapping module. In the second step, we will create graph nodes based on the decoupled features, and use the KL divergence and mutual learning mechanism mentioned in S3 to quantify and fuse nodes with high similarity between modalities, and finally obtain a global fusion graph representation of cross-modal information.

[0200] Application examples:

[0201] The method based on cross-modal decoupling and fusion graph representation is applied to the alignment and retrieval of cross-modal content, achieving efficient content management and user experience optimization. In practical applications, the content posted by users on social media platforms usually exists in a cross-modal form, such as mixed text and image posts, short videos with subtitles, or audio comments with background sound. The platform needs to process these heterogeneous data accurately to meet the needs of content review, personalized recommendation, and user interest analysis.

[0202] The present invention is used for the alignment of cross-modal content. The platform pre-processes the cross-modal content uploaded by users, and aligns the audio and video modes in time series through the dynamic time warping alignment method to ensure the consistency of the content in the temporal dimension. Next, for the image and text modalities, using the shared mapping module and the functional module, the system separates the shared information and unique information between the modalities through the "Key-Query-Value" mechanism, and performs collaborative modeling at the semantic level, thereby significantly improving the alignment accuracy of cross-modal content. Based on this method, when processing mixed text and image content uploaded by users, the platform can achieve semantic consistency detection with an image alignment accuracy of up to 94% and a text alignment accuracy of 92%, which greatly exceeds the 85% accuracy rate represented by the domestic technology "Toruis-Netcha v4.0", laying a solid foundation for subsequent cross-modal data analysis.

[0203] In addition, the present invention also plays a key role in cross-modal content retrieval. Through a cross-modal unified graph representation method, the platform models the production and propagation paths of cross-modal content as unified graph nodes, and constructs graph edges based on modal similarity, propagation paths, and user interaction information. In this unified graph structure, the platform uses a mutual graph convolutional network to perform deep representation learning on graph nodes, effectively integrating cross-modal features with propagation characteristics. For example, when a user uploads a picture or enters a text as a query, the platform can quickly retrieve related audio, video clips or text descriptions through a fusion graph representation, achieving a retrieval accuracy rate of 92% to 95%. This capability significantly exceeds the existing technology level, such as: the audio, video, and text hybrid retrieval of "Torui-Netcha v4.0" and the foreign "Meltwater-Social Media Listening and Analysis System", which not only improves the user's content retrieval efficiency, but also achieves more accurate content matching.

[0204] It can be seen that the application of the present invention has achieved remarkable results in cross-modal content management on social media platforms. It can enable the platform to quickly verify the semantic consistency of cross-modal content uploaded by users, effectively filter out bad information, and thus improve the efficiency of content review. At the same time, based on cross-modal alignment and retrieval results, the platform further optimizes the accuracy of personalized recommendations, allowing users to quickly obtain content that suits their interests, significantly improving the user experience. This method successfully solves the problems of heterogeneity and timing inconsistency in cross-modal content processing, and provides efficient and accurate technical support for content management on social media platforms.

[0205] As described above, the present invention can be preferably implemented.

[0206] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. According to the technical essence of the present invention, within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for decoupling cross-modal data, characterized in that: The following steps are involved: S1, feature extraction: extracting features of each modal data and synchronizing the features of different modal data in the time dimension, wherein the modal data includes: audio, video, image and text; S2, feature decoupling: construct a shared mapping module based on the attention mechanism, perform feature decoupling of different modal data, and output the decoupled modal features. After extracting the features of each modality, construct a shared mapping module, including the following steps: S21, the input feature-extracted data X is linearly transformed to obtain the total query vector Q, key vector K and value vector V: Where i represents the number of the head, and n represents the number of heads. represents the i-th element in Q, represents the i-th element in K, represents the i-th element in V, express The transformation matrix, express The transformation matrix, express The transformation matrix, Represents a splicing operation; S22, use the attention mechanism to calculate the attention weight matrix: in, represents the attention weight calculation result, represents the normalization function, represents the dimension of the key vector K, T represents the transpose calculation, represents the dot product computation of the transposed query and key; S23, through the multi-head attention mechanism parallel calculation, independently calculates the weighted output of each attention head: in, represents the attention output of the i-th head; S24, concatenates the outputs of multiple heads and obtains the final multi-head output through linear transformation: Multi-Head Output Concat head head Among them, Multi-Head Output represents the final multi-head output, The weight matrix representing the linear transformation; S25, the final output of the shared mapping module is obtained through multi-head output and linear transformation: X Output Multi-Head Output Among them, X Output represents the output of the shared mapping module, X Output is the modal feature after decoupling.

2. A method for decoupling cross-modal data according to claim 1, characterized in that: In step S1, when extracting features of modal data, if the modal data includes video data, the image and video frames are segmented into a series of non-overlapping visual markers, and then a video encoder is used to extract time series features from the video data in combination with local time modeling and self-attention mechanism, and functional modular modeling is performed. The formula of functional modular modeling is expressed as: Where n represents the number of layers of the neural network. represents the output of the nth layer of neural network after local time modeling, Represents the output of the nth layer network after the self-attention mechanism, Represents the final output features of the nth layer network, LN represents the normalization operation, represents the output features of the n-1th layer network, represents the local time modeling function, represents the feature extraction function of the self-attention layer, Represents the feedforward layer network calculation function; The calculation process is: in, Indicates the number in Indicates the nth layer Group feature output, The nth layer The weight matrix of the group, Represents a linear rectification operation is the linear transformation function, represents the connection operation, T represents the number of time series frames, and C represents the dimension of the features in each time series frame. represents the feature output of different groups, and G represents the number of groups.

3. A method for decoupling cross-modal data according to claim 2, characterized in that: In step S1, a dynamic time warping alignment method is used to achieve synchronization of audio data features and video data in the time dimension: Features after decoding the input video encoder And the audio feature A after short-time Fourier transform, and then calculate the distance between the video and audio at different time steps to get the cumulative cost matrix: in, represents the distance between the i-th time step of the video and the j-th time step of the audio, represents the features of the video at the i-th time step, express The i-th element in represents the features of the audio at the jth time step, represents the i-th element in A, represents the Euclidean distance, represents the cumulative cost matrix, From the starting point arrive The minimum cumulative cost of Finally pass the end point of the cumulative cost matrix Trace back and find the best path , and calculate the average of the final distances: The tracking rules are: The final distance average is: in, indicates that the last time step of video and audio is aligned, represents the total number of time steps of video features, represents the total number of time steps of audio features, is a point on the path, Represents the video time step and the audio time step The alignment relationship, Indicates choosing the path with the smallest cumulative cost. Indicates the path number, K indicates the path The total number of steps in, K represents the number of time steps after alignment, represents the average alignment distance, Indicates the path The sum of the distances on .

4. The method for decoupling cross-modal data according to claim 3, characterized in that: In step S1, after extracting the information of each single mode, the features extracted from all modes are integrated by linear weighted summation to generate a weighted global feature combination , the formula is: in, Represents the global features after weighted combination, represents the feature vector after decoding of the i-th modality, i represents the modality type, m represents the number of models of cross-modal features, Representation characteristics The weight vector of , and satisfy the constraints ; By splicing , retaining global and fine-grained features and forming the final cross-modal feature vector representation: in, Represents the index of the mode, and [·] is the concatenation operation.

5. A method for representing cross-modal data, characterized in that: A method for decoupling cross-modal data according to any one of claims 1 to 4, further comprising the following steps: S3, graph representation: construct a propagation network graph based on the decoupled modal features, and perform convolution operations on cross-modal features in the propagation network graph through a graph convolutional network to generate a graph representation of cross-modal features.

6. A method for characterizing cross-modal data according to claim 5, characterized in that: Step S3 includes the following steps: S31, use cross entropy to optimize the internal node distribution of the subgraph in the propagation network graph and calculate the i-th modal subgraph node The normalized probability distribution of and the distribution loss within the subgraph are calculated as: in, represents the jth node in the i-th modal subgraph, represents the normalized probability distribution of the jth node in the i-th modal subgraph, n i represents the total number of nodes in the ith modal subgraph, Representation Node The eigenvector of , k represents the node number in the i-th modal subgraph, represents the exponential map, It means calculating the sum of the eigenvalues ​​of all nodes in the i-th modal subgraph after exponential mapping. represents the total distribution loss within the subgraph, and m represents the total number of modal types; S32, use KL divergence to optimize the node similarity between modes, calculate the KL divergence between nodes j and K of mode i1 and mode i2, and construct a similarity matrix. The calculation formula of KL loss is: Among them, i1 and i2 represent the mode numbers. represents the KL divergence between node j in mode i1 and node k in mode i2, Normalized probability distribution of node j in mode i1; S33, construct similarity matrix of nodes between modalities , and calculate the similarity loss between modalities : in, represents the similarity matrix of nodes between modalities, Used to store the KL divergence between node pairs, represents the KL divergence of the node pair (j, k) between modalities, Represents the similarity loss between modalities.

7. A method for characterizing cross-modal data according to claim 6, characterized in that: The total loss function of the graph convolutional network operation is calculated as: in, represents the total loss, Represents the balance coefficient.

8. A method for characterizing cross-modal data according to any one of claims 5 to 7, characterized in that: Construct a joint subgraph and use the graph convolution forward propagation formula of the graph convolution network operation to achieve graph node fusion: based on Construct a joint subgraph, the formula is: ； In the formula, , ， ， ； in, is the set of nodes of the joint subgraph, represents the adjacency matrix, i represents the node number of the modal subgraph, m represents the number of nodes in the modal subgraph, represents the node numbered i of the modal subgraph, represents a sparse matrix, represents the KL divergence of the node pair (j, k) between modalities, represents the threshold parameter, Represents the layer number of the graph convolutional network, Indicates The node feature matrix of the layer, represents a nonlinear activation function, represents the normalized adjacency matrix, Indicates The node feature matrix of the layer, Indicates The learnable weight matrix of the layer, D represents the normalized degree matrix.

9. A cross-modal data representation system, characterized in that: A method for representing cross-modal data according to any one of claims 5 to 8, comprising the following modules connected in sequence: Feature extraction module: used to execute step S1; Feature decoupling module: used to execute step S2; Graph characterization module: used to execute step S3.

Images