Cross-modal semantic alignment method for authenticity identification of social media information
Through the combination of hybrid pooling expert framework and multi-task learning, negative example mining and bidirectional ternary sorting loss function are used to optimize cross-modal semantic alignment, solving the problem of insufficient understanding of cross-modal semantic relationships in social media information authenticity and false identification. , improves detection accuracy.
Patent Information
- Application Number
- CN202411165265.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-03
AI Technical Summary
The existing technology fails to effectively utilize cross-modal semantic relationships in the identification of authenticity of social media information, resulting in limited detection performance, especially in environments with complex and diverse data and noise.
Adopting a hybrid pooling expert framework combined with multi-task learning, through joint training of matching networks and authenticity identification networks through graphic and text, negative example mining and bidirectional ternary sorting loss function optimize cross-modal semantic alignment, and improve multimodal information Feature alignment quality.
The detection accuracy of the social media information authenticity identification model is significantly improved, and the alignment process is optimized through multi-task learning and negative example mining, and the understanding and alignment effect of cross-modal semantic relationships are enhanced.
Smart Images

Figure CN119067124A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of multimodal information authenticity identification, and in particular to a cross-modal semantic alignment method for social media information authenticity identification. Background Art
[0002] Currently, multimodal information on social media mostly exists in the form of a combination of images and text. When performing authenticity identification, it is biased and insufficient to rely solely on a single modality (text or vision) to make decisions. Therefore, it is necessary to combine the visual modality with the text modality for modeling. In this process, cross-modal semantic relations are the key to connecting the two modalities and are crucial to the understanding of multimodal information. Most existing work [1] [2] [3] represents the information of a single modality separately, ignoring the impact of the cross-modal semantic gap on the effective use of cross-modal semantic relationship modeling. When dealing with feature interactions between multimodal data, current research methods [4] [5] mostly focus on using attention mechanisms to simply improve the semantic alignment between text features and visual features, but do not construct a visual semantic shared space for multimodal features, nor design a special cross-modal semantic alignment strategy for the authenticity identification task. This reduces the understanding of cross-modal semantic relations and affects the performance of the detection model.
[0003] Therefore, considering the importance of cross-modal semantic alignment for the task of authenticity identification of multimodal information, Chen et al. [6] Based on the multi-task learning model, a strategy is specifically proposed to achieve cross-modal semantic alignment through the image-text binary classification auxiliary task. The semantically aligned visual and text features are used to perform subsequent cross-modal semantic relationship modeling, which significantly improves the effect of the authenticity identification task. Inspired by this, Wang et al. [7] Considering the superiority of contrastive learning in multimodal representation learning [8][9]
[10]
[11] , proposed a cross-modal semantic alignment strategy based on contrastive learning, which combines the image-text binary classification task with the image-text contrast learning objective, constructs a joint embedding space, and further effectively aligns the visual and text representations semantically. However, existing research has not taken into account the data characteristics of the authenticity identification task and needs to develop a targeted cross-modal semantic alignment strategy for it.
[0004] The cross-modal semantic alignment strategy based on the construction of visual semantic shared space is the main method at present, and it is easy to transfer and apply to different multimodal downstream tasks. However, given that different downstream tasks have individual differences due to different task objectives, when applying this framework to the task of authenticity identification of multimodal information on social media, appropriate improvements need to be made to enable it to play the corresponding semantic alignment effect. Existing alignment methods face the alignment of complex social media diversified data. Since social media data does not have explicit image-text pair labels, and the quality of image-text data is also uneven, and the multimodal data on social media contains more noise, it is impossible to align social media information well, resulting in poor performance in the task of authenticity identification of social media information. Summary of the invention
[0005] The purpose of the present invention is to provide a cross-modal semantic alignment method for social media information authenticity identification in order to improve the accuracy of social media information authenticity identification model detection.
[0006] The purpose of the present invention can be achieved by the following technical solutions:
[0007] A cross-modal semantic alignment method for authenticity identification of social media information, the method comprising the following steps:
[0008] S1. Obtain an original data set, wherein the original data set includes image-text pairs of real news and image-text pairs of fake news, and the image-text pairs of real news are used as the real data set;
[0009] S2, embedding the images and texts in the real dataset respectively;
[0010] S3, the embedded images and texts are input into the hybrid pooling expert framework, and the hybrid pooling expert framework is trained based on the loss function output by the hybrid pooling expert framework;
[0011] S4. Obtain the overall embedding representation output by the trained hybrid pooling expert framework, wherein the overall embedding representation is divided into positive samples and negative samples. Based on the image-text pair matching network, calculate the cosine similarity between the image and the text for each image-text pair in the overall embedding representation, and sort each image-text pair of the embedding representation based on the cosine similarity. Select negative samples with high similarity as new example image-text pairs and use them together with positive samples as training data for the authenticity identification network. Based on the obtained loss function, jointly train the image-text pair matching network and the authenticity identification network to obtain an image-text pair matching model and an authenticity identification model. Finally, obtain the actual social media information to be tested, input the social media information to be tested into the trained hybrid pooling expert framework, input the output actual alignment features into the authenticity identification model, and obtain the authenticity identification result of the social media information.
[0012] Furthermore, the original data set is in Refers to the picture and text of real news. It refers to the image-text pairs of fake news. The image-text pairs from the same real news in the original data set are regarded as positive samples, while other combinations of image-text pairs are regarded as negative samples.
[0013] Furthermore, there are m pairs of positive samples and m(m-1) pairs of negative samples.
[0014] Furthermore, the specific steps of embedding the images and texts in the real data set are: inputting the images and texts in the real data set into the visual encoder and the text encoder respectively, and outputting the embedded images and texts.
[0015] Furthermore, the visual encoder is a convolutional neural network or a pre-trained visual model.
[0016] Furthermore, the text encoder is a recursive neural network or a pre-trained model of text.
[0017] Furthermore, the hybrid pooling expert framework includes a routing gate module, an aggregation expert module and a loss function module.
[0018] Furthermore, the specific steps of S3 are: the embedded images and texts are input into the routing gate module, the corresponding aggregation experts in the aggregation expert module are selected, the aggregation expert module learns weights based on the selected aggregation experts and performs weighted aggregation, the loss function is calculated based on the overall vector obtained by aggregation, and the hybrid pooling expert framework is trained based on the loss function.
[0019] Furthermore, the routing gate module calculates the gate mask corresponding to the embedded image and text for denoising, and the gate mask is:
[0020]
[0021] in, It is the result of element-wise multiplication of the query and key vectors of the embedded image and text, and the gate mask M of the query and key vectors Q , M K Generated by two fully connected layers and sigmoid activation function, where σ represents the sigmoid operation, Expression weight, Describe the bias term.
[0022] Furthermore, the loss function is the sum of a bidirectional ternary sorting loss function and a load balancing loss function, and the load balancing loss function is:
[0023]
[0024] Where S i represents the proportion of samples assigned to pooling expert i, P i represents the proportion of routing probability assigned to pooling expert i.
[0025] Compared with the prior art, the present invention has the following beneficial effects:
[0026] Based on the data alignment of the hybrid pooling expert framework, the present invention provides semantically aligned multimodal vectors for multimodal information detection through multi-task learning. That is, on the basis of the identification task, an auxiliary negative example mining image-text pair matching task is used to optimize the aligned data, and more negative example image-text pairs with high similarity are added as training data for the authenticity identification network, thereby improving the training effect of the network and improving the accuracy of the social media information authenticity identification model detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a feature aggregation strategy diagram of the present invention;
[0028] Figure 2 Schematic diagram of the negative example mining process of the present invention;
[0029] Figure 3 It is a flow chart of the present invention. DETAILED DESCRIPTION
[0030] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0031] The purpose of the present invention is to propose a cross-modal semantic alignment method for social media multimodal information authenticity identification tasks. Based on the field of deep learning, a two-stage method is proposed, which fully considers the data characteristics of the current task and provides semantically aligned multimodal vectors for multimodal information detection through multi-task learning, thereby realizing the modeling of cross-modal semantic relationships and improving the accuracy of authenticity identification model detection. The flowchart of the present invention is as follows Figure 3 As shown, the process of the present invention comprises the following steps:
[0032] S1. Obtain an original data set, wherein the original data set includes image-text pairs of real news and image-text pairs of fake news, and the image-text pairs of real news are used as the real data set;
[0033] S2, embedding the images and texts in the real dataset respectively;
[0034] S3, the embedded images and texts are input into the hybrid pooling expert framework, and the hybrid pooling expert framework is trained based on the loss function output by the hybrid pooling expert framework;
[0035] S4. Obtain the overall embedding representation output by the trained hybrid pooling expert framework, wherein the overall embedding representation is divided into positive samples and negative samples. Based on the image-text pair matching network, calculate the cosine similarity between the image and the text for each image-text pair in the overall embedding representation, and sort each image-text pair of the embedding representation based on the cosine similarity. Select negative samples with high similarity as new example image-text pairs and use them together with positive samples as training data for the authenticity identification network. Based on the obtained loss function, jointly train the image-text pair matching network and the authenticity identification network to obtain an image-text pair matching model and an authenticity identification model. Finally, obtain the actual social media information to be tested, input the social media information to be tested into the trained hybrid pooling expert framework, input the output actual alignment features into the authenticity identification model, and obtain the authenticity identification result of the social media information.
[0036] In S1, the original dataset is defined as in Refers to the picture and text of real news. It refers to the image and text pairs of fake news. Based on this, a new dataset is manually created The image-text pairs from the same real news are regarded as positive sample pairs, while other combinations of image-text pairs are each other's negative samples. Assuming there are m pairs of positive image-text pair samples, there are m(m-1) pairs of negative samples.
[0037] In S2, multimodal feature extraction is performed to embed the images and texts in the dataset. For the image v and text t, the corresponding unimodal segment feature representations are first obtained based on the visual encoder and the text encoder. and in N is the number of image regions (depending on the encoder, the regions include region, grid, patch), and L is the number of words in the text. The choice of feature encoder varies depending on the modality. For images, it can include convolutional neural networks or pre-trained models for vision, and for text, it can include recursive neural networks or pre-trained models for text.
[0038] In S3, a visual semantic shared space is constructed. A shared embedding space is constructed based on the visual semantic paradigm, and the fragment features of different modalities are aggregated into an overall vector through a feature aggregation strategy, and then the cosine similarity between each other is calculated in the same visual semantic space. The present invention adopts a feature aggregation strategy based on the idea of mixed experts-Mixture of Pooling Experts Framework (MoPE for short), which consists of a routing gate module, an aggregation expert module, and a loss function module. Among them, the routing gate module can adaptively select aggregation experts for each sample according to the intra-modal relationship, the aggregation expert module can adaptively learn weights for fragment features and perform weighted aggregation, and the loss function module can optimize the alignment of semantic features and the load balancing of aggregation experts. The framework diagram of this strategy is shown in the figure Figure 1 shown.
[0039] In S4, in order to assist in learning semantic alignment, the present invention introduces the image-text pair matching task, which is essentially a binary classification task, that is, predicting whether an input image-text pair is positive (matching) or negative (mismatching). Specifically, the image-text similarity calculated based on the image-text matching task is further mined to perform the image-text pair matching task on the most difficult-to-distinguish negative sample pairs in a batch, that is, for the image v and text t in the same batch, After obtaining the overall embedding representation from the feature aggregation strategy MoPE framework, the cosine similarity is calculated. For each image and text, the samples with the highest similarity in the corresponding negative examples (text or image) are selected according to the similarity sorting, and new negative example image-text pairs are formed and then classified and trained together with the initial positive example image-text pairs. Figure 2 It is a schematic diagram of the negative example mining process.
[0040] This cross-modal semantic alignment strategy will be used as an auxiliary task of the authenticity identification model and jointly trained with the main authenticity identification task. The semantically aligned visual features and text features obtained by the auxiliary task will be used as the input of the main authenticity identification task.
[0041] In summary, this method mainly includes three parts. The first is the dataset reconstruction module, which is responsible for extracting positive image-text pairs based on the original dataset and reconstructing a new dataset. Then there is the multimodal feature extraction module, which is responsible for extracting visual fragment features and text fragment features according to the corresponding modal encoder. Finally, there is the two-stage cross-modal feature semantic alignment module, which learns semantic alignment in the embedding space through the feature aggregation strategy MoPE, and uses an auxiliary negative example mining image-text pair matching task to optimize the process.
[0042] The hybrid pooling expert framework of the present invention is specifically:
[0043] like Figure 1As shown in Figure 1, for the input image data and text data, this method makes full use of the self-attention mechanism based on the intra-modal relationship through the routing gate module to route each sample to the appropriate pooling expert for feature aggregation. It is worth noting that only the detailed routing process of the visual branch will be described here, because the routing principle of the text branch is the same. First, a set of regional features G = {g1, g2, …, g n} as the input of the routing gate strategy, where make Represent the query, key and value vectors in the attention mechanism, where n represents the number of regions and d refers to the dimension of the vector. Considering that the dot product between the query and the key may contain noise, this module proposes a gate mechanism to filter out useless information and retain useful information. First, the module needs to calculate the gate mask of the query and key vectors. After that, the model uses the gate mask to reduce the noise of the query and key vectors, and calculates the value vector attn after adding attention. Then, the attn feature with added gate attention is converted into a routing gate representation vector Z using a layer of full connection. Finally, each image G and each text T is routed to the pooling expert of the corresponding modality with the highest probability based on the normalized distribution on the m pooling experts, which can also be called an aggregation operator.
[0044] In the routing gate module, considering that the dot product between the query and the key may contain noise, this module proposes a gate mechanism to filter out useless information and retain useful information. First, the module needs to calculate the gate mask of the query and key vectors. The calculation process can be expressed as:
[0045]
[0046] in is the result of element-wise multiplication of the query and key vectors. Afterwards, the gate mask M of Q and K Q , M K Generated by two fully connected layers and sigmoid activation function, where σ represents the sigmoid operation,
[0047] After that, the model uses the gate mask to reduce the noise of the query and key vectors, and calculates the value vector attn after adding attention. Then, the attn feature with added gate attention is converted into the routing gate representation vector Z using a layer of full connection. The calculation process can be expressed as:
[0048]
[0049] Z=attnW attn +b attn
[0050] where the softmax function is applied to each row, n represents the number of pooling experts.
[0051] Finally, each image G and each text T is routed to the pooling expert of the corresponding modality with the highest probability based on the normalized distribution on the m pooling experts, which can also be called an aggregation operator. The formula process for calculating the probability of G and T being routed to expert i can be expressed as:
[0052]
[0053] The routing probability of the optimal pooling expert about G can be expressed as p(G)=max{p i (G)} is calculated. For word-level text features T = {t1, t2, ..., t s}, the optimal pooling expert routing probability calculation expression for sentence T is p(T)=max{p i (T)}.
[0054] In order to aggregate the visual and textual fragment features in the visual semantic shared space into holistic vector embeddings respectively, so as to measure the similarity of visual and textual modalities, this method designs a special aggregation expert module. This module consists of m pooling experts, each of which pools and aggregates the routed images and texts based on the same model structure. Specifically, this module first sorts the unimodal fragment features on the feature dimension to be aggregated, then assigns weights to the sorted features based on a certain mechanism, and calculates the weighted sum between the features to obtain the corresponding overall vector, and finally multiplies it with the probability value calculated by the routing gate module to obtain the final aggregation vector result. For a set of inputs with n regional features The selected pooling expert aggregates it into a fixed-length vector Let max k (·) represents the function of extracting the first k values from the feature sorting list, and the coefficient θ k It is the corresponding weight of the kth highest value after sorting the feature value on the regional feature dimension to be aggregated, and 1≤k≤n. The weight value is the normalized distribution on n regions. p(G) represents the routing probability value of the image G calculated from the routing gate module. Pooling experts need to obtain θ based on a certain calculation mechanism. kThe paper uses an encoder-decoder structure to achieve this goal. Specifically, the structure consists of two parts: the position encoder, a position encoding function based on trigonometric functions, which can encode the regional position of the feature to obtain the corresponding vectorized representation; the position decoder, a sequence model based on BiGRU that can generate pooling coefficients, which can process the vector obtained by the encoder.
[0055] In order to constrain the process of cross-modal semantic alignment, that is, to keep higher similarity scores between semantically similar samples and lower scores between semantically irrelevant samples, this paper designs a special loss function module, which includes two loss functions. One is the bidirectional ternary ranking loss function, which can optimize the distance between cross-modal semantic features; the other is the load balancing loss function, which can optimize the balanced load of the samples routed to each pooling expert.
[0056] The bidirectional ternary sorting loss function of the present invention is a semantic alignment objective function, and the load balancing loss function can achieve balanced load on routing experts. For a given m experts and batches, The formula of the load balancing loss function can be expressed as:
[0057]
[0058] Where S i represents the proportion of samples assigned to expert i, P i represents the proportion of routing probability assigned to expert i.
[0059] The routing gate module can adaptively assign each sample to the appropriate pooling operator based on the different intra-modal relations, while the aggregation expert module combines multiple pooling experts to pool and aggregate the fragment features of different samples, and the additional auxiliary load balancing objective function can promote the load balancing between pooling experts. This framework is based on the visual semantic shared space paradigm and is applicable to a variety of image encoders and text encoders. It can be plug-and-play in different multimodal related tasks to promote cross-modal semantic alignment.
[0060] This paper proposes a cross-modal semantic alignment method for social media information authenticity identification. It combines the image-text matching task based on the visual semantic paradigm with the image-text pair matching task of difficult negative example mining to optimize the feature alignment process, thereby providing a basis for modeling cross-modal semantic relationships for authenticity identification tasks and improving detection accuracy.
[0061] The documents cited in the present invention are as follows:
[0062] [1]Zhou Y,Yang Y,Ying Q,et al.Multimodal fake news detection viaclip-guided learning[C] / / 2023IEEE International Conference on Multimedia andExpo(ICME).
[0063] IEEE,2023:2825-2830.
[0064] [2]Xue J,Wang Y,Tian Y,et al.Detecting fake news by exploring theconsistency of multimodal data[J].Information Processing&Management,2021,58(5):102610.[3]Zheng J,Zhang X,Guo S,et al.MFAN:Multi-modal Feature-enhancedAttention Networks for Rumor Detection[C] / / IJCAI.2022:2413-2419.
[0065] [4]Jiang Y,Yu X,Wang Y,et al.Similarity-aware multimodal promptlearning for fake news detection[J].Information Sciences,2023,647:119446.
[0066] [5]Guo Q,Kang Z,Tian L,et al.Tiefake:Title-text similarity andemotion-aware fake news detection[C] / / 2023International Joint Conference onNeural Networks
[0067] (IJCNN).IEEE,2023:1-7.
[0068] [6]Chen Y,Li D,Zhang P,et al.Cross-modal ambiguity learning formultimodal fake news detection[C] / / Proceedings of the ACM web conference2022.2022:2897-2905.
[0070] [7]Wang L,Zhang C,Xu H,et al.Cross-modal contrastive learning formultimodal fake news detection[C] / / Proceedings of the 31st ACM InternationalConference on Multimedia.2023:5696-5704.
[0071] [8]Radford A,Kim J W,Hallacy C,et al.Learning transferable visualmodels from natural language supervision[C] / / International conference onmachine learning.
[0072] PMLR,2021:8748-8763.
[0073] [9]Jia C,Yang Y,Xia Y,et al.Scaling up visual and vision-languagerepresentation learning with noisy text supervision[C] / / Internationalconference on machine learning.PMLR,2021:4904-4916.
[0074]
[10] Li J,Selvaraju R,Gotmare A,et al.Align before fuse:Vision and language representation learning with momentum distillation[J].Advances inneural information processing systems,2021,34:9694-9705.
[0075]
[11] Yu J,Baevski A,Mohamed A,Zhou H,Bhosale S,Mahajan D,et al.Coca:
[0076] Contrastive captioners are image-text foundation models[C] / / Proceedings of the
[0077] 2022Conference on Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, 2022:1-14.
[0078] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A cross-modal semantic alignment method for social media information authenticity identification, characterized in that: The method comprises the following steps: S1. Obtain an original data set, wherein the original data set includes image-text pairs of real news and image-text pairs of fake news, and the image-text pairs of real news are used as the real data set; S2, embedding the images and texts in the real dataset respectively; S3, the embedded images and texts are input into the hybrid pooling expert framework, and the hybrid pooling expert framework is trained based on the loss function output by the hybrid pooling expert framework; S4. Obtain the overall embedding representation output by the trained hybrid pooling expert framework, wherein the overall embedding representation is divided into positive samples and negative samples. Based on the image-text pair matching network, calculate the cosine similarity between the image and the text for each image-text pair in the overall embedding representation, and sort each image-text pair of the embedding representation based on the cosine similarity. Select negative samples with high similarity as new example image-text pairs and use them together with positive samples as training data for the authenticity identification network. Based on the obtained loss function, jointly train the image-text pair matching network and the authenticity identification network to obtain an image-text pair matching model and an authenticity identification model. Finally, obtain the actual social media information to be tested, input the social media information to be tested into the trained hybrid pooling expert framework, input the output actual alignment features into the authenticity identification model, and obtain the authenticity identification result of the social media information.
2. According to claim 1, a cross-modal semantic alignment method for authenticity identification of social media information is characterized in that: The original data set is in Refers to the picture and text of real news. It refers to the image-text pairs of fake news. The image-text pairs from the same real news in the original data set are regarded as positive samples, while other combinations of image-text pairs are regarded as negative samples.
3. According to claim 2, a cross-modal semantic alignment method for authenticity identification of social media information is characterized in that: There are m pairs of positive samples and m(m-1) pairs of negative samples.
4. According to claim 3, a cross-modal semantic alignment method for authenticity identification of social media information is characterized in that: The specific steps of embedding the images and texts in the real data set are as follows: inputting the images and texts in the real data set into the visual encoder and the text encoder respectively, and outputting the embedded images and texts.
5. According to claim 4, a cross-modal semantic alignment method for authenticity identification of social media information is characterized in that: The visual encoder is a convolutional neural network or a pre-trained visual model.
6. The cross-modal semantic alignment method for social media information authenticity identification according to claim 5 is characterized in that: The text encoder is a recursive neural network or a pre-trained model of text.
7. The cross-modal semantic alignment method for authenticity identification of social media information according to claim 1 is characterized in that: The hybrid pooling expert framework includes a routing gate module, an aggregation expert module and a loss function module.
8. The cross-modal semantic alignment method for authenticity identification of social media information according to claim 1 is characterized in that: The specific steps of S3 are as follows: the embedded images and texts are input into the routing gate module, the corresponding aggregation experts in the aggregation expert module are selected, the aggregation expert module learns weights based on the selected aggregation experts and performs weighted aggregation, the loss function is calculated based on the overall vector obtained by aggregation, and the hybrid pooling expert framework is trained based on the loss function.
9. The cross-modal semantic alignment method for authenticity identification of social media information according to claim 8, characterized in that: The routing gate module calculates the gate mask corresponding to the embedded image and text for denoising, and the gate mask is: in, It is the result of element-wise multiplication of the query and key vectors of the embedded image and text, and the gate mask M of the query and key vectors Q , M K Generated by two fully connected layers and sigmoid activation function, where σ represents the sigmoid operation, Expression weight, Describe the bias term.
10. The cross-modal semantic alignment method for authenticity identification of social media information according to claim 8, characterized in that: The loss function is the sum of the bidirectional ternary sorting loss function and the load balancing loss function. The load balancing loss function is: Where S i represents the proportion of samples assigned to pooling expert i, P i represents the proportion of routing probability assigned to pooling expert i.
