WeChat public number tweet multi-modal question text inconsistency discrimination method and system based on causal inference

Abstract

The application relates to the field of artificial intelligence, and discloses a WeChat public account tweet multi-modal question-text inconsistency discrimination method and system based on causal inference, which comprises the following specific steps: acquiring training tweets and extracting multi-modal features; decoupling invariant factors and variable factors from the multi-modal features; based on a contrast learning strategy, decoupling the variable factors into causal factors reflecting deception tricks and writing styles of question-text inconsistency tweets under different scenarios and irrelevant factors containing false correlation bias; fusing the invariant factors and the causal factors; constructing a classifier; acquiring additional training tweets and performing data enhancement, training the classifier through the enhanced training tweets; identifying the titles and texts of the tweets through the trained classifier; and determining whether the tweets are question-text inconsistency. The application solves the problem that the prior art lacks modeling of false correlation bias, so that causal relationship information hidden in tweet features cannot be accurately explored, and has the characteristics of low demand for scarce labeled resources.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method and system for identifying multimodal title-text mismatch in WeChat public account posts based on causal inference. Background Technology

[0002] With the popularization of the internet and the advancement of mobile communication technology, more and more users tend to publish and share articles on social media platforms such as WeChat official accounts. The clicks on these articles can be monetized through advertising, leading some authors to prioritize commercial interests and use misleading titles and cover images to embellish low-quality articles, thus deceiving readers into clicking on irrelevant and inferior content. In recent years, such articles with mismatched titles and content have proliferated, becoming rampant. Readers who only read the title and not the full text are easily misled, even spreading misinformation, increasing distrust of various information in society, and hindering the exchange and dissemination of knowledge. In some sensitive fields, such as law and medicine, mismatched articles can even lead to direct losses. For example, incorrect legal knowledge can lead users to commit crimes, and fake medical articles can mislead people into self-treating illnesses, causing health problems. In general, these low-quality articles cause significant inconvenience and resource waste for readers, society, and social media platforms. With a large number of WeChat public accounts publishing articles daily, it is impossible to manually review them all. Therefore, developing machine-automated identification technology for articles with mismatched titles and content is of great significance and value for purifying the online environment, cleaning up information dissemination channels, and restoring the credibility of public account platforms.

[0003] Tweets with mismatched titles and content come in various forms, encompassing different modalities and combinations. Examples include fabricated headlines (false claims), mismatched images and text (unrelated cover images and text), and attention-grabbing titles and covers. However, traditional methods focus on single-type features and cannot comprehensively analyze the combined characteristics of different modalities in tweets. These methods can be divided into two categories: social behavior-based methods and content quality-based methods. Deceptive tweets can garner a large number of views and shares in a short period, but due to their poor content quality, they are quickly read. Therefore, methods based on dissemination information use comprehensive analysis of social behavior metadata such as comments, views, shares, and saves to identify mismatched tweets. These methods require sufficient reader feedback for judgment, but this feedback is usually delayed; a significant portion of readers may not even leave social traces such as comments, shares, or saves. Therefore, mismatched tweets can only be detected after widespread dissemination and significant reader participation, failing to effectively prevent the spread of low-quality tweets in a timely manner. In contrast, content-quality-based discrimination methods focus on analyzing the language patterns of tweets from the perspectives of deceptive vocabulary, grammar, subjectivity, punctuation, and content consistency. These methods can detect tweets with mismatched titles and content during the review stage on public accounts. Early work relied on manually set rules for discrimination, which heavily depended on expert knowledge and lacked scalability. With the development of deep learning, recent research has shifted to using neural networks to capture the correlation features between tweet titles and body text. However, the learned features are often mixed with false association bias and cannot effectively represent the underlying factors leading to deceptive behavior. This results in insufficient robustness and generalization to new variations in the models.

[0004] The prior art includes a method and apparatus for determining the consistency between the content and title of a document. The method for determining the consistency between the content and title includes: A. Searching at least one candidate website using the title of the target document to obtain candidate documents of the same type as the target document; B. Clustering the target document and each candidate document based on the similarity between their content; C. Determining the optimal cluster in the clustering results; D. If the target document does not belong to the optimal cluster, determining that the content and title of the target document are inconsistent; otherwise, determining that the content and title of the target document are consistent.

[0005] However, existing technologies lack modeling for false association bias, thus failing to accurately uncover causal information hidden in tweet features. Therefore, how to invent a multimodal title-text mismatch detection method and system that can model false association bias and accurately uncover causal information hidden in tweet features is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] To address the problem that existing technologies lack modeling of false association bias, thus failing to accurately uncover causal relationship information hidden in tweet features, this invention provides a multimodal title-text mismatch judgment method and system for WeChat official account tweets based on causal inference, which has the characteristic of low demand for scarce annotation resources.

[0007] To achieve the above-mentioned objectives of this invention, the technical solution adopted is as follows:

[0008] A multimodal method for identifying title-text mismatch in WeChat official account posts based on causal inference includes the following specific steps:

[0009] Obtain training tweets and extract multimodal features;

[0010] By dividing the training tweets into different scenarios, the effects of false association bias are simulated, and invariant factors that are discriminative under different scenarios and variable factors that describe the specific scenario effects are decoupled from multimodal features.

[0011] Based on the contrastive learning strategy, the variable factors are decoupled into causal factors that reflect the deceptive tactics and writing styles of mismatched tweets in different scenarios and irrelevant factors that contain false association bias.

[0012] Integrate invariant factors and causal factors;

[0013] Construct a classifier based on the fused factors; obtain additional training tweets and perform data augmentation; train the classifier using the augmented training tweets.

[0014] The trained classifier identifies the title and text of a tweet and determines whether the title and text do not match.

[0015] Preferably, the multimodal features include visual features, text features, cross-modal matching features, linguistic features, and statistical features.

[0016] Furthermore, the multimodal features of the tweets are extracted, and the specific steps are as follows:

[0017] Using the Swing Transformer pre-trained model as the basic framework, visual features of tweets are extracted through a sliding window-based self-attention mechanism.

[0018] The title and body of the tweet are segmented into words to obtain the embedding representation of each word, which is then input into the BERT pre-trained model to obtain text features;

[0019] Cross-modal matching features are obtained by using the CT Transformer pre-trained model to perform cross-modal matching between extracted visual features and text features;

[0020] Language features are based on cross-modal matching features, including text and abbreviations.Figure One Consistency features, consistency between body text and title, consistency between thumbnail and title, and title emotional polarity; the aforementioned body text and thumbnail Figure One The consistency feature is obtained by extracting the fusion features of the tweet's body text and thumbnail through the pre-trained CLIP model; the consistency feature between the body text and the title is obtained by inputting the body text and the title into a Siamese network; the consistency feature between the thumbnail and the title is obtained by inputting the tweet's thumbnail into a pre-trained text generator to generate a new title, and calculating the cosine similarity between the new title and the original title's BERT features; the title's sentiment polarity feature is obtained by inputting the tweet's title into a sentiment classifier.

[0021] The statistical features include lexical statistical features, common word statistical features, and author profile statistical features. The lexical statistical features are obtained by recording punctuation marks, emoticons, pronouns, affirmative words, and vague words appearing in the title of the tweet. The common word statistical features are obtained by recording internet slang, sensitive words, public figure names, and place names appearing in the title. The author profile statistical features are obtained by recording the author's account age, nickname, number of followers, number of followers, number of tweets published, number of low-quality tweets published in the past, the time since the first tweet was published, and the time since the last tweet was published.

[0022] By concatenating visual features, text features, cross-modal matching features, linguistic features, and statistical features, multidimensional multimodal features are obtained. .

[0023] Furthermore, by simulating the impact of spurious association bias, we decouple invariant factors that are discriminative across different scenarios and variable factors that describe the effects of specific scenarios from multimodal features. The specific steps are as follows:

[0024] Construct an invariant mask Through invariant mask Selecting multimodal features The dimension with universal discriminative power is used to obtain invariant features. ,in It is the element-wise multiplication operator;

[0025] right Inversely, extract context-dependent variable features. To simulate the effects of spurious association bias, training tweets were randomly assigned to training subsets for each scenario. The variable features are modeled by iterative scenario partitioning and sample redistribution until convergence, thus obtaining a scenario model;

[0026] Based on the scenario model, an invariant risk minimization loss function is constructed, and invariant features are learned on the invariant mask. The obtained invariant features are used as invariant factors, and the variable features are used as variable factors.

[0027] Furthermore, training tweets are randomly assigned to various scenario training subsets. The variable features are iteratively partitioned into scenarios and samples are redistributed until convergence is achieved, resulting in a scenario model. The specific steps are as follows:

[0028] Training tweets Divided into multiple scenario training subsets For each scenario Build parameters are Scenario prediction network ; Composed of a multi-layer sensing mechanism; according to Assess its likelihood in the context:

[0029]

[0030] Build A scenario prediction network is used to represent the characteristics of tweets influenced by different periods, deception types, and authors; sample reassignment is used to predict the characteristics of each scenario. Constructing a new training subset , will variable features The inputs are fed into each scenario prediction network, and the samples are assigned to the training set of the scenario with the highest likelihood to obtain the scenario model:

[0031]

[0032] in, These are the model parameters.

[0033] Furthermore, the loss function for minimizing invariant risk is specifically as follows:

[0034]

[0035] in, and To balance the parameters, Constrained by changing scenarios, In the scenario after modeling The corresponding training subset Classification loss on for Model parameters.

[0036] Furthermore, based on a contrastive learning strategy, variable factors are decoupled into causal factors reflecting the deceptive tactics and writing styles of misleading tweets in different contexts, and irrelevant factors containing false association bias. The specific steps are as follows:

[0037] Through multilayer perceptron The result obtained after learning invariant features Mapping to the latent embedding space yields the embedding encoding;

[0038] Using Gumbel-SoftMax technology, a causal mask is constructed based on the embedding encoding. ;

[0039] Will One dimension containing causal information is set to 1, and the remaining dimensions are set to 0; thus, the causal characteristics of a specific scenario are represented as follows: ;

[0040] Decoupling causal features effectively separates causal information from other irrelevant information. Causal intervention methods are used to impose comparative constraints on the decoupling process.

[0041]

[0042] in, It is a binary classification model for causal factors. After decoupling the model parameters, the resulting causal features are used as causal factors, and the remaining irrelevant features are excluded. Removed as an irrelevant factor.

[0043] Furthermore, additional training tweets are obtained and data augmentation is performed. The specific steps are as follows:

[0044] Unlabeled tweets are collected from public platforms. Based on social metadata, heuristic rules are designed and pseudo-labels are generated. Tweets that are retweeted more than a certain number of times per hour, but viewed for less than a certain number of seconds and have fewer than a set threshold of likes are designated as mislabeled samples, while the rest are normal samples, thus achieving data augmentation.

[0045] Furthermore, the classifier is trained using enhanced training tweets, specifically as follows:

[0046]

[0047] in, Represents a classifier. This represents the model parameters during training; during training, different scenarios are randomly set, and T rounds of training are performed.

[0048] A multimodal title-text mismatch detection system for WeChat official account posts based on causal inference includes a multimodal feature extraction module, an invariant factor extraction module, a scenario-specific causal factor decoupling module, and a prediction and data augmentation module.

[0049] The multimodal feature extraction module is used to acquire training tweets and extract multimodal features;

[0050] The invariant factor extraction module is used to divide the training tweets into different scenarios, simulate the impact of false association bias, and decouple invariant factors that are discriminative in different scenarios and variable factors that describe the specific scenario effects from multimodal features.

[0051] The specific scenario causal factor decoupling module is used to decouple variable factors into causal factors that reflect the deceptive tactics and writing styles of mismatched tweets in different scenarios and irrelevant factors containing false association bias based on a contrastive learning strategy.

[0052] The prediction and data augmentation module is used to fuse invariant factors and causal factors; construct a classifier based on the fused factors; obtain additional training tweets and perform data augmentation, then train the classifier using the augmented training tweets; use the trained classifier to identify the title and text of the tweets; and determine whether the title and text do not match.

[0053] The beneficial effects of this invention are as follows:

[0054] This invention discloses a multimodal title-text mismatch discrimination method for WeChat official account tweets based on causal inference. This method extracts multimodal features, providing rich evidence for title-text mismatch discrimination. It decouples multimodal features into invariant factors reflecting the author's intent and possessing discriminative power in different contexts. Based on a contrastive learning strategy, variable factors are decoupled into three latent factors: causal factors reflecting the deceptive tactics and writing style of title-text mismatch tweets in different contexts, irrelevant factors containing false association bias, and irrelevant factors reflecting false association bias. This allows for fine-grained characterization of the causal information of deceptive behavior, modeling false association bias, accurately uncovering causal relationship information hidden in tweet features, and exhibiting high generalization ability. Furthermore, this method constructs a classifier based on the fused factors, acquires additional training tweets, and performs data augmentation, thus having low requirements for scarce annotation resources. Attached Figure Description

[0055] Figure 1 This is a schematic diagram illustrating the specific process of a multimodal title-text mismatch judgment method for WeChat official account posts based on causal judgment, according to the present invention.

[0056] Figure 2 This is a schematic diagram of the causal structure of the title-text mismatch judgment task in the multimodal title-text mismatch judgment method for WeChat public account posts based on causal judgment of the present invention.

[0057] Figure 3 This is a schematic diagram of the system flow of a multimodal title-text mismatch judgment system for WeChat official account posts based on causal judgment, according to the present invention. Detailed Implementation

[0058] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0059] Example 1

[0060] like Figure 1 As shown, the multimodal title-text mismatch detection method for WeChat official account posts based on causal inference includes the following specific steps:

[0061] Obtain training tweets and extract multimodal features;

[0062] By dividing the training tweets into different scenarios, the effects of false association bias are simulated, and invariant factors that are discriminative under different scenarios and variable factors that describe the specific scenario effects are decoupled from multimodal features.

[0063] Based on the contrastive learning strategy, the variable factors are decoupled into causal factors that reflect the deceptive tactics and writing styles of mismatched tweets in different scenarios and irrelevant factors that contain false association bias.

[0064] Integrate invariant factors and causal factors;

[0065] Construct a classifier based on the fused factors; obtain additional training tweets and perform data augmentation; train the classifier using the augmented training tweets.

[0066] The trained classifier identifies the title and text of a tweet and determines whether the title and text do not match.

[0067] Unlike traditional methods, this approach analyzes the intrinsic factors leading to deceptive behavior from a novel perspective of causal representation learning, thereby eliminating false association bias and ensuring the robustness and generalization of the model. To comprehensively characterize tweet information, this patent first extracts multiple types of features, such as text modality features, visual modality features, linguistic features, cross-modal features, and author profile features. These features are intertwined, containing redundant information and unobservable false association bias, which can introduce noise during machine judgment. Therefore, this method further decouples three latent factors from the multimodal features: an invariant factor reflecting the author's intent and possessing discriminative power across different contexts; a causal factor reflecting the deceptive tactics and writing style of tweets with mismatched titles and content in different contexts based on a contrastive learning strategy; an irrelevant factor containing false association bias; and an irrelevant factor reflecting false association bias. In decoupling these three latent factors, the invariant factor and the causal factor, which contain causal information, are integrated to construct a classifier that balances robustness and generalization, effectively addressing the interference of false association bias and new variant tweets. To improve model training efficiency, this method employs data augmentation techniques to expand the training samples, reducing the need for scarce labeled data. In this embodiment, the method is applied to the WeChat Official Accounts platform, automatically identifying posts with mismatched titles and text and promptly blocking their dissemination. This demonstrates significant value in enhancing user experience, platform credibility, and purifying the online environment.

[0068] Example 2

[0069] In this embodiment, given a tweet The present invention aims to establish a binary classification model. ,according to In the context of determining whether there is a mismatch between the title text, cover image, and body text, various modal information is used. ),in These are the parameters of the model. For tweets with mismatched titles and content, each modal component contains malicious information and exhibits inconsistencies in content, such as having eye-catching titles and thumbnails that are irrelevant to the main text. Furthermore, malicious authors constantly create new variant tweets to cater to trends and evade detection. Therefore, the goal of this method is to optimize the training set... Minimize the cross-entropy classification loss The parameters obtained after optimization Enable the model The test set is constructed from samples not seen during training. It has excellent performance.

[0070] Existing methods often directly capture tweet features The feature is determined by the co-occurrence of tags that do not match the text. This feature is derived from an encoder adapted for general content understanding tasks and cannot accurately characterize deceptive behavior. It varies depending on the time period, type of deception, and author context. The following are characteristics of deceptive behavior, which exhibit specific writing styles and deceptive tactics. Existing methods do not discuss deceptive behavior in different scenarios, leading to... The model suffers from insufficient robustness due to the presence of redundant information and spurious association bias. To address this issue, this method decomposes multimodal features into three latent factors, including invariant features. Specific Contextual Causal Characteristics and irrelevant features ,in and This includes both identifiable causal information specific to the context in which the tweet is posted and biased noise. For example... Figure 2 As shown, this method uses causal inference techniques to block and , and The backdoor path of association is used to eliminate spurious association bias. In this diagram, double-headed arrows represent statistical association; single-headed arrows represent causal relationships; purple arrows represent key causal relationships affecting the judgment result; orange arrows represent situational effects; and scissors represent relationship removal.

[0071] In one specific embodiment, the multimodal features include visual features, text features, cross-modal matching features, linguistic features, and statistical features.

[0072] In one specific embodiment, the multimodal features of the tweet are extracted, and the specific steps are as follows:

[0073] The tweet mainly contains two types of visual information: cover image and text illustration. The Swin Transformer pre-trained model is used as the basic framework, and the visual features of the tweet are extracted through a sliding window-based self-attention mechanism. The SwinTransformer uses the Transformer pre-trained model as the basic framework and extracts the hierarchical information of the image through a sliding window-based self-attention mechanism.

[0074] In this embodiment, in practical applications, a significant portion of tweets involve public figures, and most of the images contain human figures; therefore, facial recognition and object detection are performed using DNN and RetinaNet networks respectively.

[0075] The title and body of the tweet are segmented into words to obtain the embedding representation of each word. This embedding representation is then input into the BERT pre-trained model to obtain a set of word features with a dimension of 768.

[0076] In this embodiment, considering that there may be deceptive text in the cover image, OCR technology is also used to extract and encode such text.

[0077] In this embodiment, tweets with mismatched titles and texts often exhibit inconsistencies in content; for example, the cover image may contain an object, but the text may not mention it. Therefore, consistency among the various parts of a tweet is one of the important criteria for identifying title-text mismatches. Since the different parts of the information belong to different modalities, there is a heterogeneous gap between the corresponding features, making direct unified modeling impossible. A CT Transformer pre-trained model is used to perform cross-modal matching between extracted visual and textual features to obtain cross-modal matching features.

[0078] In this embodiment, a CT Transformer pre-trained model is used to perform cross-modal matching between the extracted visual features and text features to obtain cross-modal matching features. Specifically, the extracted visual features and text features are encoded, and then a multi-head attention mechanism is used to output the text-aware visual features. and text features of visual perception ,in , This is the weight matrix. Considering that cross-modal matching features rely on a multi-head attention mechanism, the model can characterize deceptive behavior from multiple perspectives.

[0079] Language features are based on cross-modal matching features, including text and abbreviations. Figure One Consistency characteristics, consistency between body text and title, consistency between thumbnail and title, and emotional polarity of title;

[0080] Using visually striking cover images is a common tactic used by malicious authors when crafting tweets with titles that don't match the content; however, these cover images often have no connection to the actual content of the text. (The text and abbreviation mentioned above...) Figure One Consistency features are extracted from the main text using the pre-trained model CLIP. and thumbnails Fusion characteristics To capture this cross-modal inconsistency, where the CLIP outputs are respectively Text features and thumbnail features;

[0081] Malicious authors often create information gaps between the title and the reader's prior knowledge to trick them into clicking on tweet links and filling a gap in their curiosity. However, the content of these tweets is usually of poor quality and does not align with the title. To measure whether there is a discrepancy between the content of the body and the title, the consistency features between the body and the title are obtained by inputting the body and the title into a Siamese network. ;

[0082] Inconsistencies between the title and cover image can entice readers to click on the tweet link. To determine if there is a content conflict between the thumbnail and the title, a text generator pre-trained on the MS COCO dataset is used. The tweet thumbnail is input into the text generator to generate a new title, and the cosine similarity of the BERT features of the new title and the original title is calculated to obtain the thumbnail-title consistency feature.

[0083] Deceptive headlines typically rely on extreme emotional tone to evoke reader resonance and thus pique their curiosity. The aforementioned headline sentiment polarity feature is obtained by inputting the tweet's headline into a sentiment classifier; the headline sentiment polarity feature is a 2-dimensional feature, representing sentiment polarity (positive / neutral / negative) and intensity value, with each dimension's value standardized to the range [0, 1].

[0084] Statistical features include lexical statistical features, common word statistical features, and author profile statistical features;

[0085] The lexical statistical features mentioned above are obtained by recording punctuation marks such as "!", "?", "~", emoticons, pronouns, affirmative words, and vague words that appear in the titles of tweets;

[0086] The statistical features of commonly used words are obtained by recording online terms, sensitive words, public figures' names, and place names that appear in the title; a statistical feature of commonly used words is also constructed to record the frequency of these commonly used words.

[0087] Author profiles can reflect the quality of tweets to some extent. If an author has a history of frequently posting tweets with mismatched titles and content, they can be considered a malicious author, and their subsequent tweets are highly likely to be deceptive. To characterize each author... Whether the tweet was malicious was determined by recording the author's account age, nickname, number of followers, number of followers, number of tweets posted, number of low-quality tweets posted in the past, the time since the first tweet was posted, and the time since the last tweet was posted.

[0088] By concatenating visual features, text features, cross-modal matching features, linguistic features, and statistical features, multidimensional multimodal features are obtained. .

[0089] In this embodiment, multimodal features It is a high-dimensional phenomenon containing unobservable spurious association bias. This noise interferes with the model's judgment, reducing robustness and generalization. Spurious association bias arises because the model fails to model the impact of different time periods, deception types, and author characteristics during training. During training, tweets are divided into different contextual subsets, and common causal information is captured within each context. Invariant features that possess discriminative properties This aims to endow the model with stable generalization ability, meaning it has considerable accuracy in distinguishing between seen and unseen samples during training. The training set... Divided into multiple scenario training subsets Subsequently, the invariant feature learning stage decouples invariant features from multimodal features, extracting features that exhibit good discriminative performance across various training subsets. and the variable characteristics of the response scenario. The scenario modeling phase is based on This allows for the prediction of more suitable contextual sub-training sets for tweets, which in turn further facilitates the invariant feature learning stage. By alternately optimizing the decoupling and prediction processes described above, invariant features... It will have stable generalization ability.

[0090] In one specific embodiment, the impact of spurious association bias is simulated, and invariant factors that are discriminative under different scenarios and variable factors describing the specific scenario effects are decoupled from multimodal features. The specific steps are as follows:

[0091] Construct an invariant mask Through invariant mask Selecting multimodal features The dimension with universal discriminative power is used to obtain invariant features. ,in It is the element-wise multiplication operator; It can make consistent predictions across various scenarios;

[0092] Context is a summary of the impact of different periods, deception types, and authors. Tweets created in specific contexts have specific writing styles, topics, and deception techniques; therefore, the quality of context modeling is crucial for invariant masks. The optimization plays a decisive role; for Inversely, extract context-dependent variable features. To simulate the effects of spurious association bias, training tweets were randomly assigned to training subsets for each scenario. The variable features are modeled through iterative scenario partitioning and sample redistribution until convergence, resulting in a scenario model.

[0093] Based on the scenario model, an invariant risk minimization loss function is constructed, and invariant features are learned on the invariant mask. The obtained invariant features are used as invariant factors, and the variable features are used as variable factors.

[0094] In one specific embodiment, training tweets are randomly assigned to various scenario training subsets. The variable features are iteratively partitioned into scenarios and samples are redistributed until convergence is achieved, resulting in a scenario model. The specific steps are as follows:

[0095] Training tweets Divided into multiple scenario training subsets For each scenario Build parameters are Scenario prediction network ; Composed of a multi-layer sensing mechanism; according to Assess its likelihood in the context:

[0096]

[0097] Build A scenario prediction network is used to represent the characteristics of tweets influenced by different periods, deception types, and authors; sample reassignment is used to predict the characteristics of each scenario. Constructing a new training subset , will variable features The inputs are fed into each scenario prediction network, and the samples are assigned to the training set of the scenario with the highest likelihood to obtain the scenario model:

[0098]

[0099] in, These are the model parameters.

[0100] In one specific embodiment, the loss function for minimizing invariant risk is as follows:

[0101]

[0102] in, and To balance the parameters, Constrained by changing scenarios, In the scenario after modeling The corresponding training subset Classification loss on for Model parameters.

[0103] In this embodiment, the extracted invariant features It possesses unbiased discriminative ability applicable to various scenarios, while variable features contain a mixture of causal information reflecting scenario characteristics, multimodal redundant information, and unobservable spurious association bias. The causal information can further enrich the discriminative criteria, endowing the model with the ability to generalize to various scenarios.

[0104] In one specific embodiment, a contrastive learning strategy is used to decouple variable factors into causal factors reflecting the deceptive tactics and writing styles of misleading tweets in different scenarios, and irrelevant factors containing false association bias. The specific steps are as follows:

[0105] Through multilayer perceptron The result obtained after learning invariant features Mapping to the latent embedding space yields the embedding encoding;

[0106] In design In terms of structure, this patent does not incorporate any contextual information. Therefore, when the model proposed in this patent is deployed in a new scenario, no reconstruction is required. Or relearn the parameters;

[0107] Using Gumbel-SoftMax technology, a causal mask is constructed based on the embedding encoding. ;

[0108] Will One dimension containing causal information is set to 1, and the remaining dimensions are set to 0; thus, the causal characteristics of a specific scenario are represented as follows: ;

[0109] Decoupling causal features effectively separates causal information from other irrelevant information. Causal intervention methods are used to impose comparative constraints on the decoupling process.

[0110]

[0111] in, It is a binary classification model for causal factors. After decoupling the model parameters, the resulting causal features are used as causal factors, and the remaining irrelevant features are excluded. Removed as irrelevant factors. For tweets that do not match the title ( Irrelevant features The causal information of the discrepancy between the response and the text is masked; when this feature is used as a substitute... When making a judgment, the judgment result should be changed, that is, it should be identified as a normal tweet; for normal tweets ( ), No deceptive information was detected, therefore even if it is switched to The prediction will not change the judgment result.

[0112] In one specific embodiment, the classifier is trained using enhanced training tweets, specifically as follows:

[0113]

[0114] in, Represents a classifier. This represents the model parameters during training; during training, different scenarios are randomly set, and T rounds of training are performed.

[0115] In this embodiment, the deep learning model requires a large amount of scarce labeled sample resources for training. However, existing datasets have significant shortcomings in terms of scale and coverage, leading to difficulties and high costs in actual deployment. Therefore, in a specific embodiment, additional training tweets are obtained and data augmentation is performed. The specific steps are as follows:

[0116] We collected a large number of unlabeled tweets from the WeChat Official Accounts platform. Based on social metadata, we designed heuristic rules and generated pseudo-tags. The heuristic rules were designed to designate tweets that were forwarded more than 100,000 times per hour, but viewed for less than 10 seconds and had 0 likes as mismatched samples, while the rest of the tweets were normal samples, thus achieving data augmentation.

[0117] In this embodiment, to evaluate the effectiveness and superiority of the proposed method, a performance comparison experiment was conducted with five mainstream methods, including SVM-TS, BiLSTM, dEFEND, HPFN, and VLP. Evaluation metrics included accuracy, precision, recall, and F1 score. First, 70,794 WeChat public account posts were crawled, and pseudo-labels were generated for these posts using the data augmentation techniques mentioned in Unit 104, resulting in 32,418 posts with mismatched titles and 38,376 normal posts. These pseudo-labeled samples were used as training datasets to optimize the patented model and the comparison model. To construct the test dataset, an additional 8,327 WeChat public account posts were crawled, and 3,794 posts with mismatched titles and 4,533 normal posts were obtained through manual annotation. Experimental results show that the proposed method significantly outperforms mainstream methods; the pseudo-labeled data generated by the designed data augmentation method positively promotes the model training process and effectively improves model performance.

[0118] Furthermore, in this embodiment, to verify the reliability of the proposed method, the distribution of multimodal features, invariant features, context-specific causal features, and irrelevant features of some test samples were analyzed. After visualizing these sample features using t-SNE dimensionality reduction, the following conclusions were drawn: (a) Multimodal features are ineffective in identifying title-text mismatch; (b) Invariant features can better distinguish between title-text mismatch and normal tweet samples, but cannot determine the context to which the tweet belongs; (c) Context-specific causal features have the ability to distinguish the context to which a sample belongs and to identify title-text mismatch; (d) Irrelevant features, while capable of distinguishing the context to which a sample belongs, do not contain any causal information that can identify title-text mismatch. In summary, this method can accurately summarize the contextual characteristics of tweets and output reliable discrimination results by combining invariant features and context-specific causal features.

[0119] In summary, the method proposed in this invention has the following advantages compared to the prior art:

[0120] 1. Multiple modal features, such as text modal features, visual modal features, linguistic features, cross-modal features, and author profile features, were extracted, providing rich evidence for the judgment of title-text mismatch;

[0121] 2. Multimodal features are decoupled into multiple latent factors, including causal information that can finely characterize deceptive behavior; scenarios are used to summarize the impact of different periods, deceptive types, and authors, and the decoupling and scenario prediction processes are alternately optimized to model the scenarios. By analyzing the causal effects of specific scenarios, the model achieves high generalization ability.

[0122] 3. In specific scenarios, contrastive loss is introduced to constrain the causal inference process, accurately eliminating irrelevant factors mixed with redundant information and spurious association bias, thereby improving the robustness of the model; data augmentation techniques are also used to expand the training samples, alleviating the problem of the demand for scarce labeled resources in the model training process.

[0123] 4. The method proposed in this invention can be successfully deployed in demanding real-world business scenarios such as those involving false association bias, multimodal content, constantly evolving variant tweets, and scarce annotation resources. It can significantly improve the user experience of WeChat official accounts, enhance platform credibility, and purify the online environment.

[0124] Example 3

[0125] like Figure 3 As shown, the WeChat official account article multimodal title-text mismatch judgment system based on causal inference includes a multimodal feature extraction module, an invariant factor extraction module, a specific scenario causal factor decoupling module, and a prediction and data augmentation module.

[0126] The multimodal feature extraction module is used to acquire training tweets and extract multimodal features;

[0127] The invariant factor extraction module is used to divide the training tweets into different scenarios, simulate the impact of false association bias, and decouple invariant factors that are discriminative in different scenarios and variable factors that describe the specific scenario effects from multimodal features.

[0128] The specific scenario causal factor decoupling module is used to decouple variable factors into causal factors that reflect the deceptive tactics and writing styles of mismatched tweets in different scenarios and irrelevant factors containing false association bias based on a contrastive learning strategy.

[0129] The prediction and data augmentation module is used to fuse invariant factors and causal factors; construct a classifier based on the fused factors; obtain additional training tweets and perform data augmentation, then train the classifier using the augmented training tweets; use the trained classifier to identify the title and text of the tweets; and determine whether the title and text do not match.

[0130] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A multimodal title-text mismatch detection method for WeChat official account posts based on causal inference, characterized by: The specific steps include the following: Obtain training tweets and extract multimodal features; By dividing the training tweets into different scenarios, the effects of false association bias are simulated, and invariant factors that are discriminative under different scenarios and variable factors that describe the specific scenario effects are decoupled from multimodal features. To simulate the impact of spurious association bias, the following steps are taken to decouple invariant factors that are discriminative across different scenarios and variable factors that describe the effects of specific scenarios from multimodal features: Construct an invariant mask Through invariant mask Selecting multimodal features The dimension with universal discriminative power is used to obtain invariant features. ,in It is the element-wise multiplication operator; right Inversely, extract context-dependent variable features. ; To simulate the effects of spurious association bias, training tweets were randomly assigned to training subsets for each scenario. The variable features are modeled by iterative scenario partitioning and sample redistribution until convergence, thus obtaining a scenario model; Training tweets were randomly assigned to each scenario training subset. The variable features are iteratively partitioned into scenarios and samples are redistributed until convergence is achieved, resulting in a scenario model. The specific steps are as follows: Training tweets Divided into multiple scenario training subsets For each scenario Build parameters are Scenario prediction network ; Composed of a multi-layer sensing mechanism; according to Assess its likelihood in the context: Build A scenario prediction network is used to represent the characteristics of tweets influenced by different periods, deception types, and authors; sample reassignment is used to predict the characteristics of each scenario. Constructing a new training subset , will variable features The inputs are fed into each scenario prediction network, and the samples are assigned to the training set of the scenario with the highest likelihood to obtain the scenario model: in, These are model parameters; Based on the scenario model, an invariant risk minimization loss function is constructed, and invariant features are learned on the invariant mask. The obtained invariant features are used as invariant factors, and the variable features are used as variable factors. Based on the contrastive learning strategy, the variable factors are decoupled into causal factors that reflect the deceptive tactics and writing styles of tweets that do not match the content in different scenarios, and irrelevant factors that contain false association bias. Integrate invariant factors and causal factors; Construct a classifier based on the fused factors; obtain additional training tweets and perform data augmentation; train the classifier using the augmented training tweets. The trained classifier identifies the title and text of a tweet and determines whether the title and text do not match.

2. The method for determining multimodal title-text mismatch in WeChat official account posts based on causal inference as described in claim 1, characterized in that: The multimodal features include visual features, text features, cross-modal matching features, linguistic features, and statistical features.

3. The method for determining multimodal title-text mismatch in WeChat official account posts based on causal inference as described in claim 2, characterized in that: The specific steps for extracting multimodal features from tweets are as follows: Using the Swin Transformer pre-trained model as the basic framework, visual features of tweets are extracted through a sliding window-based self-attention mechanism. The title and body of the tweet are segmented into words to obtain the embedding representation of each word, which is then input into the BERT pre-trained model to obtain text features; Cross-modal matching features are obtained by using the CT Transformer pre-trained model to perform cross-modal matching between extracted visual features and text features; The language features are based on cross-modal matching features, including text-to-thumbnail consistency features, text-to-title consistency features, thumbnail-to-title consistency features, and title sentiment polarity features. The text-to-thumbnail consistency features are obtained by extracting the fusion features of the tweet's text and thumbnails using the pre-trained CLIP model. The text-to-title consistency features are obtained by inputting the text and title into a Siamese network. The thumbnail-to-title consistency features are obtained by using a pre-trained text generator, inputting the tweet's thumbnail into the text generator to generate a new title, and calculating the cosine similarity between the new title and the original title's BERT features. The title sentiment polarity features are obtained by inputting the tweet's title into a sentiment classifier. Statistical features include lexical statistical features, common word statistical features, and author profile statistical features; The lexical statistical features are obtained by recording punctuation marks, emoticons, pronouns, affirmative words, and vague words appearing in the titles of tweets; the common word statistical features are obtained by recording internet slang, sensitive words, public figure names, and place names appearing in the titles; the author profile statistical features are obtained by recording the author's account age, nickname, number of followers, number of followers, number of tweets published, number of low-quality tweets published in the past, the time since the first tweet was published, and the time since the last tweet was published. By concatenating visual features, text features, cross-modal matching features, linguistic features, and statistical features, multidimensional multimodal features are obtained. .

4. The method for determining multimodal title-text mismatch in WeChat official account posts based on causal inference as described in claim 1, characterized in that: The loss function for minimizing constant risk is as follows: in, and To balance the parameters, This represents a binary classification model based on invariant masks. Constrained by changing scenarios, In the scenario after modeling The corresponding training subset Classification loss on for Model parameters.

5. The method for determining multimodal title-text mismatch in WeChat official account posts based on causal inference as described in claim 4, characterized in that: Based on a contrastive learning strategy, variable factors are decoupled into causal factors reflecting the deceptive tactics and writing styles of misleading tweets in different scenarios, and irrelevant factors containing false association bias. The specific steps are as follows: Through multilayer perceptron The result obtained after learning invariant features Mapping to the latent embedding space yields the embedding encoding; Using Gumbel-SoftMax technology, a causal mask is constructed based on the embedding encoding. ; Will One dimension containing causal information is set to 1, and the remaining dimensions are set to 0; thus, the causal characteristics of a specific scenario are represented as follows: ; Decoupling causal features effectively separates causal information from other irrelevant information. Causal intervention methods are used to impose comparative constraints on the decoupling process. in, It is a binary classification model for causal factors. After decoupling the model parameters, the resulting causal features are used as causal factors, and the remaining irrelevant features are excluded. Removed as an irrelevant factor.

6. The method for determining multimodal title-text mismatch in WeChat official account posts based on causal inference as described in claim 1, characterized in that: To obtain additional training tweets and perform data augmentation, the specific steps are as follows: Unlabeled tweets are collected from public platforms. Based on social metadata, heuristic rules are designed and pseudo-labels are generated. Tweets that are retweeted more than a certain number of times per hour, but viewed for less than a certain number of seconds and have fewer than a set threshold of likes are designated as mislabeled samples, while the rest are normal samples, thus achieving data augmentation.

7. The method for determining multimodal title-text mismatch in WeChat official account posts based on causal inference as described in claim 6, characterized in that: The classifier is trained using tweets with enhanced training, specifically as follows: in, Represents a classifier. This represents the model parameters during training; during training, different scenarios are randomly set, and T rounds of training are performed.

8. A multimodal title-text mismatch detection system for WeChat official account posts based on causal inference, characterized by: The method for implementing the method as described in any one of claims 1 to 7 includes a multimodal feature extraction module, an invariant factor extraction module, a scenario-specific causal factor decoupling module, and a prediction and data augmentation module. The multimodal feature extraction module is used to acquire training tweets and extract multimodal features; The invariant factor extraction module is used to divide the training tweets into different scenarios, simulate the impact of false association bias, and decouple invariant factors that are discriminative in different scenarios and variable factors that describe the specific scenario effects from multimodal features. The specific scenario causal factor decoupling module is used to decouple variable factors into causal factors that reflect the deceptive tactics and writing styles of mismatched tweets in different scenarios and irrelevant factors containing false association bias based on a contrastive learning strategy. The prediction and data augmentation module is used to fuse invariant factors and causal factors; and to construct a classifier based on the fused factors. Acquire additional training tweets and perform data augmentation; train a classifier using the augmented training tweets; use the trained classifier to identify the title and text of the tweets. Determine whether the title and text are inconsistent.

Images