Microblog-oriented information popularity evaluation and multi-mode false information detection method

Abstract

The invention provides a microblog-oriented information popularity evaluation and multi-modal false information detection method, which comprises the following steps: acquiring news data released by a target official news account on a microblog platform, performing data cleaning and processing to form a news corpus, and screening out news samples containing image and text modals; selecting news data within a preset time range before the target information is released to construct a hot news time window; performing feature learning on the image and text modals in the time window based on pre-trained ResNet50 and BERT models, realizing bimodal feature fusion through a collaborative attention mechanism, and generating multi-modal feature representation of target information; based on the multi-modal feature representation, calculating the similarity between the target information and the news corpus in the time window, and evaluating whether the target information has popularity or not through a similarity threshold value; through popularity and novelty analysis, correlation and difference between target information and a time window and between the target information and a high-similarity space window are learned in combination with a kernel function, and finally target information authenticity detection is completed.

Description

Technical Field

[0001] The present application relates to the technical field of false information detection, and in particular to an information heat evaluation and multimodal false information detection method for microblogs. Background Art

[0002] Driven by the digital age, social networking platforms have profoundly influenced the public's lifestyle with their real-time and convenience. They have not only changed people's way of obtaining and exchanging information, but also significantly reshaped the path of information dissemination and reception. However, the review mechanism of social platforms is mostly focused on identifying political or inappropriate content, while the review of false information is insufficient, resulting in the widespread dissemination of false information. Studies have shown that false information spreads much faster than real information on social media platforms, and its spread speed can even reach six times that of real information. At the same time, in the process of information dissemination, the original information is easily misinterpreted or tampered with, resulting in multiple versions of false information, which quickly spread in areas of great public concern, including politics, economy, health, technology, and entertainment. The rapid spread of this false information not only misleads public cognition, threatens information security and property security, but also has a serious impact on the credibility of mainstream media. Therefore, the development of efficient false information detection technology, the use of machine learning and deep learning methods to replace manual verification to a certain extent, and reduce human and financial costs have become key issues that need to be solved urgently.

[0003] Existing false information detection technology has gone through a development process from analyzing information content to mining potential connections between information. Early research mainly focused on unimodal content feature extraction, such as using the intrinsic properties of text or images to judge authenticity. With the development of technology, research began to explore the potential relationship between information. For example, learning rumor representations between texts through generative adversarial networks, or mining the hidden differences between true and false news in linguistics, emotional expression, and writing style through encoding-decoding models. In addition, the combination of structured and unstructured data has also become an important direction, such as using encyclopedia knowledge bases to verify the authenticity of text and tabular information, or by constructing a collection of verified facts in a specific field to assist in true and false detection. Some studies have also introduced external fact bases, such as retrieving relevant articles through search engines and combining heterogeneous graph neural network models to analyze the relationship between facts. However, these methods perform well in specific situations, but their limitations are gradually emerging.

[0004] As information forms shift towards multimodality, multimodal false information detection technology has received increasing attention. Some studies classify false information using neural network models by performing feature conversion on text and images respectively and splicing multimodal representations; some studies try to extract similar features between text and images by analyzing the semantic consistency between news images and titles; and some studies introduce knowledge graph modules based on bimodal fusion to improve the generalization performance of detection using external knowledge. However, while these studies have made technical progress, they have also exposed some problems that need to be solved urgently.

[0005] First, existing technologies often lag behind. False information spreads extremely quickly and widely, but detection technology is difficult to quickly identify and effectively contain in the early stages of information diffusion. For example, although external knowledge bases are highly authoritative, they update content slowly and have limited effectiveness in distinguishing the authenticity of recent information. Second, existing methods mostly focus on a single modality or simple multimodal fusion, failing to fully explore the complex connections between information, especially failing to effectively integrate the potential connections between information content and external facts, making it difficult to further improve detection accuracy and robustness.

[0006] Therefore, how to build a false information detection technology that can timely and comprehensively capture the complex relationship between the internal characteristics of information and the external environment has become an important issue that needs to be solved urgently. Summary of the invention

[0007] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0008] To this end, the first purpose of this application is to propose an information heat evaluation and multimodal false information detection method for Weibo.

[0009] The second purpose of this application is to propose an information heat evaluation and multimodal false information detection device for Weibo.

[0010] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a method for evaluating information popularity and detecting multimodal false information for Weibo, including:

[0011] Collecting news data published by a target official news account on the Weibo platform, preprocessing the news data to form a news corpus, and screening news samples containing image and text modalities from the news corpus;

[0012] Selecting a news collection within a preset time before the target information is released as a time window, wherein the news samples in the time window are obtained by screening the news corpus;

[0013] Based on the pre-trained ResNet50 and BERT models, feature learning is performed on the image and text modalities of the news samples in the time window, and a collaborative attention mechanism is used to achieve dual-modal feature fusion to obtain a multi-modal feature representation of the target information;

[0014] Based on the multimodal feature representation, calculating the similarity between the target information and the news sample in the time window, and determining whether the target information has popularity by setting a similarity threshold;

[0015] Based on popularity and novelty analysis, the first several news samples related to the target information are selected to form a spatial window, and the correlation and difference between the target information and the time window and the spatial window are analyzed through the kernel function to realize the authenticity detection of the target information.

[0016] Optionally, collecting news data published by a target official news account on the Weibo platform and preprocessing the news data to form a news corpus includes:

[0017] Regularly capture all news data published by the target official news account on the Weibo platform through automated collection methods. The news content includes image and text modalities.

[0018] It is defined that each news sample consists of a long text t and an image i. For news samples with more than one image, the images they contain are spliced, and the text content is cleaned to remove irrelevant content and data missing samples to form a news corpus.

[0019] Optionally, for a news sample having more than one image, splicing the images contained therein includes:

[0020] Calculate the number of images n in the news sample;

[0021] Calculate the square root of n and round down to get And define the number of rows and columns as:

[0022]

[0023] Splice the pictures and stitch the first s*s pictures into a complete square. If n>s 2 , arrange the remaining images in the order of row and col. If the last row is not completely filled, the remaining parts are set to all 0s;

[0024] The stitched images are compressed and the output size is 900×900.

[0025] Optionally, the selecting a news collection within a preset time before the target information is released as a time window, and the news samples in the time window are obtained by screening the news corpus, including:

[0026] Get the release time t of the target information p ;

[0027] According to t p and the release time t of each news sample in the news corpus e The time difference is 0. <t p -t e ≤T news samples, define the time window ε time For all news sample sets that meet the conditions, the formula is as follows:

[0028] ε time ={e:e∈ε,0 <t p -t e ≤T}

[0029] Where ε represents all news samples in the news corpus, ε represents the release time of the target information p, and t e represents the release time of news sample e, and T is the preset time.

[0030] Optionally, the pre-trained ResNet50 and BERT models are used to perform feature learning on the image and text modalities of the news sample in the time window, respectively, including:

[0031] For the image modality, the pre-trained ResNet50 model is used as the embedding model to obtain the embedding representation R of the image modality. i ; Through the fully connected layer W i Embedding the image R i Map it to the same preset feature dimension d as the text modality to obtain the image feature matrix. The formula is as follows:

[0032] E i =σ(W i *R i +d)

[0033] Among them, d is the bias vector, σ is the activation function, E t is the image feature matrix;

[0034] For the text modality, the pre-trained BERT model is used as the embedding model to obtain the embedding representation R of the text modality. t ; Through the fully connected layer W t For text embedding representation R t Map it to the preset feature dimension d to obtain the text feature matrix. The formula is as follows:

[0035] E t =σ(W t *R t +b)

[0036] Among them, b is the bias vector, σ is the activation function, E t is the text feature matrix.

[0037] Optionally, the adopting a collaborative attention mechanism to fuse the text features and the image features to obtain a multimodal feature representation of the target information includes:

[0038] In the collaborative attention mechanism, there are two parallel collaborative attention modules Co-Att1 and Co-Att2, where:

[0039] In Co-Att1, text features are used as queries and image features are used as keys and values. The attention score is calculated to achieve the attention of the text modality to the image modality, and the weighted representation of the image modality is obtained by weighted summation.

[0040] In Co-Att2, the image features are used as queries and the text features are used as keys and values. The attention score is calculated to realize the attention of the image modality to the text modality, and the weighted representation of the text modality is obtained by weighted summation.

[0041] The weighted representation of the text modality to the image modality in Co-Att1 is concatenated with the weighted representation of the image modality to the text modality in Co-Att2 to obtain the multimodal feature representation of the target information. The formula is as follows:

[0042]

[0043] Among them, E ti is the multimodal feature representation of the target information, E t is the text modality feature matrix, E i is the image modality feature matrix, d t and d i are the feature embedding dimensions for text and image modalities, respectively.

[0044] Optionally, the calculating the similarity between the target information and the news sample in the time window based on the multimodal feature representation, and determining whether the target information has popularity by setting a similarity threshold, includes:

[0045] The cosine similarity between the target information and each news sample in the time window is calculated respectively, and the formula is as follows:

[0046]

[0047] Where p is the multimodal feature representation of the target information, e i is the multimodal feature representation of the i-th news sample in the time window;

[0048] Arrange the calculated similarity values ​​in descending order, and select the similarity values ​​of the first α proportions to form a set S top , the formula is as follows:

[0049] S top =Top(α)(S)

[0050] Wherein, S represents the set of similarity values ​​between the target information and all news samples in the time window, and Top(α)(S) represents the subset with the top α proportions after the similarity values ​​are arranged in descending order;

[0051] For the set S top The similarity values ​​in are counted to determine whether the proportion of the similarity values ​​greater than the preset threshold β exceeds 50%. If the condition is met, the target information is determined to be hot information, otherwise it is non-hot information. The formula is as follows:

[0052]

[0053] Among them, ∏(cos(p,e i )>β) is the judgment function. When the similarity value is greater than the threshold β, the result is 1, otherwise it is 0.

[0054] Optionally, the method of selecting the first several news samples related to the target information to form a high similarity space window based on popularity and novelty analysis, and analyzing the correlation and difference between the target information and the time window and the space window through a kernel function to detect the authenticity of the target information includes:

[0055] In the time window ε time Based on this, we define the spatial window ε space , the spatial window represents a set of news most relevant to the target information, and the formula is as follows:

[0056] ε space =Top(k)(ε time )

[0057] k=max(k,len(ε time ))

[0058] Among them, k is a hyperparameter, which means selecting the first k most relevant news samples in the time window, and its value is the given k and the time window ε time The maximum number of vectors;

[0059] Through popularity analysis, the similarity distribution between the target information and all news samples in the time window is calculated, and the cosine similarity is pooled using the Gaussian kernel function. The formula is as follows:

[0060]

[0061] Among them, K j is the output of the jth kernel, μ j is the center value of the kernel, σ j is the width of the kernel, n is the time window ε time The number of news samples in the

[0062] The outputs of all cores are vectorized and normalized to obtain the similarity distribution of the target information in the time window. The formula is as follows:

[0063] K(p,ε time )=Norm(concat(K j )), j∈C

[0064] Among them, Norm() means normalization;

[0065] The similarity distribution K(p,ε time ) and the center vector m(ε time ) are spliced ​​to generate the macro time vector of the target information, and the formula is as follows:

[0066] e p,time =MLP(concat(p,m(ε time ),K(p,ε time )))

[0067] Among them, MLP represents nonlinear transformation;

[0068] Through novelty analysis, the target information is calculated in the spatial window ε space The similarity distribution in the space window is combined with the center vector m(ε space ), analyze the difference between the similarity distribution of the target information and the similarity distribution of the center vector of the spatial window, where the Hadamard product is used as the comparison function to measure the difference between the target information and the spatial window, and the formula is as follows:

[0069] g(K(p,ε space ),K(m(ε space ),ε space )))

[0070] The learned difference g(K(p,ε space ),K(m(ε space ),ε space)) and the characteristic representation of the target information and the center vector m(ε space ) are spliced ​​to generate the micro-space vector of the target information, and the formula is as follows:

[0071] e p,space =MLP(concat(p,m(ε space ),g(K(p,ε space ),K(m(ε space ),ε space ))))

[0072] The macro time vector e p,time With the micro space vector e p,space Combined, a binary classification operation is performed through a classifier to output the authenticity detection result of the target information.

[0073] Optionally, also include:

[0074] For the modality alignment task, the loss in the modality alignment process is captured by minimizing the difference between the feature vectors of the text modality and the image modality. The mean square error is used to define the modality alignment loss, and the formula is as follows:

[0075]

[0076] Where n is the number of samples, and Respectively represent the image feature vector and text feature vector of the jth sample;

[0077] For the heat information authenticity detection task, classification is performed through a binary classification model, and the cross entropy loss function is used to capture the loss in the classification process. The formula is as follows:

[0078]

[0079] Among them, y i is the true label, is the predicted value;

[0080] The final total loss function is weighted according to the modality alignment loss and classification loss. The formula is as follows:

[0081] Loss = λ c L class +λ a L align

[0082] Among them, λ c and λ a They are the weights of classification loss and modality alignment loss, respectively, which are used to balance the impact of the two parts of loss on model optimization.

[0083] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a microblog-oriented information heat evaluation and multimodal false information detection device, including:

[0084] A data collection and preprocessing module is used to collect news data published by a target official news account on the Weibo platform, preprocess the news data to form a news corpus, and filter out news samples containing image and text modalities from the news corpus;

[0085] A time window construction module is used to select a news collection and a hot news time window within a preset time before the target information is released, and the news samples in the time window are obtained by screening the news corpus;

[0086] A modal feature learning and fusion module is used to perform feature learning on the image and text modalities of the news sample in the time window based on the pre-trained ResNet50 and BERT models, and to achieve dual-modal feature fusion using a collaborative attention mechanism to obtain a multi-modal feature representation of the target information;

[0087] A heat evaluation module, used to calculate the similarity between the target information and the news sample in the time window based on the multimodal feature representation, and determine whether the target information has heat by setting a similarity threshold;

[0088] The authenticity detection module is used to select the first several news samples related to the target information to form a high-similarity spatial window based on popularity and novelty analysis, and analyze the correlation and difference between the target information and the time window and the spatial window through the kernel function to realize the authenticity detection of the target information.

[0089] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0090] This application dynamically constructs a spatiotemporal window with target information as a reference point, which can accurately capture the potential connection between information and other news in the environment, and provide comprehensive contextual support for information popularity assessment and authenticity detection; combining the bimodal features of images and texts, and achieving efficient fusion through a collaborative attention mechanism, significantly improves the accuracy of detection results and makes up for the limitations of unimodal information analysis; through popularity distribution and novelty distribution analysis, the authenticity detection of hot information is realized in a dynamic spatiotemporal window, which effectively addresses the problem of lagging detection of existing models in the early stage of false information propagation.

[0091] Compared with existing models, the present application has higher accuracy in predicting the truth or falsity of hot information, especially in the early stage of information dissemination. The technical solution of the present application not only fully explores the potential connection between the target information and real news, but also through the construction of dynamic spatiotemporal windows and multimodal feature learning, it achieves higher detection robustness and wide applicability, and improves the overall performance of false information detection.

[0092] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0094] Figure 1 A text flow diagram of a method for evaluating information popularity and detecting multimodal false information for microblogs provided in an embodiment of the present application;

[0095] Figure 2 A model architecture diagram of a method for evaluating information popularity and detecting multimodal false information for microblogs provided in an embodiment of the present application;

[0096] Figure 3 A schematic diagram of the image-text dual-modal fusion process provided in an embodiment of the present application;

[0097] Figure 4 A schematic diagram of the structure of a device for information heat evaluation and multimodal false information detection for Weibo provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0099] In response to the current research status and existing problems and challenges, the embodiments of the present application provide an information heat evaluation and multimodal false information detection method for Weibo, which not only considers the potential connection between real news and target information, but also utilizes multimodal feature learning to improve the accuracy of detection results. It judges heat information by dynamically constructing spatiotemporal windows, analyzes popularity and novelty, and integrates image and text dual modalities to ultimately achieve heat information authenticity detection.

[0100] Figure 1A text flow diagram of a method for information popularity assessment and multimodal false information detection for Weibo provided in an embodiment of the present application. Figure 2 A model architecture diagram of a method for information popularity assessment and multimodal false information detection for Weibo provided in an embodiment of the present application.

[0101] Reference Figure 1 and Figure 2 , the method comprises the following steps:

[0102] Step 101, collect news data published by a target official news account on the Weibo platform, pre-process the news data to form a news corpus, and filter out news samples containing image and text modalities from the news corpus.

[0103] The implementation example of the present application constructs a dynamic spatiotemporal window based on recent mainstream hot news to ensure accurate detection of the popularity of the target information.

[0104] Specifically, first, we collect news data published by the target official news account on the Weibo platform, and regularly capture these news data through automated collection methods. The target official news account is a Weibo account with high influence, and its news content contains both image and text modalities. Through this data source, we can ensure that the collected news content has high authority and authenticity.

[0105] In a possible embodiment, an official news account with more than 30 million followers on the Weibo platform is used as a target official news account for publication.

[0106] Next, the collected news data is preprocessed to form a news corpus. During the preprocessing process, each news sample is defined as consisting of a long text t and an image i. If a news sample contains more than one image, the images in the sample are spliced. Specifically, the splicing method is calculated according to the number of images, multiple images are arranged into square or rectangular combinations, and the unfilled areas are filled with zeros, and finally a spliced ​​image of a fixed size is output. In addition, the news text content is cleaned to remove irrelevant content and data missing samples to ensure the data integrity and quality in the news corpus. Calculate the number of images n in the news sample.

[0107] If explained with a formula, the splicing logic is as follows:

[0108] (1) Calculate the square root of n and round down to get And define the number of rows and columns as:

[0109]

[0110] (2) Stitch the pictures together and stitch the first s*s pictures into a complete square. If n>s 2 , arrange the remaining images in the order of row and col. If the last row is not completely filled, the remaining parts are set to all 0s;

[0111] (3) Compress the stitched image and output a compressed image with a size of 900×900.

[0112] Through the above steps, a news corpus containing news samples in both image and text modalities was finally constructed, providing basic support for the subsequent popularity evaluation and authenticity detection of target information.

[0113] Step 102, a news collection within a preset time before the target information is released is selected as a hot news time window, and news samples in the time window are obtained by screening a news corpus.

[0114] In the embodiment of the present application, based on the construction of the news corpus, in order to further provide contextual support for the heat evaluation and authenticity detection of the target information, a time window ε is defined. time The time window is constructed by screening a set of news samples that are temporally relevant to the target information from the news corpus. The specific implementation process is as follows:

[0115] First, obtain the release time t of the target information p . Target information refers to the information to be analyzed that requires heat assessment and authenticity detection, and its release time is the key reference point for constructing the time window.

[0116] Next, according to the release time t of the target information p , filter the release time of all news samples in the news corpus. By comparing t p and the release time t of each news sample in the news corpus e The time difference is 0. <t p -t e ≤T news samples. T is the preset time. This time range is used to ensure that the news samples in the time window have strong timeliness, thereby improving the accuracy of target information analysis.

[0117] Finally, all the qualified news samples screened out constitute the time window ε time The definition formula of the time window is as follows:

[0118] ε tine ={e:e∈ε,0 <t p -t e ≤T}

[0119] Among them, ε represents all news samples in the news corpus, ε represents the release time of the target information p, and t e Indicates the release time of news sample e.

[0120] It can be understood that by constructing the time window ε time , which can provide dynamic news context support for the popularity evaluation of target information.

[0121] Step 103, based on the pre-trained ResNet50 and BERT models, feature learning is performed on the image and text modalities of the news samples in the time window respectively, and a collaborative attention mechanism is used to realize bimodal feature fusion to obtain a multimodal feature representation of the target information.

[0122] In an embodiment of the present application, modal feature learning is first performed on news samples in a time window based on a pre-trained deep learning model, and features of the image modality and text modality of the news samples are extracted respectively, laying the foundation for subsequent multimodal fusion and target information analysis.

[0123] Specifically, for the image modality, the pre-trained ResNet50 model is used as the embedding model, and the image data of the news samples in the time window are input into the ResNet50 model to obtain the embedding representation R of the image modality. i To unify the feature dimensions of text modality and image modality, we further pass the fully connected layer W i Embedding the image R i Map it to the same preset feature dimension d as the text modality to obtain the image feature matrix. The feature representation formula of the image modality is as follows:

[0124] E i =σ(W i *R i +b)

[0125] Among them, b is the bias vector, σ is the activation function, E t is the image feature matrix.

[0126] For the text modality, the pre-trained BERT model is used as the embedding model. The text data of the news sample in the time window is input into the BERT model to extract the embedding representation R of the text modality. t Similarly, through the fully connected layer W t For text embedding representation R t Map it to the preset feature dimension d to obtain the text feature matrix. The feature representation formula of the text modality is as follows:

[0127] E t =σ(W t *R t +b)

[0128] Among them, b is the bias vector, σ is the activation function, E t is the text feature matrix.

[0129] Through the above process, the image and text modalities of the news samples in the time window are learned separately to obtain the image feature matrix and text feature matrix of unified feature dimension. These features provide the basis for the subsequent multimodal feature fusion.

[0130] After obtaining the image feature matrix and text feature matrix of uniform feature dimension, refer to Figure 3 In the embodiment of the present application, the feature fusion of text modality and image modality is realized through the co-attention mechanism, which generates a multimodal feature representation of the target information by co-aligning the features of the two modalities. The mechanism includes two parallel co-attention modules Co-Att1 and Co-Att2, and the functions of each module are as follows:

[0131] (1) In Co-Att1, text features are used as queries and image features are used as keys and values. The attention score is calculated to realize the attention of the text modality to the image modality, and the weighted representation of the image modality is obtained by weighted summation.

[0132] (2) In Co-Att2, the image features are used as queries and the text features are used as keys and values. The attention score is calculated to achieve the attention of the image modality to the text modality, and the weighted representation of the text modality is obtained by weighted summation.

[0133] Finally, the weighted representation of the text modality to the image modality in Co-Att1 and the weighted representation of the image modality to the text modality in Co-Att2 are concatenated to obtain the multimodal feature representation of the target information. The formula is as follows:

[0134]

[0135] Among them, E ti is the multimodal feature representation of the target information, E t is the text modality feature matrix, E i is the image modality feature matrix, d t and d i are the feature embedding dimensions for text and image modalities, respectively.

[0136] Through the above-mentioned collaborative attention mechanism, the features of the text modality and the image modality are effectively integrated, and the generated multimodal feature representation provides high-quality input for subsequent information heat evaluation and authenticity detection.

[0137] Step 104, based on the multimodal feature representation, calculate the similarity between the target information and the news sample in the time window, and determine whether the target information is hot by setting a similarity threshold.

[0138] In an embodiment of the present application, based on the multimodal feature representation of the target information and in combination with news samples in the time window, it is determined whether the target information is hot by calculating the similarity and setting a threshold.

[0139] The specific implementation process is as follows:

[0140] First, for the time window to which the target information constructed in step 103 belongs, the cosine similarity is calculated using the target information feature representation after multimodal feature fusion and the multimodal feature representation of each news sample in the time window. The cosine similarity calculation formula between the target information and the news sample in the time window is as follows:

[0141]

[0142] Among them, p is the multimodal feature representation of the target information, e i It is the multimodal feature representation of the i-th news sample in the time window.

[0143] Then, considering that the target information may have evolved from a hot news, if the similarity between all news samples and the target information is directly analyzed uniformly, a lot of noise may be introduced due to the uneven distribution of news. Therefore, the calculated similarity needs to be screened and counted. The specific steps are as follows:

[0144] (1) Arrange the calculated similarity values ​​in descending order and select the similarity values ​​with the first α ratio to form a set S top , the formula is as follows:

[0145] S top =Top(α)(S)

[0146] Among them, S represents the set of similarity values ​​between the target information and all news samples in the time window, and Top(α)(S) represents the subset with the top α proportions after the similarity values ​​are arranged in descending order.

[0147] In a possible embodiment, α takes a value of 0.3, which means that the similarities of the top 30% of the most relevant news are taken for statistical analysis.

[0148] (2) For the set S top The similarity values ​​in are counted to determine whether the proportion greater than the preset threshold β exceeds 50%. If the condition is met, the target information is determined to be hot information, otherwise it is non-hot information. The formula is as follows:

[0149]

[0150] Among them, ∏(cos(p,e i )>β) is the judgment function. When the similarity value is greater than the threshold β, the result is 1, otherwise it is 0.

[0151] Through the above method, it is possible to effectively evaluate whether the target information is popular, laying the foundation for subsequent authenticity detection of false information.

[0152] Step 105, based on popularity and novelty analysis, select the first several news samples related to the target information to form a high similarity space window, and use the kernel function to analyze the correlation and difference between the target information and the time window and the space window to achieve authenticity detection of the target information.

[0153] It should be noted that popular false information usually has two significant characteristics: one is that it is highly relevant to recent hot news and attracts traffic by "riding on the popularity"; the other is that it exaggerates or fabricates facts in the details of the content to attract user attention by "catching the eye". This leads to the spread speed and spread opportunities of false information being significantly higher than true information. Therefore, it is not enough to only analyze the similarity distribution of the target information in the time window. It is also necessary to introduce a spatial window to analyze the difference distribution between the target information and related news. The greater the difference, the more likely the target information is to be far from the truth.

[0154] Based on this feature, the embodiment of the present application further considers the difference distribution between the target information and related information on the basis of analyzing the time window similarity distribution of the target information. The greater the difference, the more likely the target information is to deviate from the truth, thereby assisting in true and false detection.

[0155] First, in the time window ε time Based on this, we define the spatial window ε space The spatial window represents a set of news that is most relevant to the target information, and the formula is as follows:

[0156] ε space =Top(k)(ε time )

[0157] k=max(k,len(ε time ))

[0158] Among them, k is a hyperparameter, which means selecting the first k most relevant news samples in the time window, and its value is the given k and the time window ε time The maximum number of vectors. For example, if the time window contains 500 news samples, k can be set to 50 to select the 50 news most relevant to the target information as the spatial window.

[0159] (1) Popularity analysis: Through popularity analysis, the similarity distribution between the target information and all news samples in the time window is calculated.

[0160] In the embodiment of the present application, a Gaussian kernel function is used to pool the cosine similarity between the target information and all news samples in the time window, and the formula is as follows:

[0161]

[0162] Among them, K j is the output of the jth kernel, μ j is the center value of the kernel, scattered between [-1,1], σ j is the width of the kernel, n is the time window ε time The number of news samples in the

[0163] It should be noted that the use of Gaussian kernel to pool cosine similarity is intended to focus not only on the most similar news, but also to consider the similarity of all news items in the entire time window; in addition, the Gaussian function has a smoothing effect, which can reduce the impact of noise and outliers.

[0164] In one possible embodiment, μ j The value is [-1,-0.8,-0.6,-0.4,-0.2,0,0.2,0.4,0.6,0.8,1], the interval is 0.2, σ j Both are set to 0.05.

[0165] After that, the outputs of all cores are vectorized and normalized to obtain the similarity distribution of the target information in the time window. The formula is as follows:

[0166] K(p,ε time )=Norm(concat(K j )), j∈C

[0167] Among them, Norm() means normalization.

[0168] Based on the above operations, we can fully learn the similarity distribution K(p,ε time ), and finally, the similarity distribution K(p,ε time ) and the center vector m(ε time ) to generate the macro time vector of the target information. The formula is as follows:

[0169] e p,time =MLP(concat(p,m(ε time ),K(p,ε time )))

[0170] Here, MLP represents nonlinear transformation.

[0171] (2) Novelty analysis: Different from popularity analysis, novelty analysis aims to capture the relationship between target information p and spatial window ε. space The Gaussian kernel pooling is still used to learn the similarity distribution between vectors, but the difference is that the center vector m(ε space ) as a reference, analyze the similarity distribution of p in the spatial window and m(ε space ) is used to measure the difference between the similarity distributions of the two spatial windows. Here, the Hadamard product is used as the comparison function. The formula of the Hadamard product is as follows:

[0172] g(K(p,ε space ),K(m(ε space ),ε space )))

[0173] Next, the learned difference g(K(p,ε space ),K(m(ε space ),ε space )) and the feature representation of the target information and the center vector m(ε space ) are spliced ​​to generate the micro-space vector of the target information. The formula is as follows:

[0174] e p,space =MLP(concat(p,m(ε space ),g(K(p,ε space ),K(m(ε space ),ε space ))))

[0175] Finally, the macro time vector e p,time With the micro space vector e p,space Combined, a binary classification operation is performed through the classifier to output the authenticity detection result of the target information.

[0176] Through the above steps, the embodiment of the present application can combine the popularity distribution and novelty distribution of the target information, analyze from two dimensions of time and space, and comprehensively evaluate the authenticity of the target information.

[0177] It is understandable that in the process of model training, a loss function needs to be introduced. The loss function is used to quantify the accuracy of the model's prediction and guide the model to continuously reduce the prediction error through optimization iteration during the training process.

[0178] In an embodiment of the present application, the loss function of the model training consists of two parts: modal alignment loss and classification loss, which correspond to the modal feature alignment task and the heat information authenticity classification task respectively.

[0179] The goal of modality alignment loss is to capture the potential correlation between text modality and image modality in high-dimensional space. Although multimodal feature fusion is achieved through the collaborative attention mechanism in the above steps, this mechanism does not directly capture the implicit similarity between text modality and image modality features. Therefore, the embodiment of the present application regards the modality alignment task as a regression problem, captures the loss in the modality alignment process by minimizing the difference between the feature vectors of text modality and image modality, and defines the modality alignment loss using mean square error, and the formula is as follows:

[0180]

[0181] Where n is the number of samples, and Represent the image feature vector and text feature vector of the jth sample respectively.

[0182] By minimizing L align , the model can further strengthen the alignment relationship between image and text modalities in high-dimensional feature space.

[0183] For the task of detecting the authenticity of hot information, the task of detecting the authenticity of hot information is a binary classification task. The goal is to predict the probability of the target information being true or false based on the multimodal feature representation. The embodiment of the present application uses the cross entropy loss function to quantify the error in the classification process. The formula is as follows

[0184]

[0185] Among them, y i is the true label, is the predicted value.

[0186] By minimizing L class ,The model can improve the accuracy of the classification task, and thus more effectively judge the truth or falsity of the target information.

[0187] The final loss function Loss combines the modality alignment loss L align and classification loss L class , weighing the impact of the two parts. By introducing the weight parameter λ c and λ a , controls the contribution of the alignment task and the classification task in the total loss, and the formula is as follows:

[0188] Loss = λ c L class +λ a L align

[0189] Among them, λ c and λ aThey are the weights of classification loss and modality alignment loss, respectively, which are used to balance the impact of the two parts of loss on model optimization.

[0190] It should be noted that the modality alignment weight λ c and classification weight λ a The value of is determined according to the experimental optimization results, and this application does not make any specific limitation on this.

[0191] Through the above-mentioned loss function design, this application can efficiently guide model training, which not only ensures the alignment of multimodal features, but also effectively improves the classification accuracy of heat information authenticity detection.

[0192] In order to implement the above embodiments, the present application also proposes an information heat evaluation and multimodal false information detection device for Weibo. Figure 4 A schematic diagram of the structure of a microblog-oriented information heat evaluation and multimodal false information detection device provided in an embodiment of the present application. Figure 4 As shown, the device comprises:

[0193] The data collection and preprocessing module 100 is used to collect news data published by the target official news account on the Weibo platform, preprocess the news data to form a news corpus, and filter out news samples containing image and text modalities from the news corpus;

[0194] The time window construction module 200 is used to select a news collection within a preset time before the target information is released as a hot information time window, and the news samples in the time window are obtained by screening the news corpus;

[0195] The modal feature learning and fusion module 300 is used to perform feature learning on the image and text modalities of the news sample in the time window based on the pre-trained ResNet50 and BERT models, and to achieve dual-modal feature fusion using a collaborative attention mechanism to obtain a multi-modal feature representation of the target information;

[0196] The popularity evaluation module 400 is used to calculate the similarity between the target information and the news sample in the time window based on the multimodal feature representation, and determine whether the target information has popularity by setting a similarity threshold;

[0197] The authenticity detection module 500 is used to select the first several news samples related to the target information to form a high-similarity spatial window based on popularity and novelty analysis, and analyze the correlation and difference between the target information and the time window and the spatial window through the kernel function to realize the authenticity detection of the target information.

[0198] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0199] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0200] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0201] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for evaluating information heat and detecting multimodal false information on Weibo, characterized in that: The following steps are involved: Collecting news data published by a target official news account on the Weibo platform, preprocessing the news data to form a news corpus, and screening news samples containing image and text modalities from the news corpus; Selecting a news collection within a preset time before the target information is released as a hot news time window, wherein the news samples in the time window are obtained by screening the news corpus; Based on the pre-trained ResNet50 and BERT models, feature learning is performed on the image and text modalities of the news samples in the time window, and a collaborative attention mechanism is used to achieve dual-modal feature fusion to obtain a multi-modal feature representation of the target information; Based on the multimodal feature representation, calculating the similarity between the target information and the news sample in the time window, and determining whether the target information has popularity by setting a similarity threshold; Based on popularity and novelty analysis, the first several news samples related to the target information are selected to form a high-similarity spatial window, and the correlation and difference between the target information and the time window and spatial window are analyzed through the kernel function to realize the authenticity detection of the target information.

2. The method according to claim 1, characterized in that The collecting of news data released by the target official news account on the Weibo platform and preprocessing of the news data to form a news corpus includes: Regularly capture all news data published by the target official news account on the Weibo platform through automated collection methods. The news content includes image and text modalities. It is defined that each news sample consists of a long text t and an image i. For news samples with more than one image, the images they contain are spliced, and the text content is cleaned to remove irrelevant content and data missing samples to form a news corpus.

3. The method according to claim 2, characterized in that For the news sample with more than one image, the images contained therein are spliced, including: Calculate the number of images n in the news sample; Calculate the square root of n and round down to get And define the number of rows and columns as: Splice the pictures and stitch the first s*s pictures into a complete square. If n>s 2 , arrange the remaining images in the order of row and col. If the last row is not completely filled, the remaining parts are set to all 0s; The stitched images are compressed and the output size is 900×900.

4. The method according to claim 2, characterized in that: The step of selecting a news collection within a preset time before the target information is released as a hot news time window, wherein the news samples in the time window are obtained by screening the news corpus, comprises: Get the release time t of the target information p ; According to t p and the release time t of each news sample in the news corpus e The time difference is 0. <t p -t e ≤T news samples, define the time window ε time For all news sample sets that meet the conditions, the formula is as follows: ε time ={e:e∈ε,0<t p -t e ≤T} Where ε represents all news samples in the news corpus, ε represents the release time of the target information p, and t e represents the release time of news sample e, and T is the preset time.

5. The method according to claim 4, characterized in that The pre-trained ResNet50 and BERT models respectively perform feature learning on the image and text modalities of the news samples in the time window, including: For the image modality, the pre-trained ResNet50 model is used as the embedding model to obtain the embedding representation R of the image modality. i ; Through the fully connected layer W i Embedding the image R i Map it to the same preset feature dimension d as the text modality to obtain the image feature matrix. The formula is as follows: E i σ(W i *R i +b) Among them, b is the bias vector, σ is the activation function, E t is the image feature matrix; For the text modality, the pre-trained BERT model is used as the embedding model to obtain the embedding representation R of the text modality. t ; Through the fully connected layer W t For text embedding representation R t Map it to the preset feature dimension d to obtain the text feature matrix. The formula is as follows: E t σ(W t *R t +b) Among them, b is the bias vector, σ is the activation function, E t is the text feature matrix.

6. The method according to claim 5, characterized in that The collaborative attention mechanism is used to fuse the text features and the image features to obtain a multimodal feature representation of the target information, including: In the collaborative attention mechanism, there are two parallel collaborative attention modules Co-Att1 and Co-Att2, where: In Co-Att1, text features are used as queries and image features are used as keys and values. The attention score is calculated to achieve the attention of the text modality to the image modality, and the weighted representation of the image modality is obtained by weighted summation. In Co-Att2, the image features are used as queries and the text features are used as keys and values. The attention score is calculated to realize the attention of the image modality to the text modality, and the weighted representation of the text modality is obtained by weighted summation. The weighted representation of the text modality to the image modality in Co-Att1 is concatenated with the weighted representation of the image modality to the text modality in Co-Att2 to obtain the multimodal feature representation of the target information. The formula is as follows: Among them, E ti is the multimodal feature representation of the target information, E t is the text modality feature matrix, E i is the image modality feature matrix, d t and d i are the feature embedding dimensions for text and image modalities, respectively.

7. The method according to claim 6, characterized in that The calculating the similarity between the target information and the news sample in the time window based on the multimodal feature representation, and judging whether the target information has popularity by setting a similarity threshold, includes: The cosine similarity between the target information and each news sample in the time window is calculated respectively, and the formula is as follows: Where p is the multimodal feature representation of the target information, e i is the multimodal feature representation of the i-th news sample in the time window; Arrange the calculated similarity values ​​in descending order, and select the similarity values ​​of the first α proportions to form a set S top , the formula is as follows: S top =Top(α)(S) Wherein, S represents the set of similarity values ​​between the target information and all news samples in the time window, and Top(α)(S) represents the subset with the top α proportions after the similarity values ​​are arranged in descending order; For the set S top The similarity values ​​in are counted to determine whether the proportion of the similarity values ​​greater than the preset threshold β exceeds 50%. If the condition is met, the target information is determined to be hot information, otherwise it is non-hot information. The formula is as follows: Among them, Π(cos(p, e i )>β) is a judgment function. When the similarity value is greater than the threshold β, the result is 1, otherwise it is 0.

8. The method according to claim 7, characterized in that The method of selecting the first several news samples most relevant to the target information to form a high-similarity space window based on popularity and novelty analysis, and analyzing the correlation and difference between the target information and the time window and the space window through a kernel function to detect the authenticity of the target information includes: In the time window ε time Based on this, we define the spatial window ε space , the spatial window represents a set of news most relevant to the target information, and the formula is as follows: e space =Top(k)(e) time ) k=max(k,len(ε time )) Among them, k is a hyperparameter, which means selecting the first k most relevant news samples in the time window, and its value is the given k and the time window ε time The maximum number of vectors; Through popularity analysis, the similarity distribution between the target information and all news samples in the time window is calculated, and the cosine similarity is pooled using the Gaussian kernel function. The formula is as follows: Among them, K j is the output of the jth kernel, μ j is the center value of the kernel, σ j is the width of the kernel, n is the time window ε time The number of news samples in the The outputs of all cores are vectorized and normalized to obtain the similarity distribution of the target information in the time window. The formula is as follows: K(p,ε time )=Norm(concat(K j )),j∈C Among them, Norm() means normalization; The similarity distribution K(p, ε macro ) and the center vector m(ε macro ) are spliced ​​to generate the macro time vector of the target information, and the formula is as follows: e p，time NMLP(concat(p),m(ε time ),K(p,ε time ))) Among them, MLP represents nonlinear transformation; Through novelty analysis, the target information is calculated in the spatial window ε space The similarity distribution in the space window is combined with the center vector m(ε space , analyze the difference between the similarity distribution of the target information and the similarity distribution of the center vector of the spatial window, where the Hadamard product is used as a comparison function to measure the difference between the target information and the spatial window. The formula is as follows: g(K(p,e space ),K(m(e space ),e space ))) The learned difference g(K(p, ε space ), K(m(ε space ), ε space )) and the characteristic representation of the target information and the center vector m(ε space ) are spliced ​​to generate the micro-space vector of the target information, and the formula is as follows: e p，space NMLP(concat(p),m(ε space ),g(K(p,ε space ),K(m(ε space ),ε space )))) The macro time vector e p，time With the micro space vector e p,space Combined, a binary classification operation is performed through a classifier to output the authenticity detection result of the target information.

9. The method according to claim 8, characterized in that Also includes: For the modality alignment task, the loss in the modality alignment process is captured by minimizing the difference between the feature vectors of the text modality and the image modality. The modality alignment loss is defined using the mean square error, as follows: Where n is the number of samples, and Respectively represent the image feature vector and text feature vector of the jth sample; For the task of detecting the authenticity of heat information, classification is performed through a binary classification model, and the cross entropy loss function is used to capture the loss in the classification process. The formula is as follows: Among them, yi is the true label, is the predicted value; The final total loss function is weighted according to the modality alignment loss and classification loss. The formula is as follows: Loss=λ c L class +λ a L align Among them, λ c and λ a They are the weights of classification loss and modality alignment loss, respectively, which are used to balance the impact of the two parts of loss on model optimization.

10. A device for evaluating information heat and detecting multimodal false information for microblogs, characterized in that: include: A data collection and preprocessing module is used to collect news data published by a target official news account on the Weibo platform, preprocess the news data to form a news corpus, and filter out news samples containing image and text modalities from the news corpus; A time window construction module is used to select a news collection within a preset time before the target information is released as a hot news time window, and the news samples in the time window are obtained by screening the news corpus; A modal feature learning and fusion module is used to perform feature learning on the image and text modalities of the news sample in the time window based on the pre-trained ResNet50 and BERT models, and to achieve dual-modal feature fusion using a collaborative attention mechanism to obtain a multi-modal feature representation of the target information; A heat evaluation module, used to calculate the similarity between the target information and the news sample in the time window based on the multimodal feature representation, and determine whether the target information has heat by setting a similarity threshold; The authenticity detection module is used to select the first several news samples related to the target information to form a high-similarity spatial window based on popularity and novelty analysis, and analyze the correlation and difference between the target information and the time window and the spatial window through the kernel function to realize the authenticity detection of the target information.

Images