Social big data cross-modal meta-learning early rumor detection method based on small samples

Abstract

The invention relates to the field of social media content detection, in particular to a social big data cross-modal meta-learning early rumor detection method based on small samples, and the method comprises the steps: obtaining to-be-detected multi-modal information published on social media, carrying out the feature extraction through employing a multi-modal feature extraction network, and obtaining to-be-detected multi-modal information; constructing a multi-modal hidden information extraction network to extract hidden information to obtain multi-modal hidden information; constructing a multi-modal fusion processing network, and performing rumor detection by using the multi-modal fusion processing network; and optimizing the multi-modal hidden information extraction network and the multi-modal fusion processing network through a meta-learning algorithm so as to obtain a final detection result. According to the method, the rumor detection is realized by extracting the multi-modal hidden information and carrying out deep fusion processing. In addition, a meta-learning algorithm is applied to a multi-modal hidden information extraction network and a multi-modal fusion processing network, so that early rumor detection with a small number of samples is realized.

Description

Technical Field

[0001] The present invention relates to the field of social media content detection, and more specifically, to an early rumor detection method based on small sample-based social big data cross-modal meta-learning. Background Art

[0002] With the rapid development of personal computers and mobile phones, it has become increasingly convenient to share news and obtain information through online media platforms, and news and information with subjective opinions are often shared on these platforms. However, they rarely verify the accuracy of news or information before sharing it, which leads to the widespread dissemination of unverified content, including rumors. Compared with real news, rumors are usually novel, making them more likely to attract attention and cause emotional reactions, resulting in faster and wider spread. Therefore, accurately identifying rumors in a large amount of shared information is a crucial research. The mainstream social media platforms in society include Weibo, Xiaohongshu, and Douyin. These social media network platforms can not only send the text content that people want to express, but also include multimodal information such as pictures. These multimodal information not only enriches the diversity of people's expression of information, but also increases the difficulty of rumor detection. In addition, the number of samples of rumors is often small in the early stage, and the structure of their dissemination is relatively simple, which leads to the problem of low accuracy in the early detection of rumors using general rumor detection methods.

[0003] The prior art discloses a microblog rumor detection algorithm based on a gradient boosting tree and a rumor detection feature set. A feature set for rumor detection is provided, which includes 23 features. A rumor detection algorithm based on a gradient boosting tree is provided. The algorithm first constructs training samples according to the features in the feature set, and the training samples are used for training the microblog rumor detection model; then multiple trainings are performed on the training sample set to obtain multiple regression tree models, each regression tree gives a prediction value, and the prediction values ​​of multiple regression trees are combined to obtain the final microblog rumor detection model; when performing rumor detection, the features of the microblog to be predicted are extracted according to the feature set, and the prediction value of the microblog to be predicted is calculated using the detection model, and the microblog to be predicted is judged as a rumor microblog or a non-rumor microblog according to the prediction value. However, this method still has poor detection effect in the early stage of rumor propagation when the number of samples is small. Summary of the invention

[0004] The purpose of the present invention is to disclose an early rumor detection method based on cross-modal meta-learning of social big data with small samples, which is suitable for small samples and has better detection effect.

[0005] In order to achieve the above object, the present invention provides an early rumor detection method based on small sample social big data cross-modal meta-learning, comprising:

[0006] S1: Obtain the multimodal information to be detected published on social media, use the multimodal feature extraction network to extract features from the multimodal information to obtain multimodal feature information;

[0007] S2: construct a multimodal hidden information extraction network; use the multimodal hidden information extraction network to extract hidden information from multimodal feature information to obtain multimodal hidden information;

[0008] S3: Build a multimodal fusion processing network, use the multimodal fusion processing network to detect rumors on multimodal hidden information, and obtain preliminary detection results;

[0009] S4: The multimodal hidden information extraction network and the multimodal fusion processing network are optimized through a meta-learning algorithm to obtain an optimized multimodal hidden information extraction network and an optimized multimodal fusion processing network. Rumor detection is performed based on the optimized multimodal hidden information extraction network and the optimized multimodal fusion processing network to obtain the final detection result.

[0010] Furthermore, in step S1, it includes: preprocessing the data for the multimodal information to be detected, deleting redundant content, including at least one of the following: special characters, URL hyperlinks, HTML characters, and emoticons.

[0011] In step S1, the multimodal information to be detected includes text modality and image modality; wherein the text modality adopts different types of languages;

[0012] Let the set P = {T, V1, V2, ..., V N ,} represents the multimodal information to be detected, {T} represents the text modality, {V1,V2,...,V N} represents the image modality; the text modality consists of a sequence of words, represented by T = {W1, W2, ..., W N ,};

[0013] The multimodal feature information includes text feature information and image feature information;

[0014] The multimodal hidden information extraction network includes a text hidden information extraction network and an image hidden information extraction network.

[0015] The multimodal feature extraction network includes a text feature extraction network and an image feature extraction network;

[0016] Furthermore, a multimodal feature extraction network is used to extract features from the multimodal information to be detected, and multimodal feature information is obtained, including:

[0017] The multimodal feature extraction network includes a text feature extraction network and an image feature extraction network;

[0018] Use VGG16 as the image feature extraction network to extract features from image information and obtain image feature information

[0019] Use BERT as the text feature extraction network to extract features from text information and obtain text feature information:

[0020] P T =BERT{W1,W2,...,W N}

[0021] P V =VGG{V1,V2,...,V N}

[0022] Among them, P T is the output feature information after BERT encoding, with a dimension of 393216, which is text feature information; P V It is the final layer output of the VGG model, with a dimension of 4096, which is image feature information. It takes text feature information and image feature information as multimodal feature information.

[0023] Furthermore, in step S2, constructing a multimodal hidden information extraction network includes:

[0024] Two Bi-LSTM bidirectional long short-term memory networks and a series of hidden layer structures are constructed as a text hidden information extraction network. The hidden information extraction of text feature information includes:

[0025] Reduce the dimensionality of the input text:

[0026] x=ReLU{W1P T +b1}

[0027] Among them, x is the output result after dimension conversion, W1 is the weight matrix of the activation function network, and b1 is the bias vector;

[0028] The reduced text dimension features are input into two Bi-LSTM bidirectional long short-term memory networks:

[0029]

[0030] output=cat(output1,output2)

[0031] Among them, output, output, and output2 are the output of the double-layer LSTM and the cascade output in the feature dimension, respectively. is the initial hidden state and cell state of LSTM, is the final hidden state and cell state of LSTM;

[0032] Through two hidden layers with the same structure, take the first layer as an example:

[0033] output layer1 1 =LayerNorm1(output)

[0034] output layer1 2 =ReLU(fc1(output layer1 1 ))

[0035] output layer1 3 =Dropout(output layer1 2 )

[0036] The final output of the second hidden layer is output layer2 3 ;

[0037] Through a fully connected layer, output text hidden information:

[0038] output text =fc3(output layer2 3 )

[0039] After processing the last fully connected layer, the final output dimension is 128.

[0040] Furthermore, constructing a multimodal hidden information extraction network includes:

[0041] Use the fully connected layer to build an image hidden information extraction network to extract hidden information from image feature information, including:

[0042] output3=Linear(W2P V +b2)

[0043] output image =Linear(W3ouput3+b3)

[0044] Among them, W2 is the weight matrix of the first fully connected layer, b2 is the bias vector, and P V represents the input image features, with a dimension of 4096; W3 is the weight matrix of the second fully connected layer, b3 is the bias vector, and the final output size is 128 dimensions.

[0045] Furthermore, constructing a multimodal fusion processing network includes: constructing a multimodal fusion processing network using a multi-layer perception network.

[0046] Furthermore, a multimodal fusion processing network is used to detect rumors on multimodal hidden information, and the detection results include:

[0047] Concatenating multimodal hidden information:

[0048] output=cat(output text ,output image )

[0049] Among them, output text Hide information for text, output image Hide information for the image, output is output text And output image The cascade output of represents multimodal hidden information;

[0050] Through two hidden layers with the same structure, take the first layer as an example:

[0051] output layer1 1 =Linear(output)

[0052] output layer1 2 =ReLU(W4output layer1 1 +b4)

[0053] output layer1 3 =Dropout(output layer1 2 ,p1=0.5)

[0054] Among them, W4 and b4 are the weight matrix and bias vector respectively, and p1 is the pooling probability, indicating that 50% of neurons are randomly separated;

[0055] Get the rumor probability through full connection:

[0056] output rumor =W6output layer2 2 +b6

[0057] Among them, W6 and b6 are the weight matrix and bias vector of the output layer, respectively. layer2 2 is the final output of the second hidden layer, output rumorIt is a two-dimensional array that stores the probabilities of rumors and non-rumors, and obtains the detection results based on the rumor probabilities.

[0058] Furthermore, in step S4, it includes:

[0059] Initialize the parameters of the multimodal hidden information extraction network and the multimodal fusion processing network:

[0060] θ=(θ text ,θ image ,θ classifier )

[0061] Among them, θ text ,θ image ,θ classifier They represent the initial parameters of the text hidden information extraction network, image hidden information extraction network and multimodal fusion output respectively;

[0062] Compute the loss using the support dataset:

[0063]

[0064] Where CE stands for cross entropy loss, Represents the supporting dataset, is the average loss of the model on the support set;

[0065] Perform a fast gradient update based on the support set data:

[0066]

[0067] in, represents the fast update parameter, which is used as the input for subsequent multi-step updates, α is the learning rate, It is the gradient output of the multimodal hidden information extraction network and the multimodal fusion processing network;

[0068] Calculate the loss function and update the parameters at step k:

[0069]

[0070] Where i represents the number of tasks, is the gradient output of the i-th task step k-1 of the text,

[0071] It is the first step input parameter of multi-step update

[0072] Update the meta-learning parameters. After completing the multi-step update of the support set in the previous step, it is necessary to verify the parameters of the multimodal hidden information extraction network and the multimodal fusion processing network and perform meta-update. In this step, the query data set is used to calculate the loss function, perform back propagation, and finally update the meta parameters. The specific mathematical expression is as follows:

[0073] Calculate the loss function for the query set:

[0074]

[0075] Where CE stands for cross entropy loss, represents the query set dataset, is the average loss of the model on the query set;

[0076] Calculate the final loss function for multiple tasks:

[0077]

[0078] Among them, loss q is the average loss function value on multiple task query sets, and N is the total number of tasks;

[0079] Update meta parameters:

[0080]

[0081] Among them, θ new represents the meta-parameters updated after one step of gradient descent, is the gradient of the loss on the query dataset with respect to the parameters θ.

[0082] In addition, the present invention provides an early rumor detection system based on small sample social big data cross-modal meta-learning, including:

[0083] Acquisition and feature extraction module: Acquire the multimodal information to be detected published on social media, use the multimodal feature extraction network to extract features from the multimodal information to be detected, and obtain multimodal feature information;

[0084] Hidden information extraction module: construct a multimodal hidden information extraction network; use the multimodal hidden information extraction network to extract hidden information from multimodal feature information to obtain multimodal hidden information;

[0085] Fusion processing module: Build a multimodal fusion processing network, use the multimodal fusion processing network to detect rumors on multimodal hidden information, and obtain preliminary detection results.

[0086] Final detection module: The multimodal hidden information extraction network and the multimodal fusion processing network are optimized through the meta-learning algorithm to obtain the optimized multimodal hidden information extraction network and the optimized multimodal fusion processing network. Rumor detection is performed based on the optimized multimodal hidden information extraction network and the optimized multimodal fusion processing network to obtain the final detection result.

[0087] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0088] The present invention realizes the detection of rumors and non-rumors by extracting multimodal hidden and performing deep fusion processing. In addition, the present invention realizes early detection of rumors with a small number of samples by applying the meta-learning algorithm to the multimodal hidden information extraction network and the multimodal fusion processing network. The meta-learning algorithm focuses on effective learning by using a small number of labeled samples or unlabeled samples; by learning multiple tasks, a learning strategy is constructed so that when facing new tasks, model parameters can be quickly adjusted or new features can be learned, thereby improving learning efficiency and then improving accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 This is a flow chart of the early rumor detection method based on small sample social big data cross-modal meta-learning as described in Example 1;

[0090] Figure 2 This is a diagram of the multimodal feature extraction network structure described in Example 2;

[0091] Figure 3 This is a diagram of the multimodal hidden information extraction network structure described in Example 2;

[0092] Figure 4 This is a diagram of the multimodal fusion processing network structure described in Example 2;

[0093] Figure 5 This is a block diagram of the early rumor detection system based on small sample social big data cross-modal meta-learning described in Example 2; DETAILED DESCRIPTION

[0094] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0095] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0096] Embodiment 1:

[0097] This embodiment provides Figure 1 The early rumor detection method based on small sample social big data cross-modal meta-learning includes:

[0098] S1: Obtain the multimodal information to be detected published on social media, use the multimodal feature extraction network to extract features from the multimodal information to obtain multimodal feature information;

[0099] S2: construct a multimodal hidden information extraction network; use the multimodal hidden information extraction network to extract hidden information from multimodal feature information to obtain multimodal hidden information;

[0100] S3: Build a multimodal fusion processing network, use the multimodal fusion processing network to detect rumors on multimodal hidden information, and obtain preliminary detection results;

[0101] S4: The multimodal hidden information extraction network and the multimodal fusion processing network are optimized through a meta-learning algorithm to obtain an optimized multimodal hidden information extraction network and an optimized multimodal fusion processing network. Rumor detection is performed based on the optimized multimodal hidden information extraction network and the optimized multimodal fusion processing network to obtain the final detection result.

[0102] This embodiment realizes the detection of rumors and non-rumors by extracting multimodal hidden and performing deep fusion processing. In addition, this embodiment realizes early detection of rumors with a small number of samples by applying the meta-learning algorithm to the multimodal hidden information extraction network and the multimodal fusion processing network. The meta-learning algorithm focuses on effective learning by using a small number of labeled samples or unlabeled samples; by learning multiple tasks, a learning strategy is constructed so that when faced with new tasks, model parameters can be quickly adjusted or new features can be learned, thereby improving learning efficiency and accuracy.

[0103] Embodiment 2:

[0104] This embodiment further discloses the following on the basis of the first embodiment:

[0105] Furthermore, in step S1, it includes: preprocessing the data for the multimodal information to be detected, deleting redundant content, including at least one of the following: special characters, URL hyperlinks, HTML characters, and emoticons.

[0106] In step S1, the multimodal information to be detected includes text modality and image modality; wherein the text modality adopts different types of languages;

[0107] Let the set P = {T, V1, V2, ..., V N ,} represents the multimodal information to be detected, {T} represents the text modality, {V1,V2,...,V N} represents the image modality; the text modality consists of a sequence of words, represented by T = {W1, W2, ..., W N ,};

[0108] The multimodal feature information includes text feature information and image feature information;

[0109] The multimodal hidden information extraction network includes a text hidden information extraction network and an image hidden information extraction network.

[0110] The multimodal feature extraction network includes a text feature extraction network and an image feature extraction network;

[0111] Furthermore, a multimodal feature extraction network is used to extract features from the multimodal information to be detected, and multimodal feature information is obtained, including:

[0112] like Figure 2 The multimodal feature extraction network shown includes a text feature extraction network and an image feature extraction network:

[0113] Use VGG16 as the image feature extraction network to extract features from image information and obtain image feature information

[0114] Use BERT as the text feature extraction network to extract features from text information and obtain text feature information:

[0115] P T =BERT{W1,W2,...,W N}

[0116] P V =VGG{V1,V2,...,V N}

[0117] Among them, P T is the output feature information after BERT encoding, with a dimension of 393216, which is text feature information; P V It is the final layer output of the VGG model, with a dimension of 4096, which is image feature information. It takes text feature information and image feature information as multimodal feature information.

[0118] Further, in step S2, if Figure 3 The multimodal hidden information extraction network shown in the figure includes:

[0119] Two Bi-LSTM bidirectional long short-term memory networks and a series of hidden layer structures are constructed as a text hidden information extraction network. The hidden information extraction of text feature information includes:

[0120] Reduce the dimensionality of the input text:

[0121] x=ReLU{W1P T +b1}

[0122] Among them, x is the output result after dimension conversion, W1 is the weight matrix of the activation function network, and b1 is the bias vector;

[0123] The reduced text dimension features are input into two Bi-LSTM bidirectional long short-term memory networks:

[0124]

[0125] output=cat(output1,output2)

[0126] Among them, output, output, and output2 are the output of the double-layer LSTM and the cascade output in the feature dimension, respectively. is the initial hidden state and cell state of LSTM, is the final hidden state and cell state of LSTM;

[0127] Through two hidden layers with the same structure, take the first layer as an example:

[0128] output layer1 1 =LayerNorm1(output)

[0129] output layer1 2 =ReLU(fc1(output layer1 1 ))

[0130] output layer1 3 =Dropout(output layer1 2 )

[0131] The final output of the second hidden layer is output layer2 3 ;

[0132] Through a fully connected layer, output text hidden information:

[0133] output text =fc3(output layer2 3 )

[0134] After processing the last fully connected layer, the final output dimension is 128.

[0135] Furthermore, constructing a multimodal hidden information extraction network includes:

[0136] Use the fully connected layer to build an image hidden information extraction network to extract hidden information from image feature information, including:

[0137] output3=Linear(W2P V +b2)

[0138] output image =Linear(W3ouput3+b3)

[0139] Among them, W2 is the weight matrix of the first fully connected layer, b2 is the bias vector, and P V represents the input image features, with a dimension of 4096; W3 is the weight matrix of the second fully connected layer, b3 is the bias vector, and the final output size is 128 dimensions.

[0140] Furthermore, constructing a multimodal fusion processing network includes: Figure 4 As shown, a multi-layer perception network is used to construct a multimodal fusion processing network.

[0141] Furthermore, a multimodal fusion processing network is used to detect rumors on multimodal hidden information, and the detection results include:

[0142] Concatenating multimodal hidden information:

[0143] output=cat(output text ,output image )

[0144] Among them, output text Hide information for text, output image Hide information for the image, output is output text And output image The cascade output of represents multimodal hidden information;

[0145] Through two hidden layers with the same structure, take the first layer as an example:

[0146] output layer1 1 =Linear(output)

[0147] output layer1 2 =ReLU(W4output layer1 1 +b4)

[0148] output layer1 3 =Dropout(output layer12 ,p1=0.5)

[0149] Among them, W4 and b4 are the weight matrix and bias vector respectively, and p1 is the pooling probability, indicating that 50% of neurons are randomly separated;

[0150] Get the rumor probability through full connection:

[0151] output rumor =W6output layer2 2 +b6

[0152] Among them, W6 and b6 are the weight matrix and bias vector of the output layer, respectively. layer2 2 is the final output of the second hidden layer, output rumor It is a two-dimensional array that stores the probabilities of rumors and non-rumors, and obtains the detection results based on the rumor probabilities.

[0153] Furthermore, in step S4, it includes:

[0154] Initialize the parameters of the multimodal hidden information extraction network and the multimodal fusion processing network:

[0155] θ=(θ text ,θ image ,θ classifier )

[0156] Among them, θ text ,θ image ,θ classifier They represent the initial parameters of the text hidden information extraction network, image hidden information extraction network and multimodal fusion output respectively;

[0157] Compute the loss using the support dataset:

[0158]

[0159] Where CE stands for cross entropy loss, Represents the supporting dataset, is the average loss of the model on the support set;

[0160] Perform a fast gradient update based on the support set data:

[0161]

[0162] in, represents the fast update parameter, which is used as the input for subsequent multi-step updates, α is the learning rate, It is the gradient output of the multimodal hidden information extraction network and the multimodal fusion processing network;

[0163] Calculate the loss function and update the parameters at step k:

[0164]

[0165] Where i represents the number of tasks, is the gradient output of the i-th task step k-1 of the text,

[0166] It is the first step input parameter of multi-step update

[0167] Update the meta-learning parameters. After completing the multi-step update of the support set in the previous step, it is necessary to verify the parameters of the multimodal hidden information extraction network and the multimodal fusion processing network and perform meta-update. In this step, the query data set is used to calculate the loss function, perform back propagation, and finally update the meta parameters. The specific mathematical expression is as follows:

[0168] Calculate the loss function for the query set:

[0169]

[0170] Where CE stands for cross entropy loss, represents the query set dataset, is the average loss of the model on the query set;

[0171] Calculate the final loss function for multiple tasks:

[0172]

[0173] Among them, loss q is the average loss function value on multiple task query sets, and N is the total number of tasks;

[0174] Update meta parameters:

[0175]

[0176] Among them, θ new represents the meta-parameters updated after one step of gradient descent, is the gradient of the loss on the query dataset with respect to the parameters θ.

[0177] This embodiment extracts multimodal hidden information and performs deep fusion processing to detect rumors and non-rumors. In addition, this embodiment applies the meta-learning algorithm to the multimodal hidden information extraction network and the multimodal fusion processing network to achieve early detection of rumors with a small number of samples. The meta-learning algorithm focuses on effective learning by using a small number of labeled samples or unlabeled samples; by learning multiple tasks, a learning strategy is constructed so that when faced with new tasks, model parameters can be quickly adjusted or new features can be learned, thereby improving learning efficiency and accuracy.

[0178] Embodiment three:

[0179] This embodiment provides Figure 2 The early rumor detection system based on small sample social big data cross-modal meta-learning is shown, including:

[0180] Acquisition and feature extraction module: Acquire the multimodal information to be detected published on social media, use the multimodal feature extraction network to extract features from the multimodal information to be detected, and obtain multimodal feature information;

[0181] Hidden information extraction module: construct a multimodal hidden information extraction network; use the multimodal hidden information extraction network to extract hidden information from multimodal feature information to obtain multimodal hidden information;

[0182] Fusion processing module: Build a multimodal fusion processing network, use the multimodal fusion processing network to detect rumors on multimodal hidden information, and obtain preliminary detection results.

[0183] Final detection module: The multimodal hidden information extraction network and the multimodal fusion processing network are optimized through the meta-learning algorithm to obtain the optimized multimodal hidden information extraction network and the optimized multimodal fusion processing network. Rumor detection is performed based on the optimized multimodal hidden information extraction network and the optimized multimodal fusion processing network to obtain the final detection result.

[0184] This embodiment extracts multimodal hidden information and performs deep fusion processing to detect rumors and non-rumors. In addition, this embodiment applies the meta-learning algorithm to the multimodal hidden information extraction network and the multimodal fusion processing network to achieve early detection of rumors with a small number of samples. The meta-learning algorithm focuses on effective learning by using a small number of labeled samples or unlabeled samples; by learning multiple tasks, a learning strategy is constructed so that when faced with new tasks, model parameters can be quickly adjusted or new features can be learned, thereby improving learning efficiency and accuracy.

[0185] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. An early rumor detection method based on small sample social big data cross-modal meta-learning, characterized by: include: S1: Obtain the multimodal information to be detected published on social media, use the multimodal feature extraction network to extract features from the multimodal information to be detected, and obtain multimodal feature information; S2: construct a multimodal hidden information extraction network; use the multimodal hidden information extraction network to extract hidden information from multimodal feature information to obtain multimodal hidden information; S3: Build a multimodal fusion processing network, use the multimodal fusion processing network to detect rumors on multimodal hidden information, and obtain preliminary detection results; S4: The multimodal hidden information extraction network and the multimodal fusion processing network are optimized through a meta-learning algorithm to obtain an optimized multimodal hidden information extraction network and an optimized multimodal fusion processing network. Rumor detection is performed based on the optimized multimodal hidden information extraction network and the optimized multimodal fusion processing network to obtain the final detection result.

2. The early rumor detection method based on small sample social big data cross-modal meta-learning according to claim 1 is characterized in that: In step S1, it includes: preprocessing the data of the multimodal information to be detected, deleting redundant content, including at least one of the following: special characters, URL hyperlinks, HTML characters and emoticons.

3. The early rumor detection method based on small sample social big data cross-modal meta-learning according to claim 1 is characterized in that: In step S1, the multimodal information to be detected includes text modality and image modality; wherein the text modality adopts different types of languages; Let the set P = {T, V1, V2, ..., V N ,} represents the multimodal information to be detected, {T} represents the text modality, {V1,V2,...,V N } represents the image modality; the text modality consists of a sequence of words, represented by T = {W1, W2, ..., W N ,}; The multimodal feature information includes text feature information and image feature information; The multimodal hidden information extraction network includes a text hidden information extraction network and an image hidden information extraction network; The multimodal feature extraction network includes a text feature extraction network and an image feature extraction network.

4. The early rumor detection method based on small sample social big data cross-modal meta-learning according to claim 3 is characterized in that: In step S1, a multimodal feature extraction network is used to extract features from the multimodal information to be detected, and multimodal feature information is obtained, including: The multimodal feature extraction network includes a text feature extraction network and an image feature extraction network; Use VGG16 as the image feature extraction network to extract features from image information and obtain image feature information Use BERT as the text feature extraction network to extract features from text information and obtain text feature information: P T =BERT{W1,W2,...,W N } P V =VGG{V1,V2,...,V N } Among them, P T is the output feature information after BERT encoding, with a dimension of 393216, which is text feature information; P V It is the final layer output of the VGG model, with a dimension of 4096, which is image feature information. It takes text feature information and image feature information as multimodal feature information.

5. The early rumor detection method based on small sample social big data cross-modal meta-learning according to claim 3 is characterized in that: In step S2, constructing a multimodal hidden information extraction network includes: Two Bi-LSTM bidirectional long short-term memory networks and a series of hidden layer structures are constructed as a text hidden information extraction network. The hidden information extraction of text feature information includes: Reduce the dimensionality of the input text: x=ReLU{W1P T +b1} Among them, x is the output result after dimension conversion, W1 is the weight matrix of the activation function network, and b1 is the bias vector; The reduced text dimension features are input into two Bi-LSTM bidirectional long short-term memory networks: output=cat(output1,output2) Among them, output, output, and output2 are the output of the double-layer LSTM and the cascade output in the feature dimension, respectively. is the initial hidden state and cell state of LSTM, is the final hidden state and cell state of LSTM; Through two hidden layers with the same structure, take the first layer as an example: output layer1 1 =LayerNorm1(output) output layer1 2 =ReLU(fc1(output layer1 1 )) output layer1 3 =Dropout(output layer1 2 ) The final output of the second hidden layer is output layer2 3 ; Through a fully connected layer, output text hidden information: output text =fc3(output layer2 3 ) After processing the last fully connected layer, the final output dimension is 128.

6. The early rumor detection method based on small sample social big data cross-modal meta-learning according to claim 3 is characterized in that: In step S2, constructing a multimodal hidden information extraction network includes: Use the fully connected layer to build an image hidden information extraction network to extract hidden information from image feature information, including: output3=Linear(W2P V +b2) output image =Linear(W3ouput3+b3) Among them, W2 is the weight matrix of the first fully connected layer, b2 is the bias vector, and P V represents the input image features, with a dimension of 4096; W3 is the weight matrix of the second fully connected layer, b3 is the bias vector, and the final output size is 128 dimensions.

7. The early rumor detection method based on small sample social big data cross-modal meta-learning according to claim 1 is characterized in that: In step S3, constructing a multimodal fusion processing network includes: constructing a multimodal fusion processing network using a multi-layer perception network.

8. The early rumor detection method based on small sample social big data cross-modal meta-learning according to claim 1 is characterized in that: Using a multimodal fusion processing network, rumor detection is performed on multimodal hidden information, and the detection results include: Concatenating multimodal hidden information: output=cat(output text ,output image ) Among them, output text Hide information for text, output image Hide information for the image, output is output text And output image The cascade output of represents multimodal hidden information; Through two hidden layers with the same structure, take the first layer as an example: output layer1 1 =Linear(output) output layer1 2 =ReLU(W4output layer1 1 +b4) output layer1 3 =Dropout(output layer1 2 ,p1=0.5) Among them, W4 and b4 are the weight matrix and bias vector respectively, and p1 is the pooling probability, indicating that 50% of neurons are randomly separated; Get the rumor probability through full connection: output rumor =W6output layer2 2 +b6 Among them, W6 and b6 are the weight matrix and bias vector of the output layer, respectively. layer2 2 is the final output of the second hidden layer, output rumor It is a two-dimensional array that stores the probabilities of rumors and non-rumors, and obtains the detection results based on the rumor probabilities.

9. The early rumor detection method based on small sample social big data cross-modal meta-learning according to claim 1 is characterized in that: In step S4, it includes: Initialize the parameters of the multimodal hidden information extraction network and the multimodal fusion processing network: θ=(θ text ,i image ,i classifier ) Among them, θ text ,θ image ,θ classifier They represent the initial parameters of the text hidden information extraction network, image hidden information extraction network and multimodal fusion processing respectively; Compute the loss using the support dataset: Where CE stands for cross entropy loss, Represents the supporting dataset, is the average loss of the model on the support set; Perform a fast gradient update based on the support set data: in, represents the fast update parameter, which is used as the input for subsequent multi-step updates, α is the learning rate, It is the gradient output of the multimodal hidden information extraction network and the multimodal fusion processing network; Calculate the loss function and update the parameters at step k: Where i represents the number of tasks, is the gradient output of the i-th task step k-1 of the text, It is the first step input parameter of multi-step update Update the meta-learning parameters. After completing the multi-step update of the support set in the previous step, it is necessary to verify the parameters of the multimodal hidden information extraction network and the multimodal fusion processing network and perform meta-update. In this step, the query data set is used to calculate the loss function, perform back propagation, and finally update the meta parameters. The specific mathematical expression is as follows: Calculate the loss function for the query set: Where CE stands for cross entropy loss, represents the query set dataset, is the average loss of the model on the query set; Calculate the final loss function for multiple tasks: Among them, loss q is the average loss function value on multiple task query sets, and N is the total number of tasks; Update meta parameters: Among them, θ new represents the meta-parameters updated after one step of gradient descent, is the gradient of the loss on the query dataset with respect to the parameters θ.

10. An early rumor detection system based on small sample social big data cross-modal meta-learning, characterized by: include: Acquisition and feature extraction module: Acquire the multimodal information to be detected published on social media, use the multimodal feature extraction network to extract features from the multimodal information to be detected, and obtain multimodal feature information; Hidden information extraction module: construct a multimodal hidden information extraction network; use the multimodal hidden information extraction network to extract hidden information from multimodal feature information to obtain multimodal hidden information; Fusion processing module: build a multimodal fusion processing network, use the multimodal fusion processing network to detect rumors on multimodal hidden information, and obtain preliminary detection results; Final detection module: The multimodal hidden information extraction network and the multimodal fusion processing network are optimized through a meta-learning algorithm to obtain an optimized multimodal hidden information extraction network and an optimized multimodal fusion processing network. Rumor detection is performed based on the optimized multimodal hidden information extraction network and the optimized multimodal fusion processing network to obtain the final detection result.

Images