Unsupervised cross-domain rumor detection method based on teacher-student learning framework

Abstract

The invention belongs to the technical field of rumor detection, and discloses an unsupervised cross-domain rumor detection method based on a teacher-student learning framework, and the method comprises the following steps: carrying out the supervised training of a teacher detection model through source domain data, synchronizing the trained parameters to a student detection model, and completing the parameter initialization of the model; performing pseudo label labeling on the target domain data by using the teacher detection model, and screening out a high-quality pseudo label of the target domain data; performing self-training on the target domain data by using a student detection model, and obtaining training loss of the target domain data by combining the cross entropy loss and the contrast learning loss of the high-quality pseudo tag; and S2, performing back propagation optimization on parameters of the student detection model by using the training loss of the target domain data, smoothly superposing the parameters of the student detection model subjected to training optimization into the parameters of the teacher detection model by using a moving index average algorithm, and returning to the step S2 to start the next round of training. According to the method, the unsupervised cross-domain rumor detection effect is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of rumor detection and relates to an unsupervised cross-domain rumor detection method based on a teacher-student learning framework. Background Art

[0002] The task of rumor detection generally refers to judging the authenticity of text content posted on social media. Most existing rumor detection technologies use social media information with authenticity labels to conduct supervised training on the model, which can enable the model to have the ability to detect rumors of similar topic information to a certain extent. However, in actual application scenarios, the ever-changing event topics on social media have large contextual differences, which makes it difficult for existing rumor detection technologies to extend the learned rumor feature detection patterns to other event topics, resulting in low detection accuracy in test data of different topics. Faced with massive amounts of social media information, it is obviously not realistic to label rumor detection training data for each subject area. Therefore, how to effectively detect rumor information in different subject areas in the absence of labeled data is a technical problem that needs to be solved urgently.

[0003] The differences in semantic features and propagation structure features of data in different subject areas are the main reason for the poor generalization in the field of rumor detection technology. Existing cross-domain rumor detection technologies mainly focus on strengthening the model's ability to represent semantic features, using the Transformer model to learn semantic features, and bridging the differences in semantic features between domains through unsupervised methods such as cross-attention and contrastive learning. However, the Transformer model is suitable for processing sequential data, and most existing cross-domain rumor detection technologies simplify the tree-like propagation structure of social media information into a text sequence, which will cause the loss of information propagation structure features and reduce the effectiveness of cross-domain rumor detection technology. Summary of the invention

[0004] In order to solve the technical problem that the existing cross-domain rumor detection technology may cause the loss of information propagation features and reduce the detection effectiveness, the present invention provides an unsupervised cross-domain rumor detection method based on a teacher-student learning framework. The method adopts an unsupervised self-training method, combined with a heavily weighted cross entropy loss function and two types of contrast learning loss functions, to impose consistency regularization constraints on the feature vectors output by the teacher detection model and the student detection model, thereby improving the effect of unsupervised cross-domain rumor detection.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides an unsupervised cross-domain rumor detection method based on a teacher-student learning framework, comprising the following steps:

[0007] S1: Construct a teacher detection model and a student detection model with the same structure;

[0008] S2: Obtain source domain data and target domain data;

[0009] S3: Use the source domain data to perform supervised training on the teacher detection model, and synchronize the trained parameters to the student detection model to complete the parameter initialization of the teacher detection model and the student detection model;

[0010] S4: Use the teacher detection model to annotate the target domain data with pseudo labels and select high-quality pseudo labels for the target domain data;

[0011] S5: Use the student detection model to self-train the target domain data, and combine the cross entropy loss and contrastive learning loss of high-quality pseudo labels to obtain the training loss of the target domain data;

[0012] S6: Use the training loss back propagation of the target domain data to optimize the student detection model parameters, and use the moving exponential average algorithm to smoothly superimpose the parameters of the trained and optimized student detection model onto the teacher detection model parameters, and return to step S2 to start the next round of training.

[0013] The teacher detection model and the student detection model of the present invention adopt two detection models with exactly the same structure, and each detection model is composed of a pre-trained language model and a graph convolutional neural network model. For the pre-trained language model, the Roberta model can be selected but not limited to be adopted. For the graph convolutional neural network model, the BiGCN model can be selected but not limited to be adopted. The model architecture of BiGCN is composed of two graph convolutional neural networks with the same structure, respectively called a top-down graph convolutional neural network and a bottom-up graph convolutional neural network, each of which contains two graph convolutional layers for processing directed graph input data in the corresponding direction. In addition, during the feature extraction process, the features of the root node in the previous layer will be spliced ​​with the features of all nodes in the next layer between each layer of the model to strengthen the semantic feature extraction of the root node, i.e., the original news information.

[0014] The present invention uses a propagation graph to represent social media information. Each propagation graph consists of a root node, a child node, and an edge. The root node is the original information, such as a blog, news, etc., the child node is the comment information, and the edge represents the interactive relationship between the comment information. The propagation graph is a directed graph, and the direction of the edge needs to be set to top-down and bottom-up, respectively, to simulate the propagation and aggregation of information. The source domain data and the target domain data are both divided into two categories: non-rumor and rumor. The non-rumor data is marked as 0 and the rumor data is marked as 1.

[0015] In the above technical solution of the present invention, the specific steps of using source domain data to perform supervised training on the teacher detection model in step S3 include:

[0016] Discard the edge of source domain data;

[0017] The root node text is concatenated with the text of each child node to form a node pair, and the text content of the node pair is encoded with semantic features;

[0018] Calculate the cross entropy loss of the teacher detection model and iteratively optimize the parameters of the teacher detection model; the cross entropy loss of the above teacher detection model The calculation formula is as follows:

[0019]

[0020] Where: |B Tea | is the amount of data for the current training round of the teacher detection model, y i is the source domain data, is the probability that the sample in the source domain data is a rumor.

[0021] In the above technical solution of the present invention, the above step S4 specifically includes the following steps:

[0022] S401: discarding edge data in the target domain;

[0023] S402: Pseudo-label the target domain data using the teacher detection model trained with the source domain data in step S3;

[0024] S403: Using the logical values ​​of non-rumor and rumor output by the teacher detection model as the confidence of pseudo label classification;

[0025] S404: according to the number of pseudo labels of non-rumors and rumors in the target domain data, set the upper and lower limits of the confidence of the pseudo labels to obtain the confidence thresholds of non-rumors and rumors in the target domain data;

[0026] S405: According to the confidence thresholds of non-rumors and rumors in the target domain data, the pseudo labels of non-rumors and rumors in the target domain data are screened respectively to obtain high-quality pseudo labels of non-rumors and rumors in the target domain data.

[0027] In the above technical solution of the present invention, the calculation formula of the non-rumor confidence threshold T0 in the target domain data is as follows:

[0028]

[0029] The calculation formula of the confidence threshold T1 of the rumor in the target domain data is as follows:

[0030]

[0031] Where: p high and plow are the upper and lower limits of the confidence of the pseudo-label, respectively. n0 and n1 are the number of non-rumors and rumors in the pseudo-labels of the target domain data in the current round, respectively.

[0032] In the above technical solution of the present invention, the above step S5 specifically includes the following steps:

[0033] S501: Based on the number of high-quality pseudo labels of non-rumors and rumors in the target domain data, an upper limit and a lower limit of the cross entropy loss weight are set to obtain the cross entropy loss weights of non-rumors and rumors in the high-quality pseudo labels;

[0034] S502: Use the student detection model to self-train the target domain data, combine the cross entropy loss weights and cross entropy loss of non-rumors and rumors in the high-quality pseudo-labels, and obtain the re-weighted cross entropy loss of the high-quality pseudo-labels;

[0035] S503: The training loss of the target domain data is obtained according to the cross entropy loss, the category-level contrastive learning loss and the sample-level contrastive learning loss reweighted by the high-quality pseudo-labels.

[0036] In the above technical solution of the present invention, the calculation formula of the cross entropy loss weight ω0 of the non-rumor in the high-quality pseudo-label is as follows:

[0037]

[0038] The calculation formula of the cross entropy loss weight ω1 of the rumor described in the high-quality pseudo-label is as follows:

[0039]

[0040] Where: high and ω low are the upper and lower bounds of the cross entropy loss weight, respectively. n'0 and n'1 are the number of non-rumors and rumors in the high-quality pseudo-labels of the target domain data in the current round.

[0041] In the above technical solution of the present invention, the cross entropy loss of the non-rumor in the high-quality pseudo-label is The calculation formula is as follows:

[0042]

[0043] Cross entropy loss for rumors in high-quality pseudo-labels The calculation formula is as follows:

[0044]

[0045] The cross entropy loss of high-quality pseudo-label reweighted The calculation formula is as follows:

[0046]

[0047] in: is high-quality pseudo-label data, N p is the number of high-quality pseudo labels, is the probability of predicting a sample as a rumor among samples with high-quality pseudo labels.

[0048] In the above technical solution of the present invention, the category-level contrastive learning loss The calculation formula is as follows:

[0049]

[0050] The sample-level contrastive learning loss The calculation formula is as follows:

[0051]

[0052] Where: In the category-level contrastive learning loss, k is the target domain data with high-quality pseudo labels in the current round, is the feature vector of k output from the student detection model, N p is the number of high-quality pseudo labels, τ is the temperature coefficient, and sim(·) represents the cosine similarity; is the set of positive samples in the current round of target domain data with the same prediction probability as the high-quality pseudo-label, a is an element in the positive sample set, is the feature vector of a output from the teacher detection model, q is the high-quality pseudo label traversed during the current round of training, is the feature vector of q output from the teacher detection model;

[0053] In sample-level contrastive learning loss, |B Stu | is the amount of data for the current round of training of the student detection model, m is the target domain data for the current round of training, n is the target domain data traversed during the current round of training, and are the output feature vectors of m output from the teacher detection model and the student detection model, respectively, is the feature vector of n output from the Student detection model.

[0054] In the above technical solution of the present invention, the training loss L of the target domain data is T The calculation formula is as follows:

[0055]

[0056] in: is the cross entropy loss reweighted by the high-quality pseudo-labels, is the category-level contrastive learning loss, is the sample-level contrastive learning loss, λ1, λ2 and λ3 are the weights of each loss, λ1+λ2+λ3=1.

[0057] In a second aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the processor and the memory communicate with each other via a bus; the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the above-mentioned unsupervised cross-domain rumor detection method based on the teacher-student learning framework.

[0058] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned unsupervised cross-domain rumor detection method based on a teacher-student learning framework.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] Driven by the pre-trained language model and graph convolutional neural network model, the method of the present invention overcomes the differences in semantic and propagation structure characteristics of social media information between different subject areas. By designing dynamic screening of high-quality pseudo-labels and introducing contrastive learning loss, the teacher detection model's ability to guide the student detection model in the training of target domain data in each round of training is enhanced.

[0061] The method of the present invention adopts an unsupervised self-training approach, combined with a heavily weighted cross-entropy loss function and two types of contrastive learning loss functions, to impose consistency regularization constraints on the feature vectors output by the teacher detection model and the student detection model, thereby improving the effect of unsupervised cross-domain rumor detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a schematic diagram of the principle of the unsupervised cross-domain rumor detection method based on the teacher-student learning framework of the present invention. DETAILED DESCRIPTION

[0063] The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention. Unless otherwise specified, the technical means used in the examples are conventional means known to those skilled in the art. The test methods in the following examples are conventional methods unless otherwise specified.

[0064] Embodiment 1

[0065] refer to Figure 1 The schematic diagram of the unsupervised cross-domain rumor detection method based on the teacher-student learning framework of the present invention is shown. The unsupervised cross-domain rumor detection method based on the teacher-student learning framework of this embodiment specifically includes the following steps:

[0066] S1: Use the Roberta model and BiGCN model to build a teacher detection model and a student detection model with the same structure, with θ Tea represents the teacher detection model, with θ Stu Represents the student detection model.

[0067] S2: Get the source domain data and target domain data (as shown in Table 1), and use D S Denotes the labeled source domain dataset, with D T It represents an unlabeled target domain dataset. Both the source domain data and the target domain data are divided into two categories: non-rumor data and rumor data. The non-rumor data is marked as 0 and the rumor data is marked as 1.

[0068] Table 1 Dataset data

[0069]

[0070]

[0071] S3: The source domain data of the input propagation graph structure is calculated according to the probability p drop Edge discard, p drop Set it to 0.1, concatenate the root node text with each child node text to form a node pair, use the pre-trained language model to encode the semantic features of the text content of the node pair, and then use formula (1) to calculate the cross entropy loss of the teacher detection model Iteratively optimize the teacher detection model parameters, and synchronize the trained parameters to the student detection model to complete the parameter initialization of the teacher detection model and the student detection model.

[0072] S4: Use the teacher detection model to annotate the target domain data in different subject areas with pseudo labels and select high-quality pseudo labels. Step S4 specifically includes the following steps:

[0073] S401: The target domain data of the propagation graph structure is processed according to the probability p drop Edge discard, p drop Set to 0.1.

[0074] S402: Use the teacher detection model trained on the source domain data in step S3 to pseudo-label the target domain data.

[0075] S403: Using the logical values ​​of non-rumor and rumor output by the teacher detection model as the confidence of the pseudo label classification.

[0076] S404: Based on the number of pseudo labels of non-rumors and rumors in the target domain data, i.e., n0 and n1, set the upper and lower limits of the confidence of the pseudo labels, i.e., p high =0.9, p low=0.7, and substitute it into formula (2) and formula (3) to calculate the confidence thresholds of non-rumor and rumor in the target domain data, namely T0 and T1.

[0077] S405: According to the confidence thresholds of non-rumors and rumors in the target domain data, the pseudo labels of non-rumors and rumors in the target domain data are screened respectively to obtain high-quality pseudo labels of non-rumors and rumors in the target domain data.

[0078] S5: Use the student detection model to self-train the target domain data, and combine the cross entropy loss and contrastive learning loss of high-quality pseudo labels to obtain the training loss of the target domain data. Step S5 specifically includes the following steps:

[0079] S501: Combined with the number of non-rumors and rumors in the high-quality pseudo-labels, i.e., n'0 and n'1, set the upper and lower limits of the cross entropy loss weights of non-rumors and rumors, i.e., ω high =3,ω low =1, and substitute it into formula (4) and formula (5) to obtain the cross entropy loss weights of rumors and non-rumors in high-quality pseudo labels, namely ω0 and ω1.

[0080] S502: Use formula (6) and formula (7) to calculate the cross entropy loss of non-rumor and rumor in high-quality pseudo-labels respectively and Substitute it into formula (8) to calculate the cross entropy loss of high-quality pseudo-label reweighting

[0081] S503: Calculate the category-level contrastive learning loss according to formula (9) and formula (10) and sample-level contrastive learning loss Set λ1 = 0.8, λ2 = λ3 = 0.1, and substitute into formula (11) to calculate the training loss L of the target domain data T .

[0082] S6: Use the training loss back propagation of the target domain data to optimize the student detection model parameters, and use the moving exponential average algorithm to smoothly superimpose the parameters of the trained and optimized student detection model onto the teacher detection model parameters, and return to step S2 to start the next round of training.

[0083] It is worth noting that in the first round of training, the upper limit of the cross entropy loss weight ω between non-rumors and rumors needs to be set high and the lower limit ω low , and the upper limit of the cross entropy loss weight ω between non-rumor and rumor high and the lower limit ω lowIn each subsequent round of training, the system will dynamically adjust these upper and lower limits according to the high-quality pseudo labels obtained in each round of training to balance the quality and quantity of pseudo labels of target domain data and achieve dynamic screening of high-quality pseudo labels.

[0084] The training rounds are set to 200 times, and the experimental results are as follows:

[0085] As shown in Table 2, WCT represents the test results obtained according to the steps of this embodiment. WCT-init represents the test results of the model proposed in the present invention in the target domain data, which is trained only in the source domain data. TD-RvNN, BU-RvNN, Bi-GCN, PLAN, PPA-WAE, and PPA-WAE are the results of different detection methods that are only trained in the source domain data and tested in the target domain data; UCD-RD is the test result of another unsupervised cross-domain rumor detection method. Difference represents the difference in detection performance between the model proposed in the present invention and WCT-init after completing the target domain training. It can be seen that the test results of the present invention (WCC) are significantly better than other detection methods.

[0086] Table 2 Test results of unsupervised cross-domain rumor detection method

[0087]

[0088]

[0089] The embodiments described above are only preferred embodiments of the present invention and are only used to explain the present invention, not to limit the scope of implementation of the present invention. For those skilled in the art, other implementation methods can certainly be easily made by replacement or modification based on the technical contents disclosed in this specification. Therefore, all changes and improvements made on the principles of the present invention should be included in the scope of the patent application of the present invention.

Claims

1. An unsupervised cross-domain rumor detection method based on a teacher-student learning framework, characterized in that: The following steps are involved: S1: Construct a teacher detection model and a student detection model with the same structure; S2: Obtain source domain data and target domain data; S3: Use the source domain data to perform supervised training on the teacher detection model, and synchronize the trained parameters to the student detection model to complete the parameter initialization of the teacher detection model and the student detection model; S4: Use the teacher detection model to annotate the target domain data with pseudo labels and select high-quality pseudo labels for the target domain data; S5: Use the student detection model to self-train the target domain data, and combine the cross entropy loss and contrastive learning loss of high-quality pseudo labels to obtain the training loss of the target domain data; S6: Use the training loss back propagation of the target domain data to optimize the student detection model parameters, and use the moving exponential average algorithm to smoothly superimpose the parameters of the trained and optimized student detection model onto the teacher detection model parameters, and return to step S2 to start the next round of training.

2. According to claim 1, the unsupervised cross-domain rumor detection method based on the teacher-student learning framework is characterized in that: The source domain data and the target domain data are divided into two categories: non-rumor and rumor, and the non-rumor data is marked as 0 and the rumor data is marked as 1.

3. The unsupervised cross-domain rumor detection method based on the teacher-student learning framework according to claim 1 or 2 is characterized in that: The specific steps of using source domain data to supervise the teacher detection model in step S3 include: Discard the edge of source domain data; The root node text is concatenated with the text of each child node to form a node pair, and the text content of the node pair is encoded with semantic features; Calculate the cross entropy loss of the teacher detection model and iteratively optimize the parameters of the teacher detection model; the cross entropy loss of the teacher detection model The calculation formula is as follows: Where: |B Tea | is the amount of data for the current training round of the teacher detection model, y i is the source domain data, is the probability that the sample in the source domain data is predicted to be a rumor.

4. The unsupervised cross-domain rumor detection method based on the teacher-student learning framework according to claim 1 or 2 is characterized in that: Step S4 specifically includes the following steps: S401: discarding edge data in the target domain; S402: Pseudo-label the target domain data using the teacher detection model trained with the source domain data in step S3; S403: Using the logical values ​​of non-rumor and rumor output by the teacher detection model as the confidence of pseudo label classification; S404: according to the number of pseudo labels of non-rumors and rumors in the target domain data, set the upper and lower limits of the confidence of the pseudo labels to obtain the confidence thresholds of non-rumors and rumors in the target domain data; S405: According to the confidence thresholds of non-rumors and rumors in the target domain data, the pseudo labels of non-rumors and rumors in the target domain data are screened respectively to obtain high-quality pseudo labels of non-rumors and rumors in the target domain data.

5. According to claim 4, the unsupervised cross-domain rumor detection method based on the teacher-student learning framework is characterized in that: The calculation formula of the non-rumor confidence threshold T0 in the target domain data is as follows: The calculation formula of the confidence threshold T1 of the rumor in the target domain data is as follows: Where: p high and p low are the upper and lower limits of the confidence of the pseudo-label, respectively. n0 and n1 are the number of non-rumors and rumors in the pseudo-labels of the target domain data in the current round, respectively.

6. The unsupervised cross-domain rumor detection method based on the teacher-student learning framework according to claim 1 or 2 is characterized in that: Step S5 specifically includes the following steps: S501: Based on the number of high-quality pseudo labels of non-rumors and rumors in the target domain data, an upper limit and a lower limit of the cross entropy loss weight are set to obtain the cross entropy loss weights of non-rumors and rumors in the high-quality pseudo labels; S502: Use the student detection model to self-train the target domain data, combine the cross entropy loss weights and cross entropy loss of non-rumors and rumors in the high-quality pseudo-labels, and obtain the re-weighted cross entropy loss of the high-quality pseudo-labels; S503: The training loss of the target domain data is obtained according to the cross entropy loss, the category-level contrastive learning loss and the sample-level contrastive learning loss reweighted by the high-quality pseudo-labels.

7. The unsupervised cross-domain rumor detection method based on the teacher-student learning framework according to claim 6 is characterized in that: The calculation formula of the cross entropy loss weight ω0 of the non-rumor in the high-quality pseudo-label is as follows: The calculation formula of the cross entropy loss weight ω1 of the rumor described in the high-quality pseudo-label is as follows: Where: high and ω low are the upper and lower bounds of the cross entropy loss weight, respectively. n'0 and n'1 are the number of non-rumors and rumors in the high-quality pseudo-labels of the target domain data in the current round.

8. The unsupervised cross-domain rumor detection method based on the teacher-student learning framework according to claim 6 or 7 is characterized in that: Cross entropy loss of non-rumor in high-quality pseudo-labels The calculation formula is as follows: Cross entropy loss for rumors in high-quality pseudo-labels The calculation formula is as follows: The cross entropy loss of high-quality pseudo-label reweighted The calculation formula is as follows: in: is high-quality pseudo-label data, N p is the number of high-quality pseudo labels, is the probability of predicting a sample as a rumor among samples with high-quality pseudo labels.

9. The unsupervised cross-domain rumor detection method based on the teacher-student learning framework according to claim 6 or 7 is characterized in that: The class-level contrastive learning loss The calculation formula is as follows: The sample-level contrastive learning loss The calculation formula is as follows: Where: In the category-level contrastive learning loss, k is the target domain data with high-quality pseudo labels in the current round, is the feature vector of k output from the student detection model, N p is the number of high-quality pseudo labels, τ is the temperature coefficient, and sim(·) represents the cosine similarity; is the set of positive samples in the current round of target domain data with the same prediction probability as the high-quality pseudo-label, a is an element in the positive sample set, is the feature vector of a output from the teacher detection model, q is the high-quality pseudo label traversed during the current round of training, is the feature vector of q output from the teacher detection model; In sample-level contrastive learning loss, |B Stu | is the amount of data for the current round of training of the student detection model, m is the target domain data for the current round of training, n is the target domain data traversed during the current round of training, and are the output feature vectors of m output from the teacher detection model and the student detection model, respectively, is the feature vector of n output from the Student detection model.

10. The unsupervised cross-domain rumor detection method based on the teacher-student learning framework according to claim 1 or 6, characterized in that: The training loss L of the target domain data T The calculation formula is as follows: in: is the cross entropy loss reweighted by the high-quality pseudo-labels, is the category-level contrastive learning loss, is the sample-level contrastive learning loss, λ1, λ2 and λ3 are the weights of each loss, λ1+λ2+λ3=1.

Images