Expert guidance type multi-mode mutual distillation false news detection method and device oriented to field deviation alleviation

Abstract

The invention discloses an expert guidance type multi-mode mutual distillation false news detection method and device oriented to domain deviation alleviation. According to the method, domain specific features are modeled through an expert-guided teacher model, so that domain deviation caused by unbalanced distribution of multi-modal data is reduced. The teacher model integrates image-text features through an attention mechanism and a gating fusion module, and dynamic field normalization is introduced to realize adaptive adjustment of feature distribution. Student models are respectively designed for each field, a prototype memory library is introduced to store field features, and cross-field knowledge sharing is promoted through a mutual distillation mechanism. And finally, a double-strategy knowledge distillation framework is adopted, soft label distillation and comparative representation distillation are combined, so that the student model efficiently learns fusion knowledge of the teacher model in a unified multi-modal embedding space, and prediction distribution and feature representation alignment is realized. According to the method, the domain deviation can be effectively reduced, and the accuracy and fairness of cross-domain false news detection are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cross-technology of natural language processing and computer vision, in particular to an expert-guided multi-modal mutual distillation fake news detection method and device for field bias alleviation. BACKGROUND

[0002] With the rapid development and extensive use of the Internet, the efficiency and breadth of information dissemination have been significantly improved, and emerging platforms such as social media have gradually become the core channel for the public to obtain and share information. However, while technological progress brings convenience for dissemination, it also provides favorable conditions for the generation and spread of false information. Fake news often appears in the form of text and images, prompting multi-modal fake news detection to become a research hotspot.

[0003] Although existing multi-modal fake news detection methods have made significant progress in overall performance and exhibit strong recognition ability, most researches mainly focus on the average effect of the model on the global data set, ignoring the performance difference problem between different fields. For example, the model performs well on political news but performs poorly on health or entertainment news, affecting the fairness and robustness of actual application. Therefore, how to reduce the field bias and improve the cross-field detection performance has become an important challenge, and the current multi-modal fake news detection faces the following challenges:

[0004] 1) Multi-modal information is heterogeneous, and the feature structure difference between text and image leads to fusion difficulty;

[0005] 2) There is a large semantic gap between different modal information, which is difficult to model in a unified semantic space;

[0006] 3) Fake news samples are unevenly distributed across fields, and the model has significant performance differences between different fields.

[0007] Therefore, the present application proposes an expert-guided multi-modal mutual distillation fake news detection method for field bias alleviation to alleviate the field bias problem in multi-modal fake news detection. SUMMARY

[0008] The purpose of the present application is to solve the deficiencies of the prior art and provide an expert-guided multi-modal mutual distillation fake news detection method and device for field bias alleviation. Through multi-level knowledge distillation and dynamic field normalization mechanism, feature alignment and field debiasing are achieved, cross-field detection performance and fairness are improved, and the spread of false information is reduced, protecting the public's right to know and social stability.

[0009] The application is realized by the technical scheme as follows: a field bias alleviation-oriented expert guidance type multi-modal mutual distillation fake news detection method, comprising a multi-modal feature extraction module, a teacher module, a student module and a fake news classifier, and realized by a multi-modal fake news detection device, comprising the following steps:

[0010] 1) input the news text and image into the pre-trained text encoder and image encoder respectively, extract single-modal features, and unify the coding of the text and image through the CLIP model to obtain multi-modal feature representation;

[0011] 2) integrate the multi-modal features through the attention mechanism and the gate fusion module of the teacher model, and adaptively adjust the feature distribution of each field based on the dynamic field normalization module;

[0012] 3) independently train the student model of each field, and dynamically update the feature prototype of each field through the prototype memory bank;

[0013] 4) encourage each field student model to learn the output distribution of other field student models through the mutual distillation mechanism, and promote knowledge sharing between different field student models;

[0014] 5) through double-strategy knowledge distillation, the student model effectively learns the multi-modal fusion knowledge of the teacher model in the unified multi-modal embedding space, so as to approximate the similarity of the prediction distribution and the alignment of the feature representation;

[0015] 6) input the output result of the student model into the fake news classifier to obtain the prediction label of whether the news is true or false.

[0016] Further, in order to better realize the field bias alleviation-oriented expert guidance type multi-modal mutual distillation fake news detection method, the following setting mode is particularly adopted: the step 1) comprises the following steps:

[0017] 1.1) for the text information in the news, use the pre-trained model BERT to extract the text representation X text , and generate the embedding through the MLP, and use the CLIP text encoder to obtain the aligned text feature T clip ;

[0018] 1.2) for the image information in the news, use the pre-trained MAE model to extract the image feature X image , and convert it into embedding representation through the fully connected layer, and use the CLIP image encoder to obtain the aligned image feature V clip ;

[0019] 1.3) Considering that there is a semantic gap between the text and image features, it is difficult to directly fuse them, so the CLIP model is introduced to uniformly encode the text and image, the text and image embeddings are spliced and input into the MLP network to obtain the multi-modal feature representation F clip .

[0020] Further to better implement the expert-guided multi-modal mutual distillation fake news detection method for field bias mitigation described in the application, the following setting method is particularly adopted: the step 2) comprises the following steps:

[0021] 2.1) The above teacher model comprises a mixed expert architecture, and given input and output features r∈{r t , r i , r m}, the mixed expert module is composed of m s shared experts and m k field experts, and the gating network is weighted combined according to the splicing of the input feature r and the field embedding r d : , wherein, is the gating network of field k, S k (r) is a set composed of outputs of all experts, denotes the representation after attention weighting;

[0022] 2.2) The dynamic field normalization module dynamically adjusts the parameters (scaling factor γ and offset term β) of normalization according to the field ID of the input sample, so as to map the features of different fields to a unified distribution space, and the normalization process is realized by the following formula: , wherein, denotes the feature after dynamic field normalization; f is an input feature vector; μ and σ are the mean and standard deviation of the feature respectively; is a learnable scaling factor and offset parameter depending on the field ID d; d∈{1,2,...,K} represents the field identifier, and K is the total number of fields.

[0023] Further to better implement the expert-guided multi-modal mutual distillation fake news detection method for field bias mitigation described in the application, the following setting method is particularly adopted: the step 3) comprises the following steps:

[0024] 3.1) A prototype vector p d is maintained for each field d to capture the feature distribution of each field, and the vector is dynamically updated during the training process to reflect the changes of the field feature distribution. The update process adopts a momentum update strategy, which is realized by the following formula: , wherein, B d denotes a sample set belonging to field d in the current training batch, where is the feature vector of sample i (from the domain adaptation layer), and is the momentum coefficient.

[0025] 3.2) Promote semantic consistency between different domain prototypes through prototype alignment loss, which is achieved by minimizing the mean square error (MSE) between any two different domain prototypes: where K is the total number of domains.

[0026] Further to better implement the expert-guided multi-modal mutual distillation fake news detection method for domain bias mitigation, the following setting method is particularly adopted: in step 4), for domains d and d' (d ≠ d'), we calculate the KL divergence between the student model output probability distribution of domain d and the student model output probability distribution of domain d': where is the softmax function, is the temperature parameter (used to smooth the probability distribution), and the KL divergence is calculated by .

[0027] Further to better implement the expert-guided multi-modal mutual distillation fake news detection method for domain bias mitigation, the following setting method is particularly adopted: step 5) includes the following steps:

[0028] 5.1) For the soft label distillation module, the student model approximates the softmax output of the teacher model by minimizing the KL divergence between its predicted distribution and the teacher model soft label. Introducing a temperature factor to scale to smooth the distribution helps the student model learn the implicit relationship between classes. This is achieved by: .

[0029] 5.2) For the contrastive learning distillation module, the student features and teacher features are mapped to a unified embedding space through a shared nonlinear projection head by: , then maximize the similarity of the student-teacher feature pair of the same sample (positive sample) while minimizing the similarity with other sample pairs (negative sample) by: .

[0030] An expert-guided multi-modal mutual distillation fake news detection device for domain bias mitigation, for implementing the expert-guided multi-modal mutual distillation fake news detection for domain bias mitigation, comprising:

[0031] Feature extraction module: for extracting text feature vectors from news and image feature vectors from images and multi-modal features;

[0032] The teacher model module is based on a hybrid expert network structure, realizes adaptive adjustment of field distribution through dynamic field normalization, and generates fused features.

[0033] The student model module trains a lightweight student model for each field, combines a prototype memory bank and a mutual distillation mechanism, retains each field specificity, and promotes field knowledge sharing.

[0034] The distillation optimization module uses soft label distillation and contrast distillation to jointly optimize the student model, so that the student model learns teacher knowledge in a unified embedding space.

[0035] The fake news classifier discriminates news samples to obtain the classification result of the news, and completes the fake news detection task.

[0036] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0037] (1) The present application adopts a hybrid expert architecture and a dynamic field normalization module in the teacher model, which can adaptively adjust the feature distribution of different fields, map heterogeneous multi-modal features to a unified semantic space, thereby reducing the performance difference between fields and improving the robustness and fairness of the model in cross-field scenarios.

[0038] (2) The present application realizes efficient sharing and alignment of field knowledge through a prototype memory bank and a mutual distillation strategy. The student model of each field is independently trained and dynamically updates the feature prototype, while the KL divergence minimization promotes the consistency of the output distribution of different field student models, thereby maintaining the field-specific expression ability while enhancing the generalization performance of the model to unknown field samples, realizing efficient sharing and migration of field knowledge.

[0039] (3) The present application adopts a double-strategy knowledge distillation (soft label distillation and contrast learning distillation), soft label distillation captures the implicit relationship between classes through temperature smoothing mechanism, contrast learning distillation optimizes feature alignment through positive and negative samples, thereby improving the feature representation quality and classification accuracy of the student model, approximating the performance of the teacher model, and enabling the student model to learn the multi-modal fusion knowledge of the teacher model in a unified embedding space. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A field bias alleviation-oriented expert-guided multi-modal mutual distillation fake news detection method flowchart is provided for the embodiments of the present application.

[0041] Figure 2 A field bias alleviation-oriented expert-guided multi-modal mutual distillation fake news detection network model architecture diagram is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0042] The application will be described in further detail below with reference to the drawings of the embodiments of the application, but the embodiments of the application are not limited thereto.

[0043] To make the objects, technical solutions and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings of the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the application. Therefore, the following detailed description of the embodiments of the application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the application.

[0044] In addition, the terms “first”, “second” are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with “first”, “second” can explicitly or implicitly include one or more of the features. In the description of the application, the meaning of “multiple” is two or more, unless otherwise explicitly and specifically limited.

[0045] First, combined with Figure 1 An expert-guided multi-modal mutual distillation fake news detection method for field bias mitigation according to an embodiment of the application is introduced as follows. Figure 1 As shown in the figure, the detection method includes:

[0046] Step S01, the text and image of the news are respectively input into different single-modal feature extraction modules to extract respective feature vectors, and text feature vectors and image feature vectors are obtained. In addition, the CLIP model is used to uniformly encode the text and image, the text and image embeddings are spliced and input into the MLP network to obtain multi-modal feature representation.

[0047] Specifically, for the text information in the news, the pre-trained model BERT is used to extract the text representation X text , and the MLP is used to generate the embedding, and the CLIP text encoder is used to obtain the aligned text feature T clip .

[0048] For the image information in the news, the pre-trained MAE model is used to extract the image feature X image , and the fully connected layer is used to convert it into an embedding representation, and the CLIP image encoder is used to obtain the aligned image feature V clip .

[0049] Considering the semantic gap between image and text features, direct fusion is difficult, so the CLIP model is introduced to unify the encoding of image and text, and the multi-modal feature representation F is obtained by splicing the image and text embedding and inputting it into the MLP network. clip .

[0050] In step S02, the attention mechanism of the teacher model is integrated with the gated fusion module to integrate multi-modal features, and the feature distribution of each domain is adaptively adjusted based on the dynamic domain normalization module.

[0051] Specifically, the teacher model includes a hybrid expert architecture, given the input and output features r∈{r t , r i , r m}, the hybrid expert module is composed of m s shared experts and m k domain experts, and the gating network is weighted combined according to the splicing of the input feature r and the domain embedding r d . , where is the gating network of domain k, S k (r) is a set composed of outputs of all experts, represents the representation after attention weighting. This fusion mechanism can capture complementary information across modalities and provide more rich features for fake news detection;

[0052] Through the dynamic domain normalization module, the parameters (scaling factor γ and offset term β) of the normalization are dynamically adjusted according to the domain ID of the input sample, so as to map the features of different domains to a unified distribution space. The normalization process is realized by the following formula: , where represents the feature after dynamic domain normalization; f is the input feature vector; μ and σ are the mean and standard deviation of the feature, respectively; is a learnable scaling factor and offset parameter dependent on the domain ID d; d∈{1,2,...,K} represents the domain identifier, and K is the total number of domains.

[0053] In step S03, the student model is trained independently for each domain, and the prototype memory bank is combined with the dynamic and mutual distillation mechanism to retain the specificity of each domain while promoting knowledge sharing.

[0054] Specifically, a prototype vector p d is maintained for each domain d to capture the feature distribution of each domain, and the vector is dynamically updated during training to reflect changes in the domain feature distribution. The update process uses a momentum update strategy, which is realized by the following formula: , where B dThis represents the set of samples belonging to domain d in the current training batch. Let λ be the feature vector of sample i (from the domain adaptation layer), and λ be the momentum coefficient.

[0055] Prototype alignment loss promotes semantic consistency between prototypes from different domains, which is achieved by minimizing the mean squared error (MSE) between any two prototypes from different domains. Where K is the total number of domains. This loss function encourages prototypes from different domains to be as close as possible in the feature space, thereby enhancing the model's cross-domain generalization ability;

[0056] To facilitate knowledge sharing among student models from different domains, we designed a mutual distillation mechanism. This mechanism encourages student models from each domain to learn the output distributions of student models from other domains. Specifically, for each input sample, we extract the output logits (i.e., unnormalized prediction scores) of each student model to compare their distribution differences. Subsequently, we quantify the differences in the output distributions of each student model using the KL divergence measure, and use this as an optimization objective to promote knowledge transfer between models. Mutual knowledge distillation, specifically, for domains d and d' (d ≠ d'), calculates the KL divergence between the output probability distributions of student models in domain d and d': Where σ is the softmax function, τ is the temperature parameter (used to smooth the probability distribution), and the KL divergence is calculated using the following formula: .

[0057] Step S04: Through dual-strategy knowledge distillation, the student model effectively learns the multimodal fusion knowledge of the teacher model in a unified multimodal embedding space, thereby approximating the similarity of its prediction distribution and the alignment of its feature representation.

[0058] For the soft label distillation module, the student model minimizes the KL divergence between its predicted distribution and the teacher model's soft labels, causing its output probability distribution to approximate the teacher model's output. Introducing a temperature factor τ for scaling to smooth the distribution helps the student model learn the implicit relationships between categories. This is achieved through the following formula: ;

[0059] The comparative learning distillation module is implemented using the following formula: Student and teacher features are mapped to a unified embedding space through a shared nonlinear projection head, and then processed by the following formula. The goal is to maximize the similarity of student-teacher feature pairs (positive samples) within the same sample while minimizing their similarity to other sample pairs (negative samples), thereby enabling the student model to better reproduce the feature space structure of the teacher model.

[0060] Step S05, input the output result of the student model into the fake news classifier to obtain a prediction label of whether the news belongs to true or fake, and finally complete the classification task.

[0061] Further, the present application is combined with specific embodiments, and is further described in detail with reference to the drawings:

[0062] Step 1: The pre-trained model extracts single-modal and multi-modal features of the news data respectively;

[0063] Step 2: The expert-guided teacher model combines attention mechanism and gating mechanism to fuse the multi-modal features, and adaptively adjusts the feature distribution of each field through dynamic field normalization;

[0064] Step 3: Each field trains a student model separately, and combines the prototype memory bank and mutual knowledge distillation mechanism to realize the reservation of field specificity while promoting knowledge sharing;

[0065] Step 4: Through the double-strategy knowledge distillation mechanism, that is, the combination of soft label knowledge distillation and contrast learning knowledge distillation mechanism, the student model can learn the knowledge of the teacher model more effectively;

[0066] Step 5: Finally, the result is sent to the fake news classifier to complete the classification task.

[0067] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification or equivalent change of the above embodiment according to the technical essence of the present application falls within the protection scope of the present application.

Claims

1. A field bias mitigation oriented expert-guided multi-modal mutual distillation fake news detection method, characterized in that, The method comprises the following steps: 1) inputting news text and images into pre-trained text encoder and image encoder respectively to extract single-modal features, and uniformly encoding the text and images through a CLIP model to obtain multi-modal feature representation; 2) integrating the multi-modal features through the attention mechanism of the teacher model and the gating fusion module, and adaptively adjusting the feature distribution of each domain based on the dynamic domain normalization module; 3) independently training the student model of each domain, and dynamically updating the feature prototype of each domain through the prototype memory bank; 4) encouraging the student model of each domain to learn the output distribution of the student model of other domains through the mutual distillation mechanism, and promoting knowledge sharing between the student models of different domains; 5) enabling the student model to effectively learn the multi-modal fusion knowledge of the teacher model in a unified multi-modal embedding space through double-strategy knowledge distillation, so as to approximate the similarity of the prediction distribution and the alignment of the feature representation; 6) inputting the output result of the student model into the fake news classifier to obtain the prediction label of whether the news is real or fake.

2. The domain-biased mitigation-oriented expert-guided multi-modal mutual distillation fake news detection method according to claim 1, characterized in that: The step 1) comprises the following specific steps: For text information in news, use pre-trained model BERT to extract text representation X text , and generate embedding through MLP, while using CLIP text encoder to obtain aligned text feature T clip ; For image information in news, a pre-trained MAE model is used to extract image features X image and is converted into an embedding representation through a fully connected layer, while using a CLIP image encoder to obtain aligned image features V clip ; Considering the semantic gap between the image and text features, direct fusion is difficult, so the CLIP model is introduced to uniformly encode the image and text, and the image and text embeddings are spliced and input into the MLP network to obtain the multi-modal feature representation F clip .

3. The field bias alleviated expert-guided multi-modal mutual distillation fake news detection method according to claim 1 or 2, characterized in that: The step 2) comprises the following specific steps: The teacher model includes a hybrid expert architecture, given input features r e {r t , r i , r m}, the hybrid expert module is composed of m s shared experts and m k domain experts, and the gating network is weighted combined according to the splicing of the input features r and the domain embedding r d ; The dynamic domain normalization module dynamically adjusts the parameters (scaling factor γ and offset term β) of normalization according to the domain ID of the input sample, so as to map the features of different domains to a unified distribution space.

4. The domain-biased mitigation expert-guided multi-modal mutual distillation fake news detection method according to claim 1, wherein: The step 3) comprises the following specific steps: By maintaining a prototype vector p for each domain d d Capture the feature distribution of each domain, which is dynamically updated with a momentum update strategy during training to reflect changes in the domain feature distribution; The semantic consistency between prototypes in different domains is promoted by a prototype alignment loss, which is achieved by minimizing the mean square error (MSE) between any two prototypes in different domains. The implementation formula is as follows: where K is the total number of domains.

5. The domain-biased mitigation expert-guided multi-modal mutual distillation fake news detection method according to claim 1, wherein: In step 4), for domains d and d' (d ≠ d'), we compute the KL divergence between the student model output probability distribution of domain d and the student model output probability distribution of domain d' by: where σ is the softmax function and τ is the temperature parameter (used to smooth the probability distribution).

6. The domain-biased mitigation expert-guided multi-modal mutual distillation fake news detection method according to claim 1, wherein: The step 5) comprises the following specific steps: For soft-label distillation module, the student model approximates the teacher model's softmax output by minimizing the KL divergence between its predictive distribution and the teacher model's soft labels. A temperature factor τ is introduced to scale and smooth the distribution, which helps the student model learn the underlying relationships between classes. This is achieved by: ​ The contrastive learning distillation module, first, the student feature and the teacher feature are mapped to a unified embedding space through a shared nonlinear projection head, and then the similarity of the student-teacher feature pair of the same sample (positive sample) is maximized, and the similarity with other sample pairs (negative sample) is minimized through the following formula: , so as to promote the student model to better reproduce the feature space structure of the teacher model.

7. A field bias mitigation oriented expert-guided multi-modal mutual distillation fake news detection apparatus, characterized in that, The device comprises: a feature extraction module for extracting text, image and multi-modal features; a teacher model module for fusing multi-modal features and performing dynamic domain normalization to guide the student model; a student model module using a prototype memory bank and a mutual distillation mechanism to retain each domain-specific while promoting domain knowledge sharing; a distillation optimization module using double-knowledge distillation strategies of soft label knowledge distillation and contrastive learning knowledge distillation to enable the student model to more effectively learn the knowledge of the teacher model; a fake news classifier for discriminating news samples to obtain the classification result of the news.

Images