Counterfeit information detection method combined with big language model knowledge distillation

By calculating the weighting and aggregating features of cross entropy loss, combined with the LoRA optimization method, the shortcomings of knowledge distillation of large language models in knowledge-intensive tasks are solved, and the accuracy and applicability of small language models in falsified information detection are improved.

CN119088973AActive Publication Date: 2024-12-06SHENZHEN KIM DAI INTELLIGENCE INNOVATION TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411574096.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-06
Publication Date
2024-12-06
Estimated Expiration
2044-11-06

AI Technical Summary

Technical Problem

The existing knowledge distillation technology is difficult to simultaneously transfer the external knowledge and reasoning and judgment capabilities of large language models to small language models, especially in knowledge-intensive tasks, and lacks adjustments to model classification differences, which affects the accuracy of falsified information detection. .

Method used

By calculating the cross entropy loss of labels and real labels predicted by large language model, the basis generated by large language model is given weights, and the basis features with weights are aggregated with image text features and input into the small language model to achieve the final classification and knowledge distillation, using Transformer text encoder and Resnet network to extract features, and fine-tune student models in combination with LoRA optimization method.

Benefits of technology

The knowledge distillation of large language models is realized for knowledge-intensive tasks, improves the prediction accuracy of small language models in forged information detection, can accurately learn the basis of teacher models and classify them, and adjust the basis weight to improve prediction accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088973A_ABST
    Figure CN119088973A_ABST
Patent Text Reader

Abstract

The invention discloses a forged information detection method combined with big language model knowledge distillation. The method comprises the following steps: S1, preprocessing image text features; s2, predicting a large language model; s3, feature aggregation; s4, knowledge distillation; by calculating cross entropy loss of a tag predicted by a large language model and a real tag, a basis for generating the large language model is endowed with a weight, and basis features with the weight and image text features are aggregated and input into a small language model classifier, so that final classification is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of natural language understanding, and in particular to a forged information detection method combining knowledge distillation of a large language model. Background Art

[0002] At present, although the large language model LLM has reasoning ability and interpretability, it is challenging to apply it in real life because of its huge size and high price. The current large language model LLM consists of 500B parameters, requiring at least 350GB of GPU memory and complex computing workload. In order to solve this challenge, knowledge distillation KD technology has become a research focus to solve this problem. Using the large language model LLM as a teacher model to provide guidance for the small language model SLM, and migrating the reasoning and interpretation capabilities of the large language model LLM to the small language model SLM can not only reduce the model size and computing workload, but also take advantage of the feature extraction capabilities of the small language model SLM. At the same time, the small language model SLM can also generate explanations while extracting features and classification.

[0003] Among the existing research methods, the commonly used methods of knowledge distillation KD technology include black-box-based KD and white-box-based KD. Black-box-based KD uses the encoding sequence of the teacher model as the training data of the student model, and then uses the cross entropy loss of the classification result of the student model as the loss constraint of the training; while white-box-based KG minimizes the output distance between the teacher model and the student model, and the KL divergence constraint is often used here.

[0004] Although KD technology can transfer the reasoning, judgment and interpretation capabilities of LLM to SLM, the existing KD technology is not suitable for knowledge-intensive reasoning tasks. How to simultaneously distill LLM's external knowledge and reasoning and judgment capabilities and inject external knowledge into SLM is the research focus of this application. Secondly, many previous literatures combining LLM and SLM have studied the advantages of combining LLM and SLM, and the respective roles of LLM and SLM in forged information detection. Studies have shown that LLM provides guidance for SLM classification, and SLM is suitable for final classification. However, few studies have studied the classification differences between LLM and SLM, and adjusted the feature weights of the final SLM classification prediction based on the differences. Summary of the invention

[0005] The present invention overcomes the shortcomings of the prior art and provides a method for detecting forged information combined with knowledge distillation of a large language model. By calculating the cross entropy loss between the labels predicted by the large language model LLM and the true labels, weights are assigned to the basis generated by the large language model LLM. The weighted basis features and image text features are aggregated and input into a small language model classifier to achieve final classification.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for detecting forged information combined with knowledge distillation of a large language model, characterized by:

[0008] S1, image text feature preprocessing, extracting text features and image features of input text and image information respectively;

[0009] S2. Prediction of the large language model: Based on the input text and image information, the large language model in the teacher model generates a modal correlation explanation of the text and image and retrieval evidence of the text modality, and obtains the teacher model reasoning basis and teacher model prediction results by predicting the large language model based on the modal correlation explanation and the retrieval evidence of the text modality;

[0010] S3, feature aggregation, extract the text features of the teacher model reasoning basis in the teacher model, calculate the cross quotient loss of the teacher model prediction result and the actual label of the input image text, and assign weights to the text features of the teacher model reasoning basis according to the cross quotient loss, aggregate the text features, image features and the text features of the teacher model reasoning basis after assigning weights, obtain the teacher model aggregate features, and input the teacher model aggregate features into the teacher model small language model classifier to obtain the teacher model classification result; and the modal correlation explanation in the teacher model, the retrieval evidence of the text modality, the teacher model reasoning basis, the text modality and the image modality constitute the data set of knowledge distillation;

[0011] S4. Knowledge distillation. Fine-tune the large language model in the student model to generate the student model reasoning basis and the student model prediction results according to the knowledge distillation dataset, extract the text features of the teacher model reasoning basis, calculate the cross quotient loss of the student model prediction results and the actual label of the input image text, and assign weights to the text features of the student model reasoning basis according to the cross quotient loss. Aggregate the text features, image features and the text features of the student model reasoning basis after assigning weights to obtain the student model aggregated features, and input the student model aggregated features into the student model small language model classifier to obtain the student model classification results.

[0012] The forged information detection method combined with knowledge distillation of a large language model as described above is characterized in that: in S1, the text features of the input text and image information are extracted by a transformer text encoder to obtain the text features , extract the image features of the input text and image information through the Resnet network image encoder to obtain the image features .

[0013] The forged information detection method combined with knowledge distillation of a large language model as described above is characterized in that: S2 includes

[0014] S21. Explain the modal relevance of text modality and image modality. The large language model used to explain the relevance of image and text generates complementary information of images and texts to explain the truth or falsity of the input text and image information, as well as a description of the relevance of the image and text according to the prompt. ;

[0015] S22, retrieval evidence in text mode, the large language model generates a search title and a search phrase according to the prompt, and the search engine retrieves external evidence according to the title and the search phrase, and obtains the text content of the evidence;

[0016] S23, prediction results, input the search evidence, as well as the description of the association between the image and the text, into the prediction large language model, and obtain the teacher model prediction results and the teacher model reasoning basis according to the prompts.

[0017] The forged information detection method combined with knowledge distillation of a large language model as described above is characterized in that: S3 includes

[0018] S31. Weights are assigned to the text features based on which the teacher model is inferred, and the text features based on which the teacher model is inferred are extracted, denoted as , and then calculate the actual label of the input image And the teacher model prediction results The cross entropy loss is calculated as:

[0019] = log +(1- )log(1- )

[0020] set up The weight of the feature is (1- ), , the text features based on which the teacher model is reasoned after giving weights It is expressed as:

[0021] = (1- ) ;

[0022] S32, feature aggregation, , , The aggregated features m are concatenated and passed through the small language model classifier of the teacher model to obtain the classification result of the teacher model.

[0023] The method for detecting forged information by combining knowledge distillation with a large language model is characterized in that: a dataset of knowledge distillation is set as ; S4 includes

[0024] S41, generation of student model reasoning basis and student model prediction results, the knowledge distillation dataset Input into the fine-tuned large language model MiniGPTv2, and generate the student model reasoning basis according to the prompts And the student model prediction results ;

[0025] S42, prediction of the student model, extracting the reasoning basis of the student model The text features of , and then calculate the actual label of the input image And the student model prediction results The cross entropy loss is calculated as:

[0026] = log +(1- )log(1- )

[0027] set up The weight of the feature is (1- ), , the text features based on which the weighted student model is reasoned It is expressed as:

[0028] = (1- ) ;

[0029] Bundle , , The aggregated features m are concatenated and passed through the small language model classifier of the student model to obtain the classification result of the student model.

[0030] The method for detecting forged information combined with knowledge distillation of a large language model as described above is characterized in that: in S22, the being search engine is used to retrieve external evidence according to the title and the search phrase, and finally the newspaper library in Python is used to obtain the text content of the evidence.

[0031] The method for detecting forged information combined with knowledge distillation of a large language model as described above is characterized in that: in S31, a Transformer text encoder is used to extract text features of the basis.

[0032] The method for detecting forged information combined with knowledge distillation of a large language model as described above is characterized in that: a Softmax classifier is used to complete classification in S32.

[0033] The forged information detection method combined with knowledge distillation of a large language model as described above is characterized in that: in S41, a Transformer text encoder is used to extract the reasoning basis of the student model The text features of the student model are optimized by using the LoRA optimization method to optimize the rank decomposition component of the Transformer layer in the large language model in the student model. The output y containing L tags is The generated fine-tuned large language model MiniGPTv2 model is modeled:

[0034] p(y|T,V, )= ( )

[0035] in, As a reminder, Represents input text and images, represents a trainable parameter, P represents the assigned probability, and y represents the predicted label.

[0036] The method for detecting forged information combined with knowledge distillation of a large language model as described above is characterized in that: a Softmax classifier is used to complete classification in S42.

[0037] The beneficial effects of the present invention are:

[0038] 1. The present invention makes knowledge distillation of large language models suitable for knowledge-intensive tasks.

[0039] 2. The present invention combines the respective advantages of the large language model and the small language model in forged information detection, and provides guidance for the small language model through the large language model, so that the small language model can make accurate predictions.

[0040] 3. The student model of the present invention can not only learn the basis for teacher model generation, but also achieve classification at the same time to improve prediction accuracy.

[0041] 4. The present invention adjusts the weight of the generation basis according to the cross entropy loss between the classification results of the large language model and the actual labels to improve the prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of the present invention. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings.

[0044] It should be noted that all directional indications (such as up, down, left, right, front, back...) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly. In addition, the descriptions of "preferred", "sub-preferred", etc. in the present invention are only used for descriptive purposes, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "preferred" or "sub-preferred" may explicitly or implicitly include at least one such feature.

[0045] like Figure 1 As shown in FIG, a forged information detection method combining knowledge distillation of a large language model includes image text feature preprocessing, large language prediction, feature aggregation and knowledge distillation. Specifically,

[0046] S1. Image text feature preprocessing, using transformer text encoder to extract text features and obtain text features ; Use the Resnet network image encoder to extract image features and obtain image features ;

[0047] S2. Large language model LLM accurate prediction. First, based on the input image and text information, the modal association between the image and text is explained. Then, the large language model LLM is used to generate a search phrase for the text modality. The search phrase uses the Bing search engine to retrieve evidence. Finally, the modal association is explained and the retrieved evidence is input into the accurate prediction large language model LLM to obtain the teacher model reasoning basis. And accurate teacher model prediction results ;

[0048] S3, feature aggregation, using transformer text encoder to represent the teacher model reasoning basis The basis characteristics are ; First calculate the teacher model prediction results And the actual label of the input image The smaller the cross entropy loss, the more accurate the prediction of the large language model LLM. Therefore, a larger weight is given to the reasoning basis of the large language model LLM, and vice versa. The text features of the teacher model reasoning basis after weighting are Then, , ,and Aggregation obtains the aggregated feature m, and finally the aggregated feature is input into the classifier to achieve the final result. In this process, it is necessary to save the retrieval evidence of the text modality of the teacher model, the reasoning basis of the teacher model, the description of the modality relevance of the teacher model, the image and text modalities, and construct a knowledge distillation dataset for the knowledge distillation step, denoted as .

[0049] S4, knowledge distillation. Fine-tune the student model by maximizing the likelihood of the sequence generated by the large language model LLM of the student model to ensure that the prediction results of the student model and the teacher model are consistent. , fine-tuning an off-the-shelf language model following its original autoregressive training objective. To facilitate this fine-tuning process, a LoRA optimization approach is employed, in which the language model is frozen and only the rank decomposition components of the Transformer layers are optimized.

[0050] like Figure 1 As shown in Figure 2, S2 includes the modality relevance, retrieval evidence, and accurate prediction of the image and text modalities.

[0051] S21. Explain the modal relevance of image and text modalities. Input the image and text information of the news to the large language model LLM that explains the relevance of images and texts, and prompt the learning template to be "Please describe the connection between the [IMAGE] and [TEXT] and provide complementary information between images and text which is conducive to determining the authenticity of news", so that the large language model LLM that explains the relevance of images and texts generates complementary information of images and texts that explain the authenticity of news, as well as a description of the relevance of images and texts. ;

[0052] S22. Retrieve evidence. First, let the large language model LLM generate search titles and search phrases according to the prompts. Then, based on the titles and search phrases, use the being search engine to retrieve external evidence. Finally, use the newspaper library in Python to obtain the text content of the evidence.

[0053] S23. Accurate prediction. The retrieved evidence and the description of the association between the image and the text are input into the large language model LLM for accurate prediction. According to the prompt learning template, the results and basis of the accurate prediction of the large language model LLM are obtained. The prompt learning template is: "The connection between the image and text, and the complementary information between them are [CONNECTION], then the [EVIDENCE] provides more information to verify the authenticity of the news. Please generate the label of the news is fake or true and the explanations for the judgement." In this case, if only the prediction result generated by the large language model LLM is used as the final prediction, hallucinations will occur, resulting in a decrease in experimental performance. Therefore, it is necessary to combine the small language model SLM for the final classification. At the same time, since the basis generated by LLM can guide SLM classification, the subsequent steps need to use a text encoder to represent the features of the generated basis, and the features of the generated basis and , Aggregate and input into the classifier to obtain the final prediction result.

[0054] like Figure 1 As shown, S3 includes reasoning based on feature weighting and feature fusion, specifically

[0055] S31, weights are assigned based on the inference feature. The Transformer text encoder is used to extract the text features based on the inference feature, denoted as , and then calculate the actual label of the input image And the teacher model prediction results The cross entropy loss is calculated as:

[0056] = log +(1- )log(1- ) (1)

[0057] Since the smaller the cross entropy, the higher the classification accuracy, the more important the basis is, so we set The weight of the feature is (1- ), The text features that the teacher model uses for reasoning after assigning weights The characteristic expression is:

[0058] = (1- ) (2)

[0059] S32, feature fusion. , , The features are concatenated to form an aggregate feature m, and the Softmax classifier is used to complete the final classification.

[0060] like Figure 1 As shown, S4 includes the reasoning basis for generating the student model And the prediction results of Minigptv2 , the final prediction of the student model.

[0061] S41、Student model reasoning basis And the student model prediction results The motivation of knowledge distillation is to make the reasoning and prediction results generated by the student model as aligned as possible with the teacher model. Therefore, the goal of training and fine-tuning the student model is to maximize the possibility of the teacher model's large language model LLM generating sequences. Following its original autoregressive training objective, the LoRA optimization method is adopted, in which the language model is frozen and only the rank decomposition component of the Transformer layer is optimized. According to the dataset , construct a prompt learning template based on the teacher language model. For the output y containing L tags, according to the dataset The generated fine-tuned large language model MiniGPTv2 model is modeled:

[0062] p(y|T,V, )= ( ) (3)

[0063] To prompt the learning template, Represents input text and images, represents a trainable parameter, P represents the assigned probability, and y represents the predicted label;

[0064] Datasets for knowledge distillation Input into the fine-tuned large language model MiniGPTv2, and generate the student model reasoning basis according to the prompts And the student model prediction results ;

[0065] S42, the final prediction of the student model. Following the teacher model, the Transformer text encoder is used to extract the reasoning basis of the student model The text features of , and then calculate the actual label of the input image And the student model prediction results The cross entropy loss is calculated as:

[0066] = log +(1- )log(1- )

[0067] set up The weight of the feature is (1- ), , the text features based on which the student model is reasoned after giving weights It is expressed as:

[0068] = (1- ) ;

[0069] Bundle , , The aggregated features m are concatenated and the final classification is completed through the student model small language model classifier to obtain the student model classification result.

[0070] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or directly or indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A method for detecting forged information by combining knowledge distillation of a large language model, characterized in that: Included S1, image text feature preprocessing, extracting text features and image features of input text and image information respectively; S2. Prediction of the large language model: Based on the input text and image information, the large language model in the teacher model generates a modal correlation explanation of the text and image and retrieval evidence of the text modality, and obtains the teacher model reasoning basis and teacher model prediction results by predicting the large language model based on the modal correlation explanation and the retrieval evidence of the text modality; S3, feature aggregation, extract the text features of the teacher model reasoning basis in the teacher model, calculate the cross quotient loss of the teacher model prediction result and the actual label of the input image text, and assign weights to the text features of the teacher model reasoning basis according to the cross quotient loss, aggregate the text features, image features and the text features of the teacher model reasoning basis after assigning weights, obtain the teacher model aggregate features, and input the teacher model aggregate features into the teacher model small language model classifier to obtain the teacher model classification result; and the modal correlation explanation in the teacher model, the retrieval evidence of the text modality, the teacher model reasoning basis, the text modality and the image modality constitute the data set of knowledge distillation; S4. Knowledge distillation. Fine-tune the large language model in the student model to generate the student model reasoning basis and the student model prediction results according to the knowledge distillation dataset, extract the text features of the teacher model reasoning basis, calculate the cross quotient loss of the student model prediction results and the actual label of the input image text, and assign weights to the text features of the student model reasoning basis according to the cross quotient loss. Aggregate the text features, image features and the text features of the student model reasoning basis after assigning weights to obtain the student model aggregated features, and input the student model aggregated features into the student model small language model classifier to obtain the student model classification results.

2. The method for detecting forged information by combining knowledge distillation with a large language model according to claim 1, characterized in that: In S1, the text features of the input text and image information are extracted through the transformer text encoder to obtain the text features , extract the image features of the input text and image information through the Resnet network image encoder to obtain the image features .

3. The method for detecting forged information by combining knowledge distillation with a large language model according to claim 2, characterized in that: S2 includes S21. Explain the modal relevance of text modality and image modality. The large language model used to explain the relevance of image and text generates complementary information of images and texts to explain the truth or falsity of the input text and image information, as well as a description of the relevance of the image and text according to the prompt. ; S22, retrieval evidence in text mode, the large language model generates a search title and a search phrase according to the prompt, and the search engine retrieves external evidence according to the title and the search phrase, and obtains the text content of the evidence; S23, prediction results, input the search evidence, as well as the description of the association between the image and the text, into the prediction large language model, and obtain the teacher model prediction results and the teacher model reasoning basis according to the prompts.

4. The method for detecting forged information by combining knowledge distillation with a large language model according to claim 3, characterized in that: S3 includes S31. Weights are assigned to the text features based on which the teacher model is inferred, and the text features based on which the teacher model is inferred are extracted, denoted as , and then calculate the actual label of the input image And the teacher model prediction results The cross entropy loss is calculated as: = log +(1- )log(1- ) set up The weight of the feature is (1- ), , the text features based on which the teacher model is reasoned after giving weights It is expressed as: = (1- ) ; S32, feature aggregation, , , The aggregated features m are concatenated and passed through the small language model classifier of the teacher model to obtain the classification result of the teacher model.

5. The method for detecting forged information by combining knowledge distillation with a large language model according to claim 4, characterized in that: The dataset for knowledge distillation is set as ; S4 includes S41, generation of student model reasoning basis and student model prediction results, the knowledge distillation dataset Input into the fine-tuned large language model MiniGPTv2, and generate the student model reasoning basis according to the prompts And the student model prediction results ; S42, prediction of the student model, extracting the reasoning basis of the student model The text features of , and then calculate the actual label of the input image And the student model prediction results The cross entropy loss is calculated as: = log +(1- )log(1- ) set up The weight of the feature is (1- ), , the text features based on which the weighted student model is reasoned It is expressed as: = (1- ) ; Bundle , , The aggregated features m are concatenated and passed through the small language model classifier of the student model to obtain the classification result of the student model.

6. The method for detecting forged information by combining knowledge distillation with a large language model according to claim 3, characterized in that: In S22, the being search engine is used to retrieve external evidence based on the title and search phrase, and finally the newspaper library in Python is used to obtain the text content of the evidence.

7. The method for detecting forged information by combining knowledge distillation with a large language model according to claim 4, characterized in that: In S31, the Transformer text encoder is used to extract text features.

8. The method for detecting forged information by combining knowledge distillation with a large language model according to claim 4, characterized in that: In S32, classification is done using the Softmax classifier.

9. The method for detecting forged information by combining knowledge distillation with a large language model according to claim 5, characterized in that: In S41, the Transformer text encoder is used to extract the reasoning basis of the student model Text features of The LoRA optimization method is used to optimize the rank decomposition component of the Transformer layer in the large language model in the student model; for the output y containing L tokens, according to the dataset The generated fine-tuned large language model MiniGPTv2 model is modeled: p(y|T,V, )= ( ) in, As a reminder, Represents input text and images, represents a trainable parameter, P represents the assigned probability, and y represents the predicted label.

10. The method for detecting forged information combined with knowledge distillation of a large language model according to claim 5, characterized in that: In S42, classification is performed using the Softmax classifier.

Citation Information

Patent Citations

  • Language model training method and device based on knowledge distillation and text classification method and device

    CN111506702A

  • Student model training method based on pre-training language model and text classification system

    CN115526332A

  • Knowledge distillation method and system based on multi-teacher multi-modal model

    CN117669693A

  • Information detection method combining multi-modal knowledge graph and multi-modal large language model

    CN118839019A

Cited By

  • False news detection method and system based on big and small model collaboration

    CN120822108A

  • Smart park information generation method and device, equipment, storage medium and product

    CN121636695A