Forgery information detection method based on dynamic knowledge distillation

Abstract

The invention discloses a forged information detection method based on dynamic knowledge distillation, which comprises the following steps: small language model evidence retrieval: enabling a first small language model to obtain a search phrase generation capability and an evidence retrieval capability from a large language model; classifying positive basis and negative basis, performing positive and negative division on reasoning basis in a knowledge distillation process, simultaneously distilling the positive basis and the negative basis by using a comparative learning method, and endowing proper weights according to importance degrees; and multi-task learning is adopted to learn the ability of label prediction and the ability of basis generation, so that the accuracy of information detection and judgment is greatly improved.

Description

[Technical field]

[0001] The invention relates to a forged information detection method based on dynamic knowledge distillation. [Background Technology]

[0002] Large language models (LLMs) have powerful reasoning capabilities and interpretability, and are widely used in the field of forged information detection. However, large language models (LLMs) have a large number of parameters and high computational costs. In order to solve this problem, a knowledge distillation (KD) method is proposed to transfer the reasoning ability of large language models (LLMs) to small language models (SLMs), allowing small language models (SLMs) to imitate large language models (LLMs) to generate reasoning evidence and reasoning labels. However, due to the small number of parameters of small language models (SLMs) and limited ability to store knowledge, directly inputting the knowledge retrieved by large language models (LLMs) into small language models (SLMs) will result in insufficient memory. Therefore, most of the existing forged information detection methods that use knowledge distillation (KD) only distill the reasoning basis of large language models (LLMs) to judge the authenticity of news, and cannot distill the ability of large language models (LLMs) to retrieve evidence and use evidence to make inferences and judgments into small language models (SLMs), resulting in low accuracy in forged information detection. [Summary of the invention]

[0003] The present invention overcomes the shortcomings of the prior art and provides a method for detecting forged information based on dynamic knowledge distillation.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A forged information detection method based on dynamic knowledge distillation, characterized by:

[0006] S1. Small language model evidence retrieval: The reasoning ability of the large language model is transferred to the first small language model through knowledge distillation. The first small language model generates a small language model search phrase based on the news text, and retrieves the small language model search phrase through the search engine to obtain the small language model evidence text.

[0007] S2. Classify positive and negative evidence, construct positive and negative evidence sets, and design a discriminator trained using contrastive learning loss. Use the positive and negative evidence sets as the training set data for the discriminator. Use the discriminator to distinguish positive and negative evidence from the inference evidence generated by the news text and the large language model based on the news text, and assign weights.

[0008] S3. Multi-task learning, including basis tasks and label tasks. The basis task enables the second small language model to learn to generate reasoning basis knowledge from the positive and negative basis generated by the large language model identified by the discriminator and the small language model evidence text. The label task realizes the alignment of the classification label generated by the second small language model and the true label.

[0009] The method for detecting forged information based on dynamic knowledge distillation is characterized in that: in S1, the difference between the search phrase generated by the first small language model and the search phrase generated by the large language model is measured by a loss formula,

[0010] The loss formula is calculated as:

[0011]

[0012] f1 represents the search phrase generated by the first small language model, x i Represents news text, q i represents the search phrase generated by the large language model, and l represents the cross entropy loss.

[0013] The method for detecting forged information based on dynamic knowledge distillation as described above is characterized in that the first small language model in S1 is retrieved from a newspaper library in Python through a search engine.

[0014] The method for detecting forged information based on dynamic knowledge distillation as described above is characterized in that the search engine in S1 is a being search engine.

[0015] The forged information detection method based on dynamic knowledge distillation is characterized in that: S2 includes

[0016] S21. Construct a set of positive and negative evidence. The large language model outputs reasoning evidence and prediction labels based on the news text, and divides them into positive evidence and negative evidence according to the consistency with the actual label, and constructs a set of positive evidence and negative evidence. Set the set of negative evidence to The set of positive evidence is

[0017] S22, contrast learning strategy, using S generated in S1 neg , S pos Train the discriminator. The input of the discriminator is the news text and the reasoning basis. The encoder structure is used to measure the score. The calculation formula is:

[0018]

[0019] X represents news text, and Represents the scores of positive and negative grounds respectively.

[0020] The forged information detection method based on dynamic knowledge distillation is characterized in that: S3 includes

[0021] S31. Based on the task, based on the learning, use the many-to-one contrastive learning distillation loss to distill the positive and negative evidence generated by the large language model discriminated by the discriminator into the second small language model at the same time. The calculation formula is:

[0022]

[0023] x i , Respectively represent the input text, positive evidence and negative evidence of the i-th large language model, N represents the number of samples, β is a hyperparameter greater than 0, l(.) represents the cross entropy loss, and f(.) represents the generation of evidence for a given input;

[0024] Optimize the training process with quality-guided distillation, and The calculation formula is as follows:

[0025]

[0026] Represent the positive basis score and negative basis score respectively,

[0027] S32, labeling task, calculate the cross entropy loss between the label predicted by the first small language model and the true label. The calculation formula is:

[0028] L cls = l(f(x i ), y i )

[0029] The total training loss is L cls , L distill and L prob The sum is calculated as:

[0030] L total =L cls +L distill +L prob .

[0031] The beneficial effects of the present invention are:

[0032] The present invention makes the knowledge distillation KD of the large language model LLM suitable for knowledge-intensive tasks, and allows the first small language model SLM1 to obtain the ability to generate search phrases and retrieve evidence from the large language model LLM; at the same time, in the process of knowledge distillation KD, the reasoning basis is divided into positive and negative, and the positive basis and the negative basis are distilled simultaneously using the comparative learning method, and appropriate weights are assigned according to the importance; on the other hand, the student model in this case has multi-task learning capabilities, which can learn both the ability to predict labels and the ability to generate basis from the teacher model. [Drawings]

[0033] Figure 1 This is a schematic diagram of the present invention. [Specific implementation method]

[0034] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings.

[0035] It should be noted that all directional indications (such as up, down, left, right, front, back...) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly. In addition, the descriptions of "preferred", "sub-preferred", etc. in the present invention are only used for descriptive purposes, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "preferred" or "sub-preferred" may explicitly or implicitly include at least one such feature.

[0036] like Figure 1 As shown in FIG. 1 , a forged information detection method based on dynamic knowledge distillation includes:

[0037] S1, small language model SLM evidence retrieval, through knowledge distillation KD, the reasoning ability of the LLM large language model is transferred to the first small language model SLM1, the first small language model SLM1 generates a small language model search phrase based on the news text, and retrieves the small language model search phrase through the search engine to obtain the small language model evidence text;

[0038] Specifically, the training goal of the first small language model SLM1 in S1 is to minimize the difference between the search phrases generated by the large language model LLM and the search phrases generated by the first small language model SLM1, and accurately generate phrases that can search for evidence. The loss formula for calculation is:

[0039]

[0040] f1 represents the search phrase generated by the small language model SLM, x iRepresents news text, q i represents the search phrase generated by the large language model LLM, and l represents the cross entropy loss.

[0041] Then, the search phrase generated by the small language model SLM is used to retrieve external evidence using the being search engine. Finally, the newspaper library in Python is used to obtain the small language model evidence text.

[0042] S2. Classify positive and negative evidence, construct positive and negative evidence sets, and design a discriminator trained using contrastive learning loss. Use the positive and negative evidence sets as the training set data for the discriminator. Use the discriminator to distinguish positive and negative evidence from the inference evidence generated by the news text and the large language model based on the news text, and assign appropriate weights.

[0043] Specifically, S2 includes

[0044] S21. Construct a set of positive and negative evidence. In the process of knowledge distillation KD, since the reasoning evidence of correct prediction is more useful for the final prediction of the small language model SLM than the reasoning evidence of wrong prediction, the evidence is divided into positive and negative evidence according to the consistency between the output of the large language model LLM and the actual label. If the predicted label of the large language model LLM is consistent with the true label, it is classified as a positive evidence. If it is inconsistent with the true label, it is classified as a negative evidence. The set of negative evidence is set to The set of positive evidence is

[0045] S22, contrast learning strategy, using S generated in S21 neg , S pos The discriminator is trained to distinguish between positive evidence and negative evidence and generate a score representing the quality of the evidence. The input of the discriminator is the news text and the reasoning evidence, and then the encoder structure is used to measure the score. The calculation formula is:

[0046]

[0047] Among them, X represents the news text, and Represents the scores of positive and negative grounds respectively.

[0048] S3, multi-task learning, including basis tasks and label tasks. The basis task enables the second small language model to learn to generate reasoning basis knowledge from the positive basis and negative basis generated by the large language model discriminated by the discriminator and the small language model evidence text. The label task realizes the alignment of the classification label generated by the second small language model and the real label.

[0049] Specifically, S3 includes

[0050] S31. Based on the task, the learning of the basis uses a many-to-one contrastive learning distillation loss to distill the positive and negative basis generated by the large language model LLM into the second small language model SLM2 at the same time. The calculation formula is:

[0051]

[0052] Among them, x i , Respectively represent the input text, positive evidence and negative evidence of the i-th large language model LLM, N represents the number of samples, β is a hyperparameter greater than 0, l(.) represents the cross entropy loss, and f(.) represents the generation of evidence for a given input. At the same time, distillation guided by evidence quality is introduced to optimize the training process. and The calculation formula is as follows:

[0053]

[0054] in, Represent the positive basis score and negative basis score respectively,

[0055] S32, labeling task, calculates the cross entropy loss between the label predicted by the second small language model SLM2 and the real label. The calculation formula is:

[0056] L cls = l(f(x i ), y i )

[0057] The total training loss is L cls , L distill and L prob The sum is calculated as:

[0058] L total =L cls +L distill +L prob .

[0059] In this case, two small language models SLM are trained. The first small language model SLM1 obtains the ability to generate search phrases from the large language model LLM, allowing the first small language model SLM1 to generate search phrases similar to the large language model LLM, and then use the search engine to retrieve evidence to determine whether the news is true or false based on the search phrases. The second small language model SLM2 learns the reasoning basis and labels from the large language model LLM.

[0060] At the same time, since different inference bases have different effects on classification results, not all bases play a positive role in reasoning. Previously, the forged information detection method combined with knowledge distillation KD regarded all bases as having the same effect, and did not dynamically assign weights according to the positive or negative impact of the bases during the knowledge distillation process. Therefore, a discriminator is set in this case to determine whether the basis generated by the large language model LLM is a positive basis or a negative basis after training, and to assign appropriate weights to the reasoning basis. Then, using contrastive learning loss, the positive and negative bases are distilled into the student model at the same time.

[0061] This case also involves using the basis generated by the large language model LLM, and using the knowledge distillation KD method of multi-task learning to guide the second small language model SLM2. This method involves simultaneously training the label prediction and basis generation of the second small language model SLM2, effectively utilizing the common advantages of the two.

[0062] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or directly or indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A method for detecting forged information based on dynamic knowledge distillation, characterized in that: Included S1. Small language model evidence retrieval: The reasoning ability of the large language model is transferred to the first small language model through knowledge distillation. The first small language model generates a small language model search phrase based on the news text, and retrieves the small language model search phrase through the search engine to obtain the small language model evidence text. S2. Classify positive and negative evidence, construct positive and negative evidence sets, and design a discriminator trained using contrastive learning loss. Use the positive and negative evidence sets as the training set data for the discriminator. Use the discriminator to distinguish positive and negative evidence from the inference evidence generated by the news text and the large language model based on the news text, and assign weights. S3. Multi-task learning, including basis tasks and label tasks. The basis task enables the second small language model to learn to generate reasoning basis knowledge from the positive and negative basis generated by the large language model identified by the discriminator and the small language model evidence text. The label task realizes the alignment of the classification label generated by the second small language model and the true label.

2. The method for detecting forged information based on dynamic knowledge distillation according to claim 1, characterized in that: In S1, the difference between the search phrase generated by the first small language model and the search phrase generated by the large language model is measured by the loss formula. The loss formula is calculated as: f1 represents the search phrase generated by the first small language model, x i Represents news text, q i represents the search phrase generated by the large language model, and l represents the cross entropy loss.

3. The method for detecting forged information based on dynamic knowledge distillation according to claim 1, characterized in that: The first small language model in S1 is retrieved from the newspaper library in Python through a search engine.

4. A forged information detection method based on dynamic knowledge distillation according to claim 1 or 3, characterized in that: The search engine in S1 is the being search engine.

5. The method for detecting forged information based on dynamic knowledge distillation according to claim 1, characterized in that: S2 includes S21. Construct a set of positive and negative evidence. The large language model outputs reasoning evidence and prediction labels based on the news text, and divides them into positive evidence and negative evidence according to the consistency with the actual label, and constructs a set of positive evidence and negative evidence. Set the set of negative evidence to The set of positive evidence is S22, contrast learning strategy, using S generated in S1 neg , S pos Train the discriminator. The input of the discriminator is the news text and the reasoning basis. The encoder structure is used to measure the score. The calculation formula is: X represents news text, and Represents the scores of positive and negative grounds respectively.

6. The method for detecting forged information based on dynamic knowledge distillation according to claim 1, characterized in that: S3 includes S31. Based on the task, based on the learning, use the many-to-one contrastive learning distillation loss to distill the positive and negative evidence generated by the large language model discriminated by the discriminator into the second small language model at the same time. The calculation formula is: x i , Respectively represent the input text, positive evidence and negative evidence of the i-th large language model, N represents the number of samples, β is a hyperparameter greater than 0, l(.) represents the cross entropy loss, and f(.) represents the generation of evidence for a given input; Optimize the training process with quality-guided distillation, and The calculation formula is as follows: Represent the positive basis score and negative basis score respectively, S32, labeling task, calculate the cross entropy loss between the label predicted by the first small language model and the true label. The calculation formula is: L cls =l(f(x i ),y i ) The total training loss is L cls , L distill and L prob The sum is calculated as: L total =L cls +L distill +L prob .

Images