Large language model training method and device, electronic equipment and storage medium

Abstract

The invention provides a large language model training method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, deep neural networks and large language models. The method comprises the steps of obtaining a sample text containing logical reasoning content of a target reasoning mode; obtaining a credibility evaluation result of the sample text; associating the sample text with the credibility evaluation result of the sample text to obtain a training sample; and obtaining a training sample set based on the plurality of training samples, and training the initial model based on the training sample set to obtain a large language model. Therefore, the training of the large language model can be realized based on the sample text containing the logical reasoning content of the target reasoning mode and the credibility evaluation result of the sample text, so that the large language model can learn the target reasoning mode in the training process, the text credibility evaluation effect of the large language model is improved, and the reliability of the large language model is improved. The method is especially suitable for application scenes of news credibility evaluation.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of deep learning, deep neural network, and large language model technology, and in particular to a large language model training method, news credibility evaluation method, device, electronic device, storage medium, and computer program product. Background Art

[0002] At present, with the continuous development of artificial intelligence technology, large language models have advantages such as good generalization and have been widely used in information extraction, text credibility assessment, machine translation and other fields. However, the training method of large language models in related technologies has the problem that the text credibility assessment effect of the trained large language model is poor. Summary of the invention

[0003] The present disclosure proposes a large language model training method, a news credibility evaluation method, a device, an electronic device, a storage medium and a computer program product.

[0004] According to a first aspect of the present disclosure, a method for training a large language model is proposed, comprising: obtaining a sample text containing logical reasoning content of a target reasoning method; obtaining a credibility evaluation result of the sample text; associating the sample text with the credibility evaluation result of the sample text to obtain a training sample; obtaining a training sample set based on a plurality of the training samples, and training an initial model based on the training sample set to obtain a large language model.

[0005] According to a second aspect of the present disclosure, a method for evaluating the credibility of news is proposed, comprising: obtaining target news; inputting the target news into a large language model, performing credibility evaluation on the target news through the large language model, and obtaining a credibility evaluation result of the target news, wherein the large language model is obtained using the large language model training method proposed in the first aspect above.

[0006] According to a third aspect of the present disclosure, a training device for a large language model is proposed, comprising: a first acquisition module, used to acquire a sample text containing logical reasoning content of a target reasoning method; a second acquisition module, used to acquire a credibility evaluation result of the sample text; an association module, used to associate the sample text with the credibility evaluation result of the sample text to obtain a training sample; and a training module, used to obtain a training sample set based on a plurality of the training samples, and to train an initial model based on the training sample set to obtain a large language model.

[0007] According to a fourth aspect of the present disclosure, a device for evaluating the credibility of news is proposed, comprising: an acquisition module for acquiring target news; an evaluation module for inputting the target news into a large language model, performing credibility evaluation on the target news through the large language model, and obtaining a credibility evaluation result of the target news, wherein the large language model is obtained using the large language model training method proposed in the first aspect above.

[0008] According to a fifth aspect of the present disclosure, an electronic device is proposed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large language model training method proposed in the first aspect above, or execute the news credibility evaluation method proposed in the second aspect above.

[0009] According to the sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is proposed, wherein the computer instructions are used to enable the computer to execute the large language model training method proposed in the first aspect, or to execute the news credibility evaluation method proposed in the second aspect.

[0010] According to the seventh aspect of the present disclosure, a computer program product is proposed, including a computer program, which, when executed by a processor, implements the large language model training method proposed in the first aspect, or implements the news credibility evaluation method proposed in the second aspect.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 A flowchart of a method for training a large language model according to an embodiment of the present disclosure;

[0014] Figure 2 A flowchart of a method for training a large language model according to another embodiment of the present disclosure;

[0015] Figure 3 A flowchart of a method for training a large language model according to another embodiment of the present disclosure;

[0016] Figure 4A flowchart of a method for training a large language model according to another embodiment of the present disclosure;

[0017] Figure 5 A flowchart of a method for training a large language model according to another embodiment of the present disclosure;

[0018] Figure 6 A flowchart of a method for evaluating news credibility according to an embodiment of the present disclosure;

[0019] Figure 7 A schematic diagram of the structure of a large language model training device according to an embodiment of the present disclosure;

[0020] Figure 8 This is a structural diagram of a device for evaluating news credibility according to an embodiment of the present disclosure;

[0021] Fig. 9 A schematic block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] AI (Artificial Intelligence) is a technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. At present, AI technology has the advantages of high automation, high precision and low cost, and has been widely used.

[0024] DL (Deep Learning) is a new research direction in the field of ML (Machine Learning). It is a science that learns the inherent laws and representation levels of sample data, enabling machines to have analytical learning capabilities like humans and to recognize data such as text, images, and sounds. It is widely used in speech and image recognition.

[0025] DNN (Deep Neural Network) is a technology in the field of ML (Machine Learning). It is a neural network with at least one hidden layer. In order to overcome the shortcomings of traditional neural networks such as easy overfitting and slow training speed, deep neural networks generally adopt a layer-by-layer pre-training training mechanism instead of the back-propagation training mechanism of traditional neural networks.

[0026] LLM (Large Language Model) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important path to artificial intelligence.

[0027] Figure 1 FIG. 1 is a flow chart of a method for training a large language model according to an embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0028] S101, obtaining a sample text containing logical reasoning content of a target reasoning method.

[0029] It should be noted that the execution subject of the training method of the large language model of the embodiment of the present disclosure may be a hardware device with data information processing capabilities and / or the necessary software required to drive the hardware device to work. Optionally, the execution subject may include a workstation, a server, a computer, a user terminal and other intelligent devices. Among them, the user terminal includes but is not limited to a mobile phone, a computer, an intelligent voice interaction device, an intelligent home appliance, a vehicle-mounted terminal, etc.

[0030] It should be noted that the sample text refers to the text containing the logical reasoning content of the target reasoning method. There are no excessive restrictions on the sample text. For example, the sample text may include text composed of at least one language such as Chinese and English. For example, the sample text may include news, papers, comments, literary works, etc.

[0031] It should be noted that there are no excessive restrictions on the target reasoning methods, for example, they may include hypothetical reasoning, analogical reasoning, inductive reasoning, critical reasoning, etc. Among them, critical reasoning may include logical reasoning, causal relationship analysis, evidence evaluation, etc. The number of target reasoning methods is at least one.

[0032] It should be noted that there are not too many restrictions on the content of logical reasoning. For example, it can include arguments, evidence, reasoning, premises, conclusions, etc.

[0033] In one embodiment, obtaining a sample text containing logical reasoning content of a target reasoning method includes identifying candidate reasoning methods of a set text in a set text library, and if the candidate reasoning methods include at least one target reasoning method, determining the set text as the sample text. It should be noted that the identification of candidate reasoning methods of a set text in a set text library can be referred to the following embodiment, which will not be repeated here.

[0034] It should be noted that the setting text library includes a plurality of setting texts, which can be preset.

[0035] For example, if the set text library includes set texts A, B, and C, the target reasoning method includes critical reasoning, the candidate reasoning methods for identifying the set text A include hypothetical reasoning and critical reasoning, the candidate reasoning methods for identifying the set text B include deductive reasoning, and the candidate reasoning methods for identifying the set text C include analogical reasoning, then the candidate reasoning methods for the set text A include the target reasoning method, and the candidate reasoning methods for the set texts B and C do not include the target reasoning method, and the set text A is determined as the sample text.

[0036] In one embodiment, obtaining a sample text containing logical reasoning content of a target reasoning method includes obtaining a label of a setting text in a setting text library. If the label of the setting text indicates that the setting text contains logical reasoning content of the target reasoning method, the setting text is determined as a sample text.

[0037] It should be noted that there are not too many restrictions on the label of the setting text. For example, if the label of the setting text is 1, it indicates that the setting text contains the logical reasoning content of the target reasoning method. If the label of the setting text is 0, it indicates that the setting text does not contain the logical reasoning content of the target reasoning method.

[0038] In some examples, the label of the setting text may be manually annotated, or the setting text may be input into a first annotation model, and the first annotation model may output the label of the setting text.

[0039] S102, obtaining a credibility evaluation result of the sample text.

[0040] It should be noted that there are no excessive restrictions on the credibility assessment results of the sample text. For example, it may include the credibility of the sample text and the reasons for the credibility assessment.

[0041] In one implementation, the credibility of the sample text ranges from 0 to 100.

[0042] In one implementation, the evaluation reasons may include evaluation reasons for the true part of the sample text, and may also include evaluation reasons for the false part of the sample text.

[0043] In one implementation, the credibility evaluation result of the sample text may be annotated manually, or the sample text may be input into a second annotation model, and the second annotation model outputs the credibility evaluation result of the sample text.

[0044] In one embodiment, obtaining the credibility evaluation result of the sample text includes querying target knowledge associated with the sample text from a set knowledge base based on the sample text, and performing credibility evaluation on the sample text based on the sample text and the target knowledge to obtain the credibility evaluation result of the sample text.

[0045] It should be noted that the setting knowledge base may include setting knowledge in multiple knowledge fields, wherein the knowledge fields include science, literature, philosophy, art, etc. The target knowledge refers to the setting knowledge associated with the sample text in the setting knowledge base.

[0046] In some examples, based on the sample text, querying the target knowledge associated with the sample text from the set knowledge base includes determining the knowledge field to which the logical reasoning content in the sample text belongs, and querying the target knowledge associated with the sample text from the set knowledge base based on the knowledge field corresponding to the sample text. It should be noted that determining the knowledge field to which the logical reasoning content in the sample text belongs can be referred to the following embodiment, which will not be repeated here.

[0047] For example, if the knowledge field corresponding to the sample text A is medicine, the set knowledge under medicine can be queried from the set knowledge base, and the queried set knowledge under medicine can be used as the target knowledge associated with the sample text A.

[0048] For example, if the knowledge domain corresponding to the sample text B is geography, the set knowledge under geography can be queried from the set knowledge base, and the queried set knowledge under geography can be used as the target knowledge associated with the sample text B.

[0049] In some examples, based on the sample text and the target knowledge, the sample text is credibility evaluated to obtain a credibility evaluation result of the sample text, including identifying whether there is conflicting content between the sample text and the target knowledge, and the sample text is credibility evaluated based on the identification result to obtain a credibility evaluation result of the sample text.

[0050] In some examples, the recognition result includes the number of conflicting contents, the difference level of the conflicting contents, etc. Among them, the higher the difference level, the greater the difference between the sample content and the target knowledge.

[0051] In some examples, the credibility of the sample text is negatively correlated with the amount of conflicting content, and the credibility of the sample text is negatively correlated with the difference level of the conflicting content.

[0052] In some examples, a credibility assessment rationale for the sample text may be obtained based on the conflicting content.

[0053] For example, if the value of data x in sample text A is 100, the value of data x in sample text B is 110, and the value of data x in target knowledge is 80, that is, there is conflicting content between sample text A and target knowledge, and the conflicting content is the value of data x. The difference level of the conflicting content of sample text A is the first level. There is conflicting content between sample text B and target knowledge, and the conflicting content is the value of data x. The difference level of the conflicting content of sample text B is the second level. The first level is lower than the second level.

[0054] The credibility of sample text A is 80, and the credibility assessment reason is "the value of data x in sample text A is 100, the value of data x in target knowledge is 80, and the difference between the two is 20".

[0055] The credibility of sample text B is 60, and the credibility assessment reason is "the value of data x in sample text B is 110, the value of data x in target knowledge is 80, and the difference between the two is 30".

[0056] S103, associating the sample text with the credibility evaluation result of the sample text to obtain a training sample.

[0057] It should be noted that the association between the sample text and the credibility evaluation result of the sample text can be achieved by using any data association method in the relevant technology, and no further limitation is made here.

[0058] In one implementation, the sample text and the credibility evaluation result of the sample text are associated to obtain the training sample, including establishing an association relationship such as a mapping relationship, a corresponding relationship, etc. between the sample text and the credibility evaluation result of the sample text.

[0059] For example, a mapping relationship between sample text A and the credibility evaluation result of sample text A may be established to obtain a training sample.

[0060] S104, obtaining a training sample set based on the multiple training samples, and training the initial model based on the training sample set to obtain a large language model.

[0061] It can be understood that the training sample set includes multiple training samples.

[0062] In one implementation, obtaining a training sample set based on a plurality of training samples includes randomly selecting a set number of training samples to obtain the training sample set.

[0063] It should be noted that the initial model is trained based on the training sample set to obtain a large language model, which can be implemented by using any large language model training method in the relevant technology, and no excessive restrictions are made here.

[0064] In one embodiment, an initial model is trained based on a training sample set to obtain a large language model, including inputting sample text into the initial model, having the initial model output a credibility prediction evaluation result of the sample text, obtaining a loss function of the initial model based on the credibility evaluation result of the sample text and the credibility prediction evaluation result of the sample text, and training the initial model based on the loss function of the initial model.

[0065] The training method of the large language model proposed in the present disclosure obtains a sample text containing logical reasoning content of a target reasoning method, obtains a credibility evaluation result of the sample text, associates the sample text and the credibility evaluation result of the sample text to obtain a training sample, obtains a training sample set based on multiple training samples, and trains the initial model based on the training sample set to obtain a large language model. Thus, based on the sample text containing logical reasoning content of the target reasoning method and the credibility evaluation result of the sample text, a training sample can be obtained to realize the training of the large language model, so that during the training process, the large language model can learn the target reasoning method, so that the credibility evaluation of the text can be performed according to the target reasoning method later, which helps to improve the text credibility evaluation effect of the large language model, and is particularly suitable for the application scenario of news credibility evaluation.

[0066] In the above embodiment, in step S101, the sample text containing the logical reasoning content of the target reasoning method can be obtained. Figure 2 Further understanding, Figure 2 FIG. 1 is a flow chart of a method for training a large language model according to another embodiment of the present disclosure. Figure 2 As shown, the method includes:

[0067] S201, obtaining target keywords for logical reasoning corresponding to the target reasoning method.

[0068] S202: Determine whether the setting text in the setting text library contains the target keyword.

[0069] S203: If the set text contains the target keyword, the set text is determined as a sample text.

[0070] It is understandable that the target keyword corresponding to the target reasoning method can be preset. One target reasoning method can correspond to at least one target keyword, and different target reasoning methods can correspond to different target keywords or the same target keyword, which is not limited here.

[0071] For example, if the set text library includes set texts A, B, and C, and the target reasoning method includes causal relationship analysis, the target keywords corresponding to the causal relationship analysis may include "because", "so", "reason", "result", "sufficient", "necessary", "condition", etc.

[0072] If the set text A includes the target keywords "because" and "so", the set text A may be determined as the sample text.

[0073] If the setting text B includes the target keywords "cause" and "result", the setting text B may be determined as the sample text.

[0074] If the set text C does not include any target keyword corresponding to the causal relationship analysis, the set text C is not determined as a sample text.

[0075] S204, obtaining the credibility evaluation result of the sample text.

[0076] S205, associating the sample text with the credibility evaluation result of the sample text to obtain a training sample.

[0077] S206, obtaining a training sample set based on the multiple training samples, and training the initial model based on the training sample set to obtain a large language model.

[0078] For the relevant contents of steps S204-S206, please refer to the above embodiment and will not be repeated here.

[0079] The training method of the large language model proposed in the present disclosure obtains the target keyword for logical reasoning corresponding to the target reasoning mode, determines whether the set text in the set text library contains the target keyword, and if the set text contains the target keyword, determines the set text as the sample text. Therefore, when the set text contains the target keyword, the set text can be determined as the sample text to achieve the acquisition of the sample text.

[0080] In the above embodiment, in step S101, the sample text containing the logical reasoning content of the target reasoning method can be obtained. Figure 3 Further understanding, Figure 3 FIG. 1 is a flow chart of a method for training a large language model according to another embodiment of the present disclosure. Figure 3 As shown, the method includes:

[0081] S301, identifying candidate reasoning methods of a setting text in a setting text library.

[0082] S302: If the candidate reasoning method is consistent with the target reasoning method, the set text is determined as the sample text.

[0083] In one embodiment, identifying candidate reasoning methods of a setting text in a setting text library includes performing semantic analysis on the setting text to obtain candidate reasoning methods. It should be noted that semantic analysis can be implemented using any semantic analysis method in the relevant technology, which is not limited here.

[0084] For example, if the set text library includes set texts A, B, and C, the target reasoning method includes critical reasoning, the candidate reasoning method for identifying the set text A includes hypothetical reasoning, the candidate reasoning method for identifying the set text B includes critical reasoning, and the candidate reasoning method for identifying the set text C includes analogical reasoning, then the candidate reasoning method for the set text B is consistent with the target reasoning method, and the candidate reasoning methods for the set texts A and C are inconsistent with the target reasoning method, and the set text B is determined as the sample text.

[0085] S303, obtaining the credibility evaluation result of the sample text.

[0086] S304: Associating the sample text with the credibility evaluation result of the sample text to obtain a training sample.

[0087] S305, obtaining a training sample set based on the multiple training samples, and training the initial model based on the training sample set to obtain a large language model.

[0088] For the relevant contents of steps S303 - S305 , please refer to the above embodiment and will not be described again here.

[0089] The training method of the large language model proposed in the present disclosure identifies the candidate reasoning methods of the set text in the set text library, and if the candidate reasoning method is consistent with the target reasoning method, the set text is determined as the sample text. Therefore, when the candidate reasoning method is consistent with the target reasoning method, the set text can be determined as the sample text to achieve the acquisition of the sample text.

[0090] In the above embodiment, in step S104, the training sample set is obtained based on multiple training samples. Figure 4 Further understanding, Figure 4 FIG. 1 is a flow chart of a method for training a large language model according to another embodiment of the present disclosure. Figure 4 As shown, the method includes:

[0091] S401, obtaining a sample text containing logical reasoning content of a target reasoning method.

[0092] S402, obtaining the credibility evaluation result of the sample text.

[0093] S403: Associating the sample text with the credibility evaluation result of the sample text to obtain a training sample.

[0094] For the relevant contents of steps S401 - S403 , please refer to the above embodiment and will not be described again here.

[0095] S404, selecting an i-th first subset from a plurality of training samples under the i-th target reasoning mode.

[0096] S405: Obtain a union of N first subsets as a training sample set.

[0097] Wherein, N is a positive integer, and i is a positive integer not greater than N. The number of target reasoning methods is N. There is no excessive restriction on N.

[0098] It should be noted that the target reasoning method corresponds one-to-one to the first subset, the i-th first subset includes at least one training sample under the i-th target reasoning method, and the reasoning method of the logical reasoning content of the sample text corresponding to the training sample in the i-th first subset is the i-th target reasoning method.

[0099] In one embodiment, the i-th first subset is selected from multiple training samples under the i-th target reasoning mode, including randomly selecting a set number of training samples from multiple training samples under the i-th target reasoning mode to obtain the i-th first subset.

[0100] For example, the target reasoning methods include logical reasoning and causal relationship analysis. The first subset A can be screened out from multiple training samples under logical reasoning, and the first subset B can be screened out from multiple training samples under causal relationship analysis. The union of the first subset A and the first subset B is obtained as the training sample set.

[0101] S406: Train the initial model based on the training sample set to obtain a large language model.

[0102] For the relevant contents of step S406, please refer to the above embodiment, which will not be repeated here.

[0103] The training method of the large language model proposed in the present disclosure selects the i-th first subset from multiple training samples under the i-th target reasoning mode, obtains the union of N first subsets, and uses it as a training sample set. Thus, the training samples under multiple target reasoning modes can be comprehensively considered to obtain a training sample set to realize the training of the large language model, so that the large language model can learn multiple target reasoning modes during the training process, which helps to improve the text credibility evaluation effect and generalization of the large language model.

[0104] In the above embodiment, in step S104, a training sample set is obtained based on multiple training samples. Figure 5 Further understanding, Figure 5 FIG. 1 is a flow chart of a method for training a large language model according to another embodiment of the present disclosure. Figure 5 As shown, the method includes:

[0105] S501, obtaining a sample text containing logical reasoning content of a target reasoning method.

[0106] S502, obtaining a credibility evaluation result of the sample text.

[0107] S503: Associating the sample text with the credibility evaluation result of the sample text to obtain a training sample.

[0108] For the relevant contents of steps S501-S503, please refer to the above embodiment and will not be repeated here.

[0109] S504, determining the knowledge domain to which the logical reasoning content in the sample text belongs, as the knowledge domain to which the training sample corresponding to the sample text belongs.

[0110] In one implementation, determining the knowledge domain to which the logical reasoning content in the sample text belongs includes performing semantic analysis on the logical reasoning content in the sample text to obtain the knowledge domain to which the logical reasoning content in the sample text belongs.

[0111] For example, if the logical reasoning content in sample text A includes "heatstroke" and "body temperature", it can be determined that the knowledge field to which the logical reasoning content in the sample text belongs is medicine, and medicine is taken as the knowledge field to which the training sample corresponding to sample text A belongs.

[0112] For example, if the logical reasoning content in sample text B includes "typhoon" and "rainstorm", it can be determined that the knowledge field to which the logical reasoning content in the sample text belongs is meteorology, and meteorology is used as the knowledge field to which the training sample corresponding to sample text B belongs.

[0113] S505 , based on the knowledge domain to which the training samples belong, select a training sample set from the multiple training samples.

[0114] In one implementation, based on the knowledge domain to which the training samples belong, a training sample set is screened out from a plurality of training samples, including screening out a set number of training samples in the knowledge domain from the plurality of training samples to obtain the training sample set.

[0115] In one embodiment, based on the knowledge domain to which the training samples belong, a training sample set is screened out from multiple training samples, including obtaining M target knowledge domains, screening out the jth second subset from multiple training samples under the jth target knowledge domain, and obtaining the union of the M second subsets as the training sample set. Thus, the training samples under multiple target knowledge domains can be comprehensively considered to obtain a training sample set to implement the training of a large language model, so that the large language model is applicable to multiple target knowledge domains, which helps to improve the text credibility evaluation effect and generalization of the large language model.

[0116] Wherein, M is a positive integer, and j is a positive integer not greater than M. The number of target knowledge domains is M. No excessive restrictions are made on M. No excessive restrictions are made on the target knowledge domains.

[0117] It should be noted that the target knowledge domain corresponds to the second subset one by one, the jth second subset includes at least one training sample under the jth target knowledge domain, and the knowledge domain to which the training sample in the jth second subset belongs is the jth target knowledge domain.

[0118] In one implementation, selecting the jth second subset from a plurality of training samples under the jth target knowledge domain includes randomly selecting a set number of training samples from a plurality of training samples under the jth target knowledge domain to obtain the jth second subset.

[0119] For example, the target knowledge domain includes medicine and meteorology. The second subset A can be screened out from multiple training samples under medicine, and the second subset B can be screened out from multiple training samples under meteorology. The union of the second subset A and the second subset B is obtained as the training sample set.

[0120] S506: Train the initial model based on the training sample set to obtain a large language model.

[0121] For the relevant contents of step S506, please refer to the above embodiment, which will not be described again here.

[0122] The training method of the large language model proposed in the present disclosure determines the knowledge field to which the logical reasoning content in the sample text belongs as the knowledge field to which the training sample corresponding to the sample text belongs, and based on the knowledge field to which the training sample belongs, screens out a training sample set from multiple training samples. Thus, the training sample set can be screened out from multiple training samples taking into account the knowledge field to which the training sample belongs, thereby improving the flexibility of screening the training sample set.

[0123] Figure 6 FIG. 1 is a flow chart of a method for evaluating news credibility according to an embodiment of the present disclosure. Figure 6 As shown, the method includes:

[0124] S601, obtaining target news.

[0125] It should be noted that the execution subject of the news credibility evaluation method of the embodiment of the present disclosure may be a hardware device with data information processing capabilities and / or the necessary software required to drive the hardware device to work. Optionally, the execution subject may include a workstation, a server, a computer, a user terminal and other intelligent devices. Among them, the user terminal includes but is not limited to a mobile phone, a computer, an intelligent voice interaction device, an intelligent home appliance, a vehicle-mounted terminal, etc.

[0126] It should be noted that there are no excessive restrictions on target news, for example, target news includes but is not limited to newspapers, news apps (Applications), and news web pages.

[0127] In one implementation, taking the execution subject as a user terminal as an example, the user terminal can obtain the target news from its own storage space, and / or crawl the target news from a web page or an APP (Application).

[0128] In one implementation, taking the execution subject as a server as an example, obtaining target news may include receiving target news sent by a client. It is understandable that the client may obtain target news based on the operation information of the user who operates the client (for example, the target news uploaded by the user, the user clicking the icon of the target news on the display interface of the client, etc.), and send the target news to the server, and correspondingly, the server may receive the target news sent by the client.

[0129] S602, inputting the target news into the large language model, performing credibility evaluation on the target news through the large language model, and obtaining a credibility evaluation result of the target news, wherein the large language model is obtained by using a large language model training method.

[0130] It should be noted that large language models can be used Figures 1 to 5 The training method of the large language model shown is obtained and will not be repeated here.

[0131] In one embodiment, the credibility of target news is evaluated through a large language model, and a credibility evaluation result of the target news is obtained, including extracting information from the target news through the large language model to obtain key information, querying reference knowledge of the key information from a set knowledge base based on the semantic representation of the key information through the large language model, and evaluating the credibility of the target news based on the key information and the reference knowledge through the large language model, and obtaining a credibility evaluation result. Thus, the reliability of news can be evaluated through the large language model and the set knowledge base, thereby improving the accuracy of the evaluation of news reliability.

[0132] It should be noted that there are not too many restrictions on the categories of key information. For example, it can include logical reasoning content, time, place, events, people, etc.

[0133] It should be noted that based on the semantic representation of key information, the reference knowledge of key information can be queried from the set knowledge base, which can be achieved by using any data retrieval method in the relevant technology, and no excessive restrictions are made here.

[0134] In one embodiment, based on the semantic representation of the key information, reference knowledge of the key information is queried from a setting knowledge base, including obtaining the semantic representation of the setting knowledge in the setting knowledge base, obtaining the similarity between the semantic representation of the key information and the semantic representation of the setting knowledge, and determining the setting knowledge with a similarity greater than a set threshold as reference knowledge.

[0135] It should be noted that, based on the key information and reference knowledge, the credibility of the target news is evaluated and the credibility evaluation result is obtained. The relevant content of the credibility evaluation result of the sample text obtained by performing credibility evaluation on the sample text based on the sample text and the target knowledge in the above embodiment can be referred to, which will not be repeated here.

[0136] In one embodiment, the credibility evaluation result includes the credibility of the target news and the reason for the credibility evaluation. Therefore, the large language model in this solution can not only obtain the credibility of the target news, but also obtain the reason for the credibility evaluation, thereby improving the diversity and comprehensiveness of the news credibility evaluation results.

[0137] In one embodiment, taking the execution subject as a server as an example, after obtaining the credibility assessment result of the target news, it also includes sending the credibility assessment result of the target news to the client. In response, the client can receive the credibility assessment result of the target news sent by the server, and control the display of the credibility assessment result of the target news on the display interface of the client.

[0138] The news credibility evaluation method proposed in the present disclosure obtains target news, inputs the target news into a large language model, performs credibility evaluation on the target news through the large language model, and obtains a credibility evaluation result of the target news, wherein the large language model is obtained by using a large language model training method, and the news credibility evaluation effect of the large language model is better, which helps to improve the evaluation effect of news credibility.

[0139] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0140] According to an embodiment of the present disclosure, the present disclosure also provides a large language model training device for implementing the above-mentioned large language model training method.

[0141] Figure 7 4 is a block diagram of a large language model training device according to an embodiment of the present disclosure.

[0142] like Figure 7 As shown, the large language model training device 700 includes: a first acquisition module 701, a second acquisition module 702, an association module 703 and a training module 704.

[0143] The first acquisition module 701 is used to acquire a sample text containing logical reasoning content of a target reasoning method;

[0144] The second acquisition module 702 is used to obtain the credibility evaluation result of the sample text;

[0145] The association module 703 is used to associate the sample text with the credibility evaluation result of the sample text to obtain a training sample;

[0146] The training module 704 is used to obtain a training sample set based on the multiple training samples, and train the initial model based on the training sample set to obtain a large language model.

[0147] In one embodiment of the present disclosure, the first acquisition module 701 is also used to: obtain the target keyword for logical reasoning corresponding to the target reasoning method; determine whether the set text in the set text library contains the target keyword; if the set text contains the target keyword, determine the set text as the sample text.

[0148] In one embodiment of the present disclosure, the first acquisition module 701 is further used to: identify candidate reasoning methods of a setting text in a setting text library; if the candidate reasoning method is consistent with the target reasoning method, determine the setting text as the sample text.

[0149] In one embodiment of the present disclosure, the number of target reasoning methods is N, and the training module 704 is also used to: filter out the i-th first subset from multiple training samples under the i-th target reasoning method; obtain the union of N first subsets as the training sample set; wherein N is a positive integer, and i is a positive integer not greater than N.

[0150] In one embodiment of the present disclosure, the training module 704 is also used to: determine the knowledge field to which the logical reasoning content in the sample text belongs, as the knowledge field to which the training sample corresponding to the sample text belongs; and filter out the training sample set from multiple training samples based on the knowledge field to which the training sample belongs.

[0151] In one embodiment of the present disclosure, the training module 704 is also used to: obtain M target knowledge domains; filter out the jth second subset from multiple training samples under the jth target knowledge domain; obtain the union of the M second subsets as the training sample set; wherein M is a positive integer and j is a positive integer not greater than M.

[0152] The training device of the large language model proposed in the present disclosure obtains a sample text containing logical reasoning content of a target reasoning method, obtains a credibility evaluation result of the sample text, associates the sample text and the credibility evaluation result of the sample text to obtain a training sample, obtains a training sample set based on multiple training samples, and trains the initial model based on the training sample set to obtain a large language model. Thus, based on the sample text containing logical reasoning content of the target reasoning method and the credibility evaluation result of the sample text, a training sample can be obtained to realize the training of the large language model, so that during the training process, the large language model can learn the target reasoning method, so that the credibility evaluation of the text can be performed according to the target reasoning method later, which helps to improve the text credibility evaluation effect of the large language model, and is particularly suitable for the application scenario of news credibility evaluation.

[0153] According to an embodiment of the present disclosure, the present disclosure also provides a news credibility evaluation device for implementing the above-mentioned news credibility evaluation method.

[0154] Figure 8 It is a block diagram of a device for evaluating news credibility according to an embodiment of the present disclosure.

[0155] like Figure 8 As shown, the news credibility evaluation device 800 includes: an acquisition module 801 and an evaluation module 802.

[0156] The acquisition module 801 is used to acquire target news;

[0157] The evaluation module 802 is used to input the target news into a large language model, perform credibility evaluation on the target news through the large language model, and obtain a credibility evaluation result of the target news, wherein the large language model is obtained using a large language model training method.

[0158] In one embodiment of the present disclosure, the evaluation module 802 is also used to: extract information from the target news through the large language model to obtain key information; query reference knowledge of the key information from a set knowledge base based on the semantic representation of the key information through the large language model; evaluate the credibility of the target news based on the key information and the reference knowledge through the large language model, and obtain the credibility evaluation result.

[0159] In one embodiment of the present disclosure, the credibility evaluation result includes the credibility of the target news and the evaluation reason of the credibility.

[0160] The news credibility evaluation device proposed in the present disclosure obtains target news, inputs the target news into a large language model, performs credibility evaluation on the target news through the large language model, and obtains a credibility evaluation result of the target news, wherein the large language model is obtained by using a large language model training method, and the news credibility evaluation effect of the large language model is better, which helps to improve the evaluation effect of news credibility.

[0161] According to an embodiment of the present disclosure, the present disclosure also proposes an electronic device, a readable storage medium, and a computer program product.

[0162] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0163] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0164] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 906, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0165] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a training method for a large language model and a method for evaluating news credibility. For example, in some embodiments, the training method for a large language model and the method for evaluating news credibility may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the training method for the large language model and the method for evaluating news credibility described above may be executed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute a large language model training method and a news credibility evaluation method in any other appropriate manner (eg, by means of firmware).

[0166] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0167] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be presented to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0168] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0169] To propose interactions with a user account, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user account; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user account can propose input to the computer. Other types of devices may also be used to propose interactions with the user account; for example, feedback proposed to the user account may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user account may be received in any form (including acoustic input, voice input, or tactile input).

[0170] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user account computer with a graphical user account interface or a web browser through which a user account can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0171] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0172] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the large language model training method and the news credibility evaluation method described in the above embodiments of the present disclosure are implemented.

[0173] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0174] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a large language model, comprising: Obtaining sample text containing logical reasoning content of the target reasoning method; Obtaining a credibility evaluation result of the sample text; Associating the sample text with the credibility evaluation result of the sample text to obtain a training sample; A training sample set is obtained based on the plurality of training samples, and an initial model is trained based on the training sample set to obtain a large language model.

2. The method according to claim 1, wherein: The step of obtaining a sample text containing logical reasoning content of a target reasoning method includes: Obtaining target keywords for logical reasoning corresponding to the target reasoning method; Determining whether a setting text in a setting text library contains the target keyword; If the set text contains the target keyword, the set text is determined as the sample text.

3. The method according to claim 1, wherein: The step of obtaining a sample text containing logical reasoning content of a target reasoning method includes: Identifying candidate reasoning methods for setting texts in a setting text library; If the candidate reasoning method is consistent with the target reasoning method, the set text is determined as the sample text.

4. The method according to claim 1, wherein: The number of the target reasoning methods is N, and the obtaining of a training sample set based on the plurality of training samples includes: Selecting the i-th first subset from multiple training samples under the i-th target reasoning mode; Obtain a union of N first subsets as the training sample set; wherein, N is a positive integer, and i is a positive integer not greater than N.

5. The method according to claim 1, wherein: The step of obtaining a training sample set based on the plurality of training samples comprises: Determine the knowledge domain to which the logical reasoning content in the sample text belongs, as the knowledge domain to which the training sample corresponding to the sample text belongs; The training sample set is selected from a plurality of the training samples based on the knowledge domain to which the training samples belong.

6. The method according to claim 5, wherein: The step of selecting the training sample set from a plurality of the training samples based on the knowledge domain to which the training samples belong includes: Acquire M target knowledge domains; Selecting the jth second subset from multiple training samples under the jth target knowledge domain; Obtain a union of M second subsets as the training sample set; wherein, M is a positive integer, and j is a positive integer not greater than M.

7. A method for evaluating news credibility, comprising: Get targeted news; The target news is input into a large language model, the credibility of the target news is evaluated by the large language model, and a credibility evaluation result of the target news is obtained, wherein the large language model is obtained by the training method of the large language model as described in any one of claims 1-6.

8. The method according to claim 7, wherein: The credibility evaluation of the target news is performed by the large language model, and a credibility evaluation result of the target news is obtained, including: Extracting information from the target news using the large language model to obtain key information; Using the large language model based on the semantic representation of the key information, querying reference knowledge of the key information from a set knowledge base; The credibility of the target news is evaluated based on the key information and the reference knowledge through the large language model, and the credibility evaluation result is obtained.

9. The method according to claim 7, wherein: The credibility evaluation result includes the credibility of the target news and the evaluation reason of the credibility.

10. A large language model training device, comprising: A first acquisition module is used to acquire a sample text containing logical reasoning content of a target reasoning method; A second acquisition module, used to obtain the credibility evaluation result of the sample text; An association module, used for associating the sample text with the credibility evaluation result of the sample text to obtain a training sample; The training module is used to obtain a training sample set based on the multiple training samples, and to train the initial model based on the training sample set to obtain a large language model.

11. The device according to claim 10, wherein: The first acquisition module is further used for: Obtaining target keywords for logical reasoning corresponding to the target reasoning method; Determining whether a setting text in a setting text library contains the target keyword; If the set text contains the target keyword, the set text is determined as the sample text.

12. The device according to claim 10, wherein: The first acquisition module is further used for: Identifying candidate reasoning methods for setting texts in a setting text library; If the candidate reasoning method is consistent with the target reasoning method, the set text is determined as the sample text.

13. The device according to claim 10, wherein: The number of target reasoning modes is N, and the training module is further used for: Selecting the i-th first subset from multiple training samples under the i-th target reasoning mode; Obtain a union of N first subsets as the training sample set; wherein, N is a positive integer, and i is a positive integer not greater than N.

14. The device according to claim 10, wherein: The training module is also used for: Determine the knowledge domain to which the logical reasoning content in the sample text belongs, as the knowledge domain to which the training sample corresponding to the sample text belongs; The training sample set is selected from a plurality of the training samples based on the knowledge domain to which the training samples belong.

15. The device according to claim 14, wherein: The training module is also used for: Acquire M target knowledge domains; Selecting the jth second subset from multiple training samples under the jth target knowledge domain; Obtain a union of M second subsets as the training sample set; wherein, M is a positive integer, and j is a positive integer not greater than M.

16. A device for evaluating news credibility, comprising: An acquisition module is used to acquire target news; An evaluation module is used to input the target news into a large language model, perform credibility evaluation on the target news through the large language model, and obtain a credibility evaluation result of the target news, wherein the large language model is obtained using the training method of the large language model as described in any one of claims 1-6.

17. The device according to claim 16, wherein: The evaluation module is also used to: Extracting information from the target news using the large language model to obtain key information; Using the large language model based on the semantic representation of the key information, querying reference knowledge of the key information from a set knowledge base; The credibility of the target news is evaluated based on the key information and the reference knowledge through the large language model, and the credibility evaluation result is obtained.

18. The device according to claim 16, wherein: The credibility evaluation result includes the credibility of the target news and the evaluation reason of the credibility.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images