Text evaluation method and device, electronic equipment, readable storage medium and vehicle

Abstract

The invention discloses a text evaluation method and device, electronic equipment, a readable storage medium and a vehicle. The text evaluation method comprises the steps that a to-be-evaluated text is obtained, and the text is generated by a language model; the text is subjected to information splitting processing based on the text processing model, a text feature data set of the text is obtained, the text feature data set comprises at least one text feature, each text feature comprises a subject feature and an object feature, and the object feature comprises event information associated with the subject feature; performing credibility scoring on each text feature based on a text scoring model to obtain a credibility score of each text feature; and calculating the credibility score of each text feature by adopting a preset statistical algorithm, and determining a credibility evaluation result of the text, the credibility evaluation result comprising a target score of the text. According to the embodiment of the invention, the authenticity of the text can be evaluated, and the definite index quantification of the text authenticity can be provided, so that the development of a large language model can be assisted.

Description

Technical Field

[0001] The present application belongs to the field of computer application technology, and in particular, relates to a text evaluation method, device, electronic device, readable storage medium and vehicle. Background Art

[0002] With the development of computer application technology, deep learning models are being used more and more widely. For example, a deep learning model trained with text data is also called a large language model. Large language models can be used to generate natural language text or understand the meaning of language text, thereby processing a variety of natural language tasks. However, although large language models have the ability to generate natural language text or understand language text, the text generated or understood by the large language model may contain hallucinations, that is, the large language model may output text that does not conform to the facts.

[0003] Based on this, in the process of developing a large language model, it is necessary to test the authenticity of the output of the large language model. However, the current evaluation of the text output by the large language model through summarization and other methods is not objective. In particular, for the same text, the evaluation results may output different scores, and the indicators of each score are also different. For example, the evaluation result generated by the scoring model may be a specific score, such as 70 points, or a percentage, such as 80%, or a text evaluation, such as good. Therefore, the credibility of the text evaluation results is currently low, and the quantification of indicators is also difficult. Therefore, it is currently difficult to generate evaluation results based on the scoring model to assist in the development of large language models. Summary of the invention

[0004] The embodiments of the present application provide a text evaluation method, device, electronic device, readable storage medium and vehicle, which can effectively improve the reliability and objectivity of text evaluation results, and quantify the output of evaluation results based on unified indicators, which is conducive to assisting the development of large language models.

[0005] In a first aspect, an embodiment of the present application provides a text evaluation method, the method comprising:

[0006] Obtaining text to be evaluated, where the text is generated by a language model;

[0007] Performing information splitting processing on the text based on the text processing model to obtain a text feature data set of the text, the text feature data set includes at least one text feature, each text feature includes a subject feature and an object feature, and the object feature includes event information associated with the subject feature;

[0008] Scoring the credibility of each text feature based on the text scoring model to obtain a credibility score for each text feature;

[0009] A preset statistical algorithm is used to calculate the credibility score of each text feature to determine the credibility evaluation result of the text, and the credibility evaluation result includes the target score of the text.

[0010] In some implementations of the first aspect, performing information splitting processing on a text based on a text processing model to obtain a text feature dataset of the text includes:

[0011] Acquire first reference information, the first reference information including a processing reference template of the text;

[0012] Adding a first preset mark to the text to obtain the marked text, wherein the first preset mark includes a first head mark and a first tail mark, and the first preset mark is used to instruct the text processing model to identify the text to be evaluated and a first output format of the text processing model;

[0013] Inputting the first reference information and the marked text into a text processing model, so that the text processing model parses the marked text according to a processing reference template to generate at least one subject feature and at least one object feature;

[0014] The subject features and the object features are matched one by one to generate at least one text feature in a first output format, thereby obtaining a text feature data set.

[0015] In some implementations of the first aspect, inputting the first reference information and the marked text into a text processing model includes:

[0016] Obtaining execution requirement information, the execution requirement information includes an execution role and an execution action, wherein the execution action includes information splitting processing of the text;

[0017] The execution requirement information, the first reference information and the marked text are concatenated to obtain query information;

[0018] Input query information into the text processing model.

[0019] In some implementations of the first aspect, obtaining a credibility score for each text feature based on a text scoring model includes:

[0020] Acquire second reference information, the second reference information including a text feature processing reference template;

[0021] Adding a second preset mark to the text feature data set to obtain at least one marked text feature, the second preset mark includes a second header identifier, and the second header identifier includes a processing requirement;

[0022] The second reference information and each marked text feature are input into the text scoring model, so that the text scoring model analyzes each marked text feature according to the second reference information and generates a credibility score for each text feature.

[0023] In some implementations of the first aspect, the second reference information and each marked text feature are input into a text scoring model, so that the text scoring model parses each marked text feature according to the second reference information and outputs a credibility score of each text feature, including:

[0024] Inputting the second reference information and each marked text feature into the text scoring model, so that the text scoring model parses each marked text feature according to the second reference information, obtains the analysis result of each text feature, scores each text feature according to the analysis result of each text feature, and outputs the credibility score of each text feature and the analysis result of each text feature;

[0025] The credibility score of each text feature and the analysis result of each text feature are added to the credibility evaluation result.

[0026] In some implementations of the first aspect, a credibility score of each text feature is calculated using a preset statistical algorithm to determine a credibility evaluation result of the text, including:

[0027] The mean of the credibility scores of the text features is calculated to obtain the target score, and the target score is used as the credibility evaluation result.

[0028] In a second aspect, an embodiment of the present application provides a text evaluation device, the device comprising:

[0029] An acquisition module, used for acquiring text to be evaluated, wherein the text is generated by a language model;

[0030] A processing module, used for performing information splitting processing on the text based on the text processing model to obtain a text feature data set of the text, wherein the text feature data set includes at least one text feature, each text feature includes a subject feature and an object feature, and the object feature includes event information associated with the subject feature;

[0031] The processing module is further used to perform a credibility score on each text feature based on the text scoring model to obtain a credibility score for each text feature;

[0032] The processing module is also used to calculate the credibility score of each text feature using a preset statistical algorithm to determine the credibility evaluation result of the text, and the credibility evaluation result includes the target score of the text.

[0033] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the text evaluation method described in the first aspect or any implementable manner of the first aspect is implemented.

[0034] In a fourth aspect, the present application provides a readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the text evaluation method described in the first aspect or any implementable manner of the first aspect is implemented.

[0035] In a fifth aspect, the present application provides a vehicle, characterized in that it includes a text evaluation device as in the second aspect or any implementable manner of the second aspect, or includes an electronic device as in the third aspect, or includes a readable storage medium as in the fourth aspect.

[0036] In a sixth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the text evaluation method as described in the first aspect or any implementable manner of the first aspect.

[0037] The text evaluation method, device, electronic device, readable storage medium and vehicle of the embodiments of the present application. For the text to be evaluated that is generated and output by the language model, the text is subjected to information splitting processing by the provided text processing model, so that one or more text features can be obtained. In this way, the text scoring model can conveniently score each text feature, which is conducive to improving the objectivity of text evaluation. Next, each text feature can be automatically scored respectively through the provided text scoring model, so as to obtain the credibility score of each text feature, so as to obtain a more reliable scoring score, and also realize the quantification of the authenticity of each text feature. Finally, the credibility score of each text feature is calculated in combination with a preset statistical algorithm to obtain the target score of the text, thereby obtaining an evaluation result based on a unified indicator quantification output, which is conducive to assisting the development of a large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 It is a flowchart of a text evaluation method provided in an embodiment of the present application;

[0040] Figure 2 is a structural schematic diagram of a text evaluation device provided in an embodiment of the present application;

[0041] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0043] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0044] The embodiments of the present application provide a text evaluation method, device, electronic device, readable storage medium and vehicle, which can evaluate the authenticity of the text and provide clear indicators for quantifying the authenticity of the text, which is conducive to assisting the development of large language models.

[0045] The text evaluation method provided in the embodiment of the present application is introduced below with reference to the accompanying drawings. Figure 1 is a flowchart of a text evaluation method provided in an embodiment of the present application, combined with Figure 1 As shown, the text evaluation method may include steps 110 to 140 .

[0046] Step 110, obtaining a text to be evaluated, wherein the text is generated by a language model;

[0047] Step 120, performing information splitting processing on the text based on the text processing model to obtain a text feature data set of the text, the text feature data set including at least one text feature, each text feature including a subject feature and an object feature, the object feature including event information associated with the subject feature;

[0048] Step 130, performing a credibility score on each text feature based on the text scoring model to obtain a credibility score for each text feature;

[0049] Step 140 , using a preset statistical algorithm to calculate the credibility score of each text feature, to determine a credibility evaluation result of the text, the credibility evaluation result including a target score of the text.

[0050] The above steps are described in detail below, as shown below.

[0051] First, in step 110, the language model can be a deep learning model that has not been trained or a deep learning model that has been trained. The language model can generate natural language text or understand the meaning of language text and provide it to the user in the form of text. The text to be evaluated is the text output by the language model.

[0052] For example, if the language model is asked to provide an introduction to the cartoon "Coco", the introduction to "Coco" provided by the language model is the text to be evaluated.

[0053] Next, in step 120 and step 130, the introduction of Coco is first provided to a text processing model, and the text processing model can split the text. Optionally, the text processing model can be a trained Large Language Models (LLMs), for example, a Chat Generative Pre-trained Transformer (ChatGPT) can be used as a text processing model.

[0054] The text processing model analyzes and processes the text to be evaluated to generate one or more text features. Specifically, the text features may include subject features and object features, wherein the object features include event information associated with the subject features.

[0055] After the text processing model outputs the text features, the text scoring model can then score the text features to obtain a credibility score for each text feature.

[0056] Finally, with regard to the above step 140, a preset statistical algorithm such as calculating the average, extracting the median, calculating the standard deviation, etc., is not specifically limited herein.

[0057] Taking the calculation of the average value as an example, the average value of the credibility score of each text feature is calculated, and the average value can be used as the credibility evaluation result of the text.

[0058] Based on the embodiments of the present application, for the problem to be evaluated whose text is generated and output by a language model, by providing a text processing model, one or more text features can be obtained. Next, each text feature can be automatically scored by the provided text scoring model to obtain a credibility score for each text feature, thereby quantifying the authenticity of each text feature. Finally, the credibility score of each text feature is calculated in combination with a preset statistical algorithm to obtain a credibility evaluation result of the text, thereby achieving a quantitative evaluation of the text. In addition, since the entire process does not require human intervention, it is also beneficial to save labor costs.

[0059] In some embodiments, the above-mentioned step 120 is involved, and information segmentation processing is performed on the text based on a text processing model to obtain a text feature data set of the text, which may specifically include the following steps 1201 to 1204.

[0060] Step 1201, obtaining first reference information, the first reference information including a processing reference template of a text;

[0061] Step 1202, adding a first preset mark to the text to obtain the marked text, wherein the first preset mark includes a first head mark and a first tail mark, and the first preset mark is used to instruct the text processing model to identify the text to be evaluated and the first output format of the text processing model;

[0062] Step 1203, inputting the first reference information and the marked text into a text processing model, so that the text processing model parses the marked text according to a processing reference template to generate at least one subject feature and at least one object feature;

[0063] Step 1204 , performing one-to-one matching on the subject features and the object features, generating at least one text feature in a first output format, and obtaining a text feature data set.

[0064] Specifically, the first reference information includes a text processing reference template and a first output format. Exemplarily, the text processing reference template can be understood as an input and output example of the text processing model. Optionally, a Few-Shot Learning method is used to provide an example (input and output example) for the model.

[0065] The input and output examples may include input introduction and output introduction. Specifically, the input introduction and output introduction may be determined based on the historical text and the text features corresponding to the historical text. Thus, the text processing model may process the text based on the change of text features from the historical text to the text features corresponding to the historical text, and obtain the text features corresponding to the text.

[0066] As a specific example, taking the introduction of "The Lion King" as an input and output sample, the specific input and output samples (example) corresponding to the text processing model can be shown in Table 1.

[0067] Table 1

[0068]

[0069]

[0070] For example, the subject features that the text processing model can extract from the introduction of "The Lion King" include: "The Lion King", the first "The Lion King", the second "The Lion King", the protagonist of the story, Princess Kiara, etc., which are not listed here one by one.

[0071] Correspondingly, the object features that can be extracted from the introduction of "The Lion King" are, for example: "There are two movies in total", "It is an animated film", "Produced by the British Walt Disney Pictures, co-directed by Roger Allers and Rob Minkoff, and dubbed by Matthew Broderick, James Earl Jones, Jeremy Irons, Nathan Lane and others", "It is my favorite animated film, no other film can compare to it", "It is my favorite animated film, no other film can compare to it", "Adapted from "Romeo and Juliet", "It is Princess Kiara, the daughter of the Lion King Simba", "It is the heir to this glorious land", etc., too many to list here.

[0072] Combined with the specific content of The Lion King, the text features obtained are as follows:

[0073] There are two films in The Lion King;

[0074] The Lion King is my favorite animated film, nothing else can compare to it;

[0075] The first “Lion King” was an animated film;

[0076] The first "Lion King" was produced by Walt Disney Pictures in the UK;

[0077] The first “Lion King” was co-directed by Roger Allers and Rob Minkoff;

[0078] The first "Lion King" was voiced by co-stars Matthew Broderick, Worms Earl Jones, Jeremy Irons, Nathan Lane, etc.;

[0079] The second film, The Lion King, was adapted from Romeo and Juliet;

[0080] The protagonist of the story is Princess Kiara, the daughter of the Lion King Simba;

[0081] Princess Kiara is the heir to this glorious land.

[0082] It is understandable that the above text features are merely exemplary and are not listed here one by one.

[0083] Based on the above, the text processing model can understand and learn how to split and match the files to be evaluated.

[0084] In order to facilitate the text processing model to be evaluated, a first preset mark can be added to the text, such as the first header mark and the first tail mark shown in Table 1. Exemplarily, after adding the first header mark and the first tail mark to the text to be evaluated, the content input by the text processing model can be obtained, and the content can be used as a question (query) posed to the text processing model.

[0085] Exemplarily, the first header identifier may be "Generated Content", and the first tail identifier may be "The above is the content generated by the model. Please divide it into multiple atomic information of the smallest granularity and then evaluate the correctness, and then return the result in JSON format." There is no specific limitation on the first header identifier and the first tail identifier.

[0086] Next, the first reference information and the marked text are input into the text processing model, so that the text processing model parses the marked text according to the text reference template to generate subject features and object features. It is understandable that the number of subject features can be one or more. Finally, by matching the subject features and the object features one by one, at least one text feature of the first output format is generated to obtain a text feature data set. Among them, a regular matching method can be used to match the subject features and the object features one by one.

[0087] In some embodiments, in order to make the text processing model more aware of the processing requirements for the text to be evaluated, input content including execution requirement information can also be generated. Specifically, in step 1203, the first reference information and the marked text are input into the text processing model. Specifically, the following steps can be referred to: obtaining execution requirement information, the execution requirement information includes an execution role and an execution action, wherein the execution action includes information splitting processing of the text; splicing the execution requirement information, the first reference information and the marked text to obtain query information; and inputting the query information into the text processing model.

[0088] The specific input and output requirements (instructions) are shown in the content of the execution requirement information in Table 1. For example, the execution role is an accurate large model evaluation expert, the execution action is responsible for verifying the generated content of the large language model MMM developed by XX company, and you need to divide the generated content into multiple atomic signals of the smallest granularity, etc.

[0089] The specific input and output requirements (instructions) can be specifically set according to actual application requirements. In actual application, the specific input and output requirements (instructions) mentioned above or requirements similar to the specific input and output requirements (instructions) mentioned above can be directly used.

[0090] In addition, in order to facilitate the text scoring model to distinguish and score each text feature, the output format of the text adjustment, that is, the first output format, can be preset. Optionally, the first output format can be a JSON format as shown in Table 1. Generate at least one text feature in the first output format to obtain a text feature dataset.

[0091] According to the embodiment of the present application, a formatted scoring result is obtained by fixing the input and output of the model. This method reduces the manual intervention required for scoring large models and saves a lot of manpower costs.

[0092] In some embodiments, the above step 130 is involved, and a credibility score of each text feature is obtained based on a text scoring model, and details may refer to steps 1301 to 1303 .

[0093] Step 1301, obtaining second reference information, where the second reference information includes a text feature processing reference template;

[0094] Step 1302, adding a second preset tag to the text feature data set to obtain at least one marked text feature, the second preset tag includes a second header tag, and the second header tag includes a processing requirement;

[0095] Step 1303: input the second reference information and each marked text feature into a text scoring model, so that the text scoring model analyzes each marked text feature according to the second reference information and outputs a credibility score of each text feature.

[0096] Exemplarily, the second reference information, such as the processing reference template of the text feature and the second output format, the processing reference template of the text feature can be understood as an example of the input and output of the text scoring model. Optionally, an example (input and output example) is provided to the model using the Few-Shot Learning method. It can be understood that the text processing model and the text scoring model process different contents respectively, and accordingly, the examples provided using the Few-Shot Learning method are also different.

[0097] As a specific example, we continue to take the introduction of "The Lion King" as an example to introduce the input and output samples of the text scoring model, as shown in Table 2.

[0098] Table 2

[0099]

[0100]

[0101] In some instances, in order to facilitate the text scoring model to identify text features and score the text features, a second preset tag can be added to the text. Based on the first preset tag including the second header identifier. Exemplarily, after adding the second header identifier to the text to be evaluated, the content input to the text scoring model can be obtained, and the content can be used as a query to the text scoring model.

[0102] Exemplarily, the second header identifier can be as shown in Table 2: "Given the following information, please judge their authenticity. Correct information will be scored 1, incorrect information will be scored 0, and subjective judgment or uncertain information will not be scored. You will return the result in JSON format. This JSON is a list composed of dictionary objects, each of which has three keys: "info" (atomic information), "analysis" (analysis results), and "score" (score). ", thereby instructing the text scoring model to score each of the text features according to the analysis results of each text feature.

[0103] In some examples, the second reference information and each marked text feature are input into the text scoring model, so that the text scoring model analyzes each marked text feature according to the second reference information and generates a credibility score for each text feature, wherein the credibility score is what score refers to.

[0104] In some embodiments, the second reference information and each marked text feature are input into a text scoring model, so that the text scoring model parses each marked text feature according to the second reference information and outputs a credibility score for each text feature, including:

[0105] Inputting the second reference information and each marked text feature into the text scoring model, so that the text scoring model parses each marked text feature according to the second reference information, obtains the analysis result of each text feature, scores each text feature according to the analysis result of each text feature, and outputs the credibility score of each text feature and the analysis result of each text feature;

[0106] The credibility score of each text feature and the analysis result of each text feature are added to the credibility evaluation result.

[0107] Specifically, the second header identifier can be used to instruct the text scoring model to obtain the analysis result of each text feature, and score each text feature according to the analysis result of each text feature to generate a credibility score for each text feature;

[0108] The credibility evaluation result also includes the credibility score of each text feature and the analysis result of each text feature.

[0109] Specifically, the analysis results of the text features are, for example, the contents indicated by “analysis” shown in Table 2.

[0110] According to the embodiment of the present application, firstly, a number of text features, i.e., atomic information, are separated from the text to be evaluated, and then these text features are scored one by one, and then these scores are summed and averaged, and finally the authenticity score of the text feature is obtained. Through the text evaluation method of the embodiment of the present application, the detection of possible hallucinations in the text generated or understood by the large language model can be significantly enhanced, that is, the large language model may output text that is inconsistent with the facts to achieve effective quantitative evaluation. According to the embodiment of the present application, the authenticity score obtained is more objective and clear, and has good quantitative properties, so that different texts to be evaluated can compare their authenticity indicators with each other, and the authenticity evaluation of the text to be evaluated can be stably carried out, and the scores of the same text are basically consistent.

[0111] According to the embodiment of the present application, since the analysis results of the text features are also generated when the text features are evaluated, the score of each text feature is more interpretable, which is also conducive to subsequent manual evaluation and correction.

[0112] Based on the same inventive concept, the embodiment of the present application also provides a text evaluation device to implement the text evaluation method provided in the embodiment of the present application.

[0113] Figure 2 is a structural steel schematic diagram of a text evaluation device provided in an embodiment of the present application, combined with Figure 2 As described above, the text evaluation device includes an acquisition module 210 and a processing module 220 .

[0114] An acquisition module 210, configured to acquire text to be evaluated, wherein the text is generated by a language model;

[0115] The processing module 220 is used to perform information splitting processing on the text based on the text processing model to obtain a text feature data set of the text, wherein the text feature data set includes at least one text feature, each text feature includes a subject feature and an object feature, and the object feature includes event information associated with the subject feature;

[0116] The processing module 220 is further used to perform a credibility score on each text feature based on the text scoring model to obtain a credibility score for each text feature;

[0117] The processing module 220 is further used to calculate the credibility score of each text feature using a preset statistical algorithm to determine the credibility evaluation result of the text, and the credibility evaluation result includes the target score of the text.

[0118] In some embodiments, the acquisition module 210 is further used to acquire first reference information, the first reference information including a processing reference template of the text;

[0119] The processing module 220 is further used to add a first preset mark to the text to obtain the marked text, wherein the first preset mark includes a first head mark and a first tail mark, and the first preset mark is used to instruct the text processing model to identify the text to be evaluated and the first output format of the text processing model;

[0120] The processing module 220 is further used to input the first reference information and the marked text into the text processing model, so that the text processing model parses the marked text according to the processing reference template to generate at least one subject feature and at least one object feature;

[0121] The processing module 220 is further used to perform one-to-one matching on the subject features and the object features, generate at least one text feature in the first output format, and obtain a text feature data set.

[0122] In some embodiments, the acquisition module 210 is further used to acquire execution requirement information, the execution requirement information includes an execution role and an execution action, wherein the execution action includes performing information splitting processing on the text;

[0123] The processing module 220 is further used to perform splicing processing on the execution requirement information, the first reference information and the marked text to obtain query information;

[0124] The processing module 220 is further used to input the query information into the text processing model.

[0125] In some embodiments, the acquisition module 210 is further used to acquire second reference information, where the second reference information includes a text feature processing reference template;

[0126] The processing module 220 is further used to add a second preset mark to the text feature data set to obtain at least one marked text feature, the second preset mark includes a second header mark, and the second header mark includes a processing requirement;

[0127] The processing module 220 is further used to input the second reference information and each marked text feature into the text scoring model, so that the text scoring model analyzes each marked text feature according to the second reference information and generates a credibility score for each text feature.

[0128] In some embodiments, the processing module 220 is further used to input the second reference information and each of the marked text features into the text scoring model, so that the text scoring model parses each of the marked text features according to the second reference information, obtains the analysis result of each text feature, and scores each of the text features according to the analysis result of each text feature, and outputs the credibility score of each of the text features and the analysis result of each of the text features;

[0129] The processing module is further used to add the credibility score of each of the text features and the analysis result of each of the text features to the credibility evaluation result.

[0130] In some embodiments, the processing module 220 is further used to calculate the average of the credibility scores of the text features to obtain a target score, and use the target score as a credibility evaluation result.

[0131] It can be understood that the text evaluation device 200 of the embodiment of the present application can correspond to the execution entity of the text evaluation method provided in the embodiment of the present application. The specific details of the operation and / or function of each module / unit of the text evaluation device 200 can be found in the description of the corresponding parts in the text evaluation method provided in the above embodiment of the present application. For the sake of brevity, they will not be repeated here.

[0132] According to the embodiments of the present application, for the problem to be evaluated whose text is generated and output by a language model, by providing a text processing model, one or more text features can be obtained. Next, each text feature can be automatically scored by the provided text scoring model to obtain a credibility score for each text feature, thereby quantifying the authenticity of each text feature. Finally, the credibility score of each text feature is calculated in combination with a preset statistical algorithm to obtain a credibility evaluation result of the text and a quantitative evaluation result of the text. In addition, since the entire process does not require human intervention, it is also beneficial to save labor costs.

[0133] In addition, an embodiment of the present application further provides a vehicle, including the text evaluation device provided in the embodiment of the present application or including the electronic device provided in the embodiment of the present application, or the readable storage medium provided in the embodiment of the present application.

[0134] Figure 3 FIG. 1 is a schematic diagram showing the structure of an electronic device provided by an embodiment of the present application. Figure 3As shown, the device may include a processor 301 and a memory 302 storing computer program instructions.

[0135] Specifically, the processor 301 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0136] The memory 302 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 302 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In one example, the memory 302 may include a removable or non-removable (or fixed) medium, or the memory 302 is a non-volatile solid-state memory. The memory 302 may be inside or outside the electronic device.

[0137] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0138] The processor 301 implements the method described in the embodiment of the present application by reading and executing the computer program instructions stored in the memory 302, and achieves the corresponding technical effect achieved by executing the method in the embodiment of the present application, which will not be repeated here for the sake of brevity.

[0139] In one example, the electronic device may further include a communication interface 303 and a bus 304. Figure 3 As shown, the processor 301, the memory 302, and the communication interface 303 are connected via a bus 304 and communicate with each other.

[0140] The communication interface 303 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0141] Bus 304 includes hardware, software or both, and the components of online information flow billing equipment are coupled to each other. For example, but not limitation, the bus may include Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), Hyper Transport (HT) interconnection, Industry Standard Architecture (ISA) bus, InfiniBand interconnection, Low Pin Count (LPC) bus, Memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 304 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0142] The electronic device can execute the text evaluation method in the embodiment of the present application, thereby achieving the corresponding technical effects of the text evaluation method described in the embodiment of the present application.

[0143] In addition, in combination with the text evaluation method in the above embodiment, the embodiment of the present application may provide a readable storage medium for implementation. The readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the text evaluation methods in the above embodiment is implemented. Examples of readable storage media may be non-transitory machine-readable media, such as electronic circuits, semiconductor memory devices, read-only memories (ROM), floppy disks, compact discs (CD-ROM), optical disks, hard disks, etc.

[0144] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0145] The functional blocks shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or communication link by a data signal carried in a carrier. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (Read-Only Memory, ROM), flash memory, erasable read-only memory (Erasable ReadOnly Memory, EROM), floppy disks, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical discs, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0146] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0147] In addition, in combination with the text evaluation method, device and electronic device in the above embodiments, the present application embodiment can provide a computer program product for implementation. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes any one of the text evaluation methods in the above embodiments.

[0148] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0149] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A text evaluation method, It is characterized in that The method comprises: Obtaining text to be evaluated, wherein the text is generated by a language model; Performing information splitting processing on the text based on a text processing model to obtain a text feature data set of the text, wherein the text feature data set includes at least one text feature, each of the text features includes a subject feature and an object feature, and the object feature includes event information associated with the subject feature; Performing a credibility score on each of the text features based on a text scoring model to obtain a credibility score for each of the text features; A preset statistical algorithm is used to calculate the credibility score of each of the text features to determine a credibility evaluation result of the text, wherein the credibility evaluation result includes a target score of the text.

2. The method according to claim 1, It is characterized in that The step of performing information splitting processing on the text based on the text processing model to obtain a text feature data set of the text includes: Acquire first reference information, wherein the first reference information includes a processing reference template for the text; Adding a first preset mark to the text to obtain a marked text, wherein the first preset mark includes a first head mark and a first tail mark, and the first preset mark is used to instruct the text processing model to identify the text to be evaluated and a first output format of the text processing model; Inputting the first reference information and the marked text into the text processing model, so that the text processing model parses the marked text according to the processing reference template to generate at least one subject feature and at least one object feature; The subject features and the object features are matched one by one to generate the at least one text feature in the first output format, thereby obtaining the text feature data set.

3. The method according to claim 2, It is characterized in that The step of inputting the first reference information and the marked text into the text processing model comprises: Acquire execution requirement information, wherein the execution requirement information includes an execution role and an execution action, wherein the execution action includes performing information splitting processing on the text; The execution requirement information, the first reference information and the marked text are concatenated to obtain query information; The query information is input into the text processing model.

4. The method according to claim 1, It is characterized in that The obtaining of the credibility score of each of the text features based on the text scoring model includes: Acquire second reference information, where the second reference information includes the text feature processing reference template; adding a second preset mark to the text feature data set to obtain at least one marked text feature, wherein the second preset mark includes a second header identifier, and the second header identifier includes a processing requirement; The second reference information and each of the marked text features are input into the text scoring model, so that the text scoring model parses each of the marked text features according to the second reference information and outputs a credibility score of each of the text features.

5. The method according to claim 4, It is characterized in that The step of inputting the second reference information and each of the marked text features into the text scoring model so that the text scoring model parses each of the marked text features according to the second reference information and outputs a credibility score of each of the text features comprises: Inputting the second reference information and each of the marked text features into the text scoring model, so that the text scoring model analyzes each of the marked text features according to the second reference information, obtains an analysis result of each text feature, scores each of the text features according to the analysis result of each text feature, and outputs a credibility score of each of the text features and an analysis result of each of the text features; The credibility score of each of the text features and the analysis result of each of the text features are added to the credibility evaluation result.

6. The method according to claim 1, It is characterized in that The using of a preset statistical algorithm to calculate the credibility score of each of the text features to determine the credibility evaluation result of the text includes: The mean of the credibility scores of the text features is calculated to obtain the target score, and the target score is used as the credibility evaluation result.

7. A text evaluation device, It is characterized in that The device comprises: An acquisition module, used for acquiring text to be evaluated, wherein the text is generated by a language model; A processing module, configured to perform information splitting processing on the text based on a text processing model to obtain a text feature data set of the text, wherein the text feature data set includes at least one text feature, each of the text features includes a subject feature and an object feature, and the object feature includes event information associated with the subject feature; The processing module is further used to perform credibility scoring on each of the text features based on the text scoring model to obtain a credibility score for each of the text features; The processing module is further used to calculate the credibility score of each of the text features using a preset statistical algorithm to determine a credibility evaluation result of the text, wherein the credibility evaluation result includes a target score of the text.

8. An electronic device, It is characterized in that The device comprises: a processor, and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the text evaluation method according to any one of claims 1 to 6.

9. A readable storage medium, It is characterized in that The readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the text evaluation method according to any one of claims 1 to 6 is implemented.

10. A vehicle, It is characterized in that The vehicle comprises the text evaluation device according to claim 7 , or comprises the electronic device according to claim 8 , or the readable storage medium according to claim 9 .

Images