Text evaluation method, apparatus, device, and storage medium

Abstract

The application relates to the field of artificial intelligence and discloses a text evaluation method, device and equipment and a storage medium, the method comprising the following steps: obtaining a target text to be evaluated and a reference text used for evaluating the target text; the target text is output in advance by a text generation model; a mapping relationship between the target text and the reference text is determined based on the text features of the target text and the text features of the reference text; the mapping relationship comprises a confidence degree between characters in the target text and the reference text or a confidence degree between characters in the reference text and the target text; and an evaluation result of the target text is generated based on the mapping relationship. In this way, the similarity between the generated text and the reference text is fully considered from the aspects of character features and global features of the reference text, and the quality of the generated text is objectively and accurately evaluated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a text evaluation method, device, equipment and storage medium Background Art

[0002] Text generation is a key research area in natural language processing (NLP) and has achieved remarkable success. However, evaluating the quality of generated text remains a challenging problem. Evaluating text generation can be viewed as calculating the similarity between a generated sentence and a standard reference sentence. Related techniques often segment sentences and then calculate similarity. Methods such as BLEU (Bilingual Evaluation Understudy) and ROUGE-N (Recall-Oriented Understudy for Gisting Evaluation) use an N-gram model to evaluate the similarity between generated sentences and standard reference sentences. However, these evaluation methods are limited in form and utilize the collocation information between adjacent words within a sentence, without considering global semantic matching. This results in poorly rationalized text evaluation results. Therefore, developing a reasonable evaluation metric to objectively and accurately evaluate the quality of generated text is an urgent issue that needs to be addressed. Summary of the Invention

[0003] The present invention provides a text evaluation method, apparatus, device and storage medium for improving the rationality of text evaluation.

[0004] To achieve the above-mentioned objectives, the first aspect of the present invention provides a text evaluation method, comprising: obtaining a target text to be evaluated, and a reference text for evaluating the target text; wherein the target text is pre-output by a text generation model; based on the text features of the target text and the text features of the reference text, determining a mapping relationship between the target text and the reference text; wherein the mapping relationship includes: the confidence between the characters in the target text and the reference text, or the confidence between the characters in the reference text and the target text; based on the mapping relationship, generating an evaluation result of the target text.

[0005] Optionally, in a first implementation method of the first aspect of the present invention, the above-mentioned step of determining the mapping relationship between the target text and the reference text based on the text features of the target text and the text features of the reference text includes: obtaining the character features of each character in the target text and the character features of each character in the reference file; for each character in the target text, comparing the character features of the current character with the character features of each character in the reference file to obtain the similarity; determining the maximum similarity as the confidence of the current character; and determining the similarity of each character of the target text as the mapping relationship between the target text and the reference text.

[0006] Optionally, in a second implementation method of the first aspect of the present invention, the step of determining the mapping relationship between the target text and the reference text based on the text features of the target text and the text features of the reference text includes: splicing the target text and the reference text to obtain a spliced ​​text; inputting the spliced ​​text into a pre-trained language representation model and outputting the text features of the spliced ​​text; inputting the text features into a preset multi-layer perceptron layer and outputting an intermediate result; and inputting the intermediate result into a preset activation function to obtain a mapping relationship between the target text and the reference text.

[0007] Optionally, in a third implementation of the first aspect of the present invention, the language representation model is trained using a preset training sample set; the training sample set includes multiple sets of training sample pairs; and the sample labels of the training sample pairs are used to indicate the confidence between the text samples in the training sample pairs. If the training sample pair includes the sample text and a sample summary of the sample text, the corresponding confidence for the training sample pair is one; if the training sample object includes the sample text and text unrelated to the sample text, the corresponding confidence for the training sample pair is zero.

[0008] Optionally, in a fourth implementation method of the first aspect of the present invention, the above-mentioned step of generating an evaluation result of the target text based on the mapping relationship includes: if the target text is a compressed text of the original text, obtaining a first mapping relationship in which the target text is mapped to the original text, and a second mapping relationship in which the reference text is mapped to the target text; calculating a first average value of each confidence level in the first mapping relationship, and a second average value of each confidence level in the second mapping relationship; and determining the product of the first average value and the second average value as the evaluation result of the target text.

[0009] Optionally, in the fifth implementation of the first aspect of the present invention, the above-mentioned step of generating the evaluation result of the target text based on the mapping relationship includes: if the target text is a rewritten text of the reference text, obtaining a first mapping relationship mapping the target text to the reference text, and a second mapping relationship mapping the reference text to the target text; calculating a first average value of each confidence level in the first mapping relationship, and a second average value of each confidence level in the second mapping relationship; and determining the product of the first average value and the second average value as the evaluation result of the target text.

[0010] Optionally, in a sixth implementation of the first aspect of the present invention, the above-mentioned step of generating an evaluation result of the target text based on the mapping relationship includes: if the target text is the answer text of the reference text, obtaining a first mapping relationship that maps the target text to the reference text; wherein the reference text includes: the original text and the question text; and determining the sum of each confidence level in the first mapping relationship as the evaluation result of the target text.

[0011] The second aspect of the present invention provides a text evaluation device, comprising: a first acquisition module, used to acquire a target text to be evaluated, and a reference text for evaluating the target text; wherein the target text is pre-output by a text generation model; a first determination module, used to determine the mapping relationship between the target text and the reference text based on the text features of the target text and the text features of the reference text; wherein the mapping relationship includes: the confidence between the characters in the target text and the reference text, or the confidence between the characters in the reference text and the target text; a first generation module, used to generate an evaluation result of the target text based on the mapping relationship.

[0012] Optionally, in a first implementation method of the second aspect of the present invention, the above-mentioned first determination module is also used to: obtain character features of each character in the target text and character features of each character in the reference file; for each character in the target text, compare the character features of the current character with the character features of each character in the reference file to obtain similarity; determine the maximum similarity as the confidence of the current character; determine the similarity of each character of the target text as the mapping relationship between the target text and the reference text.

[0013] Optionally, in a second implementation of the second aspect of the present invention, the above-mentioned first determination module is also used to: splice the target text and the reference text to obtain a spliced ​​text; input the spliced ​​text into a pre-trained language representation model to output text features of the spliced ​​text; input the text features into a preset multi-layer perceptron layer to output an intermediate result; input the intermediate result into a preset activation function to obtain a mapping relationship between the target text and the reference text.

[0014] Optionally, in a third implementation of the second aspect of the present invention, the language representation model is trained using a preset training sample set; the training sample set includes multiple sets of training sample pairs; and the sample labels of the training sample pairs are used to indicate the confidence between the text samples in the training sample pairs. If the training sample pair includes the sample text and a sample summary of the sample text, the corresponding confidence for the training sample pair is one; if the training sample object includes the sample text and text unrelated to the sample text, the corresponding confidence for the training sample pair is zero.

[0015] Optionally, in a fourth implementation method of the second aspect of the present invention, the above-mentioned first generation module is also used to: if the target text is a compressed text of the original text, obtain a first mapping relationship of the target text to the original text, and a second mapping relationship of the reference text to the target text; calculate a first average value of each confidence level in the first mapping relationship, and a second average value of each confidence level in the second mapping relationship; and determine the product of the first average value and the second average value as the evaluation result of the target text.

[0016] Optionally, in a fifth implementation of the second aspect of the present invention, the above-mentioned first generation module is also used to: if the target text is a rewritten text of the reference text, obtain a first mapping relationship of the target text to the reference text, and a second mapping relationship of the reference text to the target text; calculate a first average value of each confidence level in the first mapping relationship, and a second average value of each confidence level in the second mapping relationship; and determine the product of the first average value and the second average value as the evaluation result of the target text.

[0017] Optionally, in a sixth implementation of the second aspect of the present invention, the above-mentioned first generation module is also used to: if the target text is the answer text of the reference text, obtain a first mapping relationship between the target text and the reference text; wherein the reference text includes: the original text and the question text; and determine the sum of the confidence levels in the first mapping relationship as the evaluation result of the target text.

[0018] A third aspect of the present invention provides a text evaluation device, comprising: a memory and at least one processor, wherein instructions are stored in the memory; and the at least one processor calls the instructions in the memory so that the text evaluation device executes the above-mentioned text evaluation method.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is caused to execute the above-mentioned text evaluation method.

[0020] The technical solution provided by the present invention comprises obtaining a target text to be evaluated and a reference text for evaluating the target text; wherein the target text is pre-output by a text generation model; determining a mapping relationship between the target text and the reference text based on the text features of the target text and the text features of the reference text; wherein the mapping relationship includes: the confidence between characters in the target text and the reference text, or the confidence between characters in the reference text and the target text; and generating an evaluation result for the target text based on the mapping relationship. In this method, the mapping relationship between the generated text and the reference text is defined using methods such as the confidence between characters in the target text and the reference text, or the confidence between characters in the reference text and the target text, and the evaluation result for the target text is generated based on this mapping relationship. In this method, the semantic similarity between the generated text and the reference text is fully considered from the aspects of character features and global features of the reference text, objectively and accurately evaluating the quality of the generated text. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of an embodiment of a text evaluation method in an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of another embodiment of the text evaluation method in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of another embodiment of the text evaluation method in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of an embodiment of a text evaluation device in an embodiment of the present invention;

[0025] Figure 5 Schematic diagram of an embodiment of a text evaluation device in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] Embodiments of the present invention provide a text evaluation method, apparatus, device, and storage medium for improving the accuracy of facial feature extraction.

[0027] The terms "first," "second," "third," "fourth," and the like (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0028] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the text evaluation method in the embodiment of the present invention includes:

[0029] 101. Obtain a target text to be evaluated and a reference text for evaluating the target text; wherein the target text is pre-output by a text generation model;

[0030] The target text is the text to be evaluated, which can be a sentence, a paragraph, or a chapter. The target text is output by a text generation model. This text generation model can refer to existing technologies, including neural network models such as LSTM (Long-Short Term Memory) and GRU (Gate Recurrent Unit). The reference text serves as a reference target. The quality of the generated text is judged by comparing the target text with the target text on properties such as fluency and internal relevance. The number and source of reference text vary depending on the target text application scenario. Generally speaking, text generation is commonly used in scenarios such as compression, rewriting, and question-answering. When a sentence or a few sentences are extracted from the original text to form a text summary, the target text is a compressed version of the original text. In this case, the reference text can be the original text and a standard text. The standard text can be a high-quality human-written text or a text generated using a highly recognized model, without limitation. When the target text is the answer to the reference text, the reference text includes the original text and the question text. When text generation is used for rewriting, the reference text is the original text.

[0031] The target generated text is obtained through the text generation model, and the reference text is obtained from the corresponding channels based on the different application scenarios of the target text.

[0032] In this step, the target text to be evaluated and the reference text used to evaluate the target text are obtained from various channels. Among them, the original text, standard text, and problem text can all be used as reference texts. According to different application scenarios, they are included in the text evaluation reference range, and the reference text range is wider.

[0033] 102. Determine a mapping relationship between the target text and the reference text based on the text features of the target text and the text features of the reference text; wherein the mapping relationship includes: confidence levels between characters in the target text and the reference text, or confidence levels between characters in the reference text and the target text;

[0034] Here, a text feature is a basic unit within a text that can be used to identify its content and distinguish it from other texts. It includes semantics, structure, and other aspects. It can be a character, word, or phrase, and is typically acquired through a pre-trained neural network model. The mapping relationship is defined by the confidence level between character features and overall text features. This can be achieved using a multi-layer perceptron (MLP) and corresponding activation functions. Confidence, also known as reliability, can be understood as follows: the greater the confidence level in the comparison results of two objects, the higher the similarity between them.

[0035] In this method, based on the text features of the target text and the text features of the reference text, the confidence between the characters and the entire text features is used to express the mapping relationship between the two texts, and the correspondence between the two texts is obtained from the aspects of characters and text content.

[0036] 103. Based on the mapping relationship, generate the evaluation results of the target text.

[0037] Specifically, for different application scenarios, corresponding reference texts can be configured, and different calculation methods can be used to process the target text according to the mapping relationship between the target text and the reference text to generate an evaluation result of the target text.

[0038] Furthermore, the text generation quality can be determined as excellent when the evaluation result is greater than or equal to the first set threshold; the text generation quality can be determined as good when the evaluation result is greater than or equal to the second set threshold and less than the first set threshold; the text generation quality can be determined as poor when the evaluation index is less than the second set threshold; wherein the second set threshold is less than the first set threshold.

[0039] In this step, the confidence between the characters in the target text and the reference text, and the confidence between the characters in the reference text and the target text are used as evaluation indicators to evaluate the generation quality of the target text. Considering the character features, the structure of the reference text, the theme, etc., it better reflects the semantic relationship between the generated text and the reference text, and objectively and accurately evaluates the quality of the generated text.

[0040] The above-mentioned text evaluation method obtains a target text to be evaluated and a reference text for evaluating the target text; wherein the target text is pre-output by a text generation model; based on the text features of the target text and the text features of the reference text, a mapping relationship between the target text and the reference text is determined; wherein the mapping relationship includes: the confidence between the characters in the target text and the reference text, or the confidence between the characters in the reference text and the target text; based on the mapping relationship, an evaluation result of the target text is generated. In this method, the mapping relationship between the generated text and the reference text is defined by the confidence between the characters in the target text and the reference text, or the confidence between the characters in the reference text and the target text, and the evaluation result of the target text is generated based on this mapping relationship. In this method, the similarity of the semantic content between the generated text and the reference text is fully considered from the aspects of character features and global features of the reference text, and the quality of the generated text is objectively and accurately evaluated.

[0041] See Figure 2 ,This solution provides an exemplary embodiment and introduces the ,above-mentioned text evaluation method in detail.

[0042] 201. Obtain a target text to be evaluated and a reference text for evaluating the target text; wherein the target text is pre-output by a text generation model;

[0043] This step is the same as step 101, so it will not be repeated here.

[0044] 202. Obtain the character features of each character in the target text and the character features of each character in the reference file; for each character in the target text, compare the character features of the current character with the character features of each character in the reference file to obtain similarity; determine the maximum similarity as the confidence of the current character; determine the similarity of each character of the target text as a mapping relationship between the target text and the reference text.

[0045] The character features of each character in the target text and the reference text are obtained by a pre-trained neural network model. Optionally, the neural network model includes a BERT (Bidirectional Encoder Representation from Transformers) model. By training the neural network model using training samples with content similarity labels and generated sentence recognition labels, a BERT evaluation model can be obtained, which can analyze the semantic features of the text. In one embodiment, the target text and the reference text are input into a preset BERT model and the token sequence generated after lexical analysis is embedded to obtain the feature distribution vector of each character in the target text and the reference text. For each character in the target text, the character feature distribution vector of the current character is compared with the character feature distribution vector of each character in the reference file, the cosine similarity of the character feature distribution vector is calculated, and the value of the maximum similarity is determined as the confidence of the current character.

[0046] Furthermore, the mapping relationship of each character in the target text is determined to obtain the similarity of all characters in the entire target text.

[0047] In a specific implementation, this embodiment can provide a formula for defining a mapping relationship between a target text A and a reference text B. Assuming that the length of text A is N characters, the confidence level (in the range of 0-1) of each character a in text A corresponding to text B is:

[0048] reflex(a→B)={a1,a2,…a N}

[0049] Among them, a N The maximum cosine similarity value is obtained by comparing the character feature of the Nth character in the target text A with the character features of all texts in B; {a1, a2, ... a N} is the set of the maximum cosine similarity values ​​obtained after comparing the character features of all characters in the target text A with those of text B.

[0050] In this step, a pre-trained neural network model is used to analyze the semantic features of the text to obtain the character features of the target text and the reference text. The character features of each character in the target text are compared with the character features of the reference file characters for similarity, and the maximum similarity is determined as the confidence of the character, thereby obtaining a set of similarities of all characters in the entire target file.

[0051] 203. Based on the mapping relationship, generate an evaluation result of the target text.

[0052] This step is the same as step 103 and will not be described in detail.

[0053] In this text evaluation method, a trained evaluation model is used to analyze the semantic features of the text. The character features of each character in the target text are compared with those of the reference file for similarity. The confidence level of that character is determined by the highest similarity, which then yields the aggregate similarity of all characters in the entire target file. This approach fully considers the semantic similarity between the reference and generated texts, both in terms of characters and overall text content, allowing for an objective and accurate evaluation of the quality of the generated text.

[0054] See Figure 3 ,This solution also provides an exemplary embodiment to introduce the ,above-mentioned text evaluation method in detail.

[0055] 301. Obtain a target text to be evaluated and a reference text for evaluating the target text; wherein the target text is pre-output by a text generation model;

[0056] This step is the same as step 101, so it will not be repeated here.

[0057] 302. Concatenate the target text and the reference text to obtain a concatenated text; input the concatenated text into a pre-trained language representation model to output text features of the concatenated text; input the text features into a preset multi-layer perceptron layer to output an intermediate result; input the intermediate result into a preset activation function to obtain a mapping relationship between the target text and the reference text;

[0058] Here, the target text and the reference text are combined and spliced ​​to obtain text pairs, which are input into a pre-trained language representation model, and the text features of the spliced ​​text are output. The mapping relationship between the target text and the reference text is obtained through a preset multi-layer perceptron layer and a preset activation function. Among them, the language representation model can be trained by a preset training sample set, and the training sample set includes multiple groups of training sample pairs. The sample labels of the training sample pairs are used to indicate the confidence between the text samples in the training sample pairs. When the training sample pair includes sample text and a sample summary of the sample text, the confidence corresponding to the training sample pair is set to one; it can be understood that the sample summary is obtained by extracting the original content of the sample through multiple features such as keywords and positions. Therefore, the characters in each sample summary can be matched with the characters in the sample text, and the confidence corresponding to the training sample pair is the highest. When the training sample object includes sample text and text unrelated to the sample text, the confidence corresponding to the training sample pair is set to the lowest, which is set to zero.

[0059] In one embodiment, the language representation model includes a pre-trained BERT model for analyzing the semantic features of text. The target text and reference text are concatenated to form a text pair, which is then input into the model to obtain text features. The text features are then fed into a pre-set multi-layer perceptron (MLP) layer, which outputs a multidimensional feature vector. The output vector is then passed through a sigmoid activation function (which maps numbers to a range of 0-1) to obtain the similarity of all characters in the entire target text.

[0060] 303. Generate an evaluation result of the target text based on the mapping relationship.

[0061] This step is the same as step 103 and will not be described in detail.

[0062] In this text evaluation method, a language representation model is trained using a preset training sample set. The target text and reference text are concatenated to form text pairs, which are then fed into the model to obtain the semantic features of the texts. A mapping relationship between the target text and the reference text is then generated using a preset multi-layer perceptron layer and activation function. Based on this mapping relationship, an evaluation metric is generated that fully considers the semantic similarity between the reference and generated texts in terms of both character and text content, objectively and accurately evaluating the quality of the generated texts.

[0063] The following embodiments provide a specific implementation method for generating evaluation results of a target text based on a mapping relationship.

[0064] As mentioned above, the reference text serves as a reference target. The quality of the generated text is judged by comparing the properties of the target text with the reference text, such as fluency and internal relevance. The number and source of reference texts vary depending on the scenario in which the target text is used.

[0065] In one specific method, the reference text includes the original text of the standard text and the target text. If the target text is a compressed text of the original text, a first mapping relationship of mapping the target text to the original text and a second mapping relationship of mapping the standard text to the target text are obtained; a first average value of each confidence level in the first mapping relationship and a second average value of each confidence level in the second mapping relationship are calculated; and the product of the first average value and the second average value is determined as the evaluation result of the target text.

[0066] Specifically, when extracting a sentence or a few sentences from the original text to form a text summary, the target text is a compressed version of the original text, and the parameter text is the original text of the standard text and the target text. The standard text is a high-quality text produced by human editing. In this case, we hope that the target text does not exceed the scope of the original text, and we also hope that the generated target text can perfectly cover the content of the standard text.

[0067] Therefore, this embodiment can provide an evaluation index formula for this scenario, namely:

[0068] Score=mean(reflex(y→X))×mean(reflex(z→Y))

[0069] Wherein, X is the original text; Y is the target text, Z is the standard text, mean(reflex(y→X)) is the first average value of each confidence level in the first mapping relationship; mean(reflex(z→Y)) is the second average value of each confidence level in the second mapping relationship.

[0070] In another specific method, if the target text is a rewritten text of the reference text, a first mapping relationship between the target text and the reference text, and a second mapping relationship between the reference text and the target text are obtained; a first average value of each confidence level in the first mapping relationship and a second average value of each confidence level in the second mapping relationship are calculated; and the product of the first average value and the second average value is determined as the evaluation result of the target text.

[0071] Specifically, for example, in scenarios such as text generation and text rewriting, the reference text is the original text. Here, we hope that the target text can cover the content of the original text, and the original text can also cover the content of the target text.

[0072] Therefore, this embodiment can provide an evaluation index formula for this scenario, namely:

[0073] Score=mean(reflex(y→X))×mean(reflex(x→Y))

[0074] Wherein, X is the original text; Y is the target text, mean(reflex(y→X)) is the first average value of each confidence level in the first mapping relationship; mean(reflex(x→Y)) is the second average value of each confidence level in the second mapping relationship.

[0075] There is another specific method. If the target text is the answer text of the reference text, a first mapping relationship is obtained in which the target text is mapped to the reference text; wherein the reference text includes: the original text and the question text; and the sum of the confidence levels in the first mapping relationship is determined as the evaluation result of the target text.

[0076] Specifically, this scenario includes online customer service, knowledge acquisition, and emotional chat. In this scenario, the reference text includes the original text and the question text. Here, the question text is used to generate the target text (the answer text). We want the target text to be generated based on the question text and the reference text. For each character in the target text, the higher the character feature evaluation score, the better. Therefore, we use a summation method to calculate the confidence results.

[0077] Therefore, this embodiment can provide an evaluation index formula for this scenario, namely:

[0078] Score=sum(reflex(a→[Q,C]))

[0079] Among them, a is the character in the target text (answer text); Q is the question text, and C is the original text.

[0080] For the compression, rewriting, and question-and-answer application scenarios of generated texts, corresponding reference texts are configured. Evaluation indicators are generated by considering multiple aspects such as the original text, standard text, and question text. Based on the evaluation indicators, the properties of the target text and the reference text, such as fluency and internal correlation, are compared. This indicator expands the calculation scope and makes the judgment of the quality of the generated text more reasonable and accurate.

[0081] The above-mentioned text evaluation method obtains a target text to be evaluated and a reference text for evaluating the target text; wherein the target text is pre-output by a text generation model; based on the text features of the target text and the text features of the reference text, a mapping relationship between the target text and the reference text is determined; wherein the mapping relationship includes: the confidence between the characters in the target text and the reference text, or the confidence between the characters in the reference text and the target text; based on the mapping relationship, an evaluation result of the target text is generated. In this method, the mapping relationship between the generated text and the reference text is defined by the confidence between the characters in the target text and the reference text, or the confidence between the characters in the reference text and the target text, and the evaluation result of the target text is generated based on this mapping relationship. In this method, the similarity of the semantic content between the generated text and the reference text is fully considered from the aspects of character features and global features of the reference text, and the quality of the generated text is objectively and accurately evaluated.

[0082] The text evaluation device in the embodiment of the present invention is described below. Figure 4 , an embodiment of the text evaluation device in the embodiment of the present invention includes:

[0083] The first acquisition module 401 is used to acquire a target text to be evaluated and a reference text for evaluating the target text; wherein the target text is output in advance by a text generation model;

[0084] A first determination module 402 is configured to determine a mapping relationship between a target text and a reference text based on text features of the target text and text features of the reference text; wherein the mapping relationship includes: confidence levels between characters in the target text and the reference text, or confidence levels between characters in the reference text and the target text;

[0085] The first generating module 403 is used to generate an evaluation result of the target text based on the mapping relationship.

[0086] The above-described text evaluation method, apparatus, device, and storage medium obtain a target text to be evaluated and a reference text for evaluating the target text. The target text is pre-output by a text generation model. A mapping relationship between the target text and the reference text is determined based on the text features of the target text and the text features of the reference text. The mapping relationship includes the confidence level between characters in the target text and the reference text, or the confidence level between characters in the reference text and the target text. Based on the mapping relationship, an evaluation result for the target text is generated. This method fully considers the semantic content similarity between the generated text and the reference text from the perspective of character features and global features of the reference text, objectively and accurately evaluating the quality of the generated text.

[0087] The above-mentioned first determination module is also used to: obtain the character features of each character in the target text and the character features of each character in the reference file; for each character in the target text, compare the character features of the current character with the character features of each character in the reference file to obtain the similarity; determine the maximum similarity as the confidence of the current character; determine the similarity of each character of the target text as the mapping relationship between the target text and the reference text.

[0088] The above-mentioned first determination module is also used to: splice the target text and the reference text to obtain a spliced ​​text; input the spliced ​​text into a pre-trained language representation model to output the text features of the spliced ​​text; input the text features into a preset multi-layer perceptron layer to output an intermediate result; input the intermediate result into a preset activation function to obtain a mapping relationship between the target text and the reference text.

[0089] The language representation model is trained using a preset training sample set, which includes multiple training sample pairs. The sample labels for each training sample pair are used to indicate the confidence level between the text samples in the training sample pair. If a training sample pair includes both the sample text and a sample summary of the sample text, the corresponding confidence level for the training sample pair is one; if a training sample object includes both the sample text and text unrelated to the sample text, the corresponding confidence level for the training sample pair is zero.

[0090] The above-mentioned first generation module is also used to: if the target text is a compressed text of the original text, obtain a first mapping relationship between the target text and the original text, and a second mapping relationship between the reference text and the target text; calculate the first average value of each confidence level in the first mapping relationship, and the second average value of each confidence level in the second mapping relationship; and determine the product of the first average value and the second average value as the evaluation result of the target text.

[0091] The above-mentioned first generation module is also used to: if the target text is a rewritten text of the reference text, obtain a first mapping relationship between the target text and the reference text, and a second mapping relationship between the reference text and the target text; calculate the first average value of each confidence level in the first mapping relationship, and the second average value of each confidence level in the second mapping relationship; and determine the product of the first average value and the second average value as the evaluation result of the target text.

[0092] The above-mentioned first generation module is also used to: if the target text is the answer text of the reference text, obtain a first mapping relationship between the target text and the reference text; wherein the reference text includes: the original text and the question text; and determine the sum of the confidence levels in the first mapping relationship as the evaluation result of the target text.

[0093] above Figure 4 The text evaluation apparatus in the embodiment of the present invention is described in detail from the perspective of modularization. The text evaluation device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0094] Figure 5 is a schematic diagram of the structure of a text evaluation device provided by an embodiment of the present invention. The text evaluation device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. The memory 520 and the storage medium 530 may be temporary storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), each of which may include a series of instruction operations in the text evaluation device 500. Furthermore, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the text evaluation device 500.

[0095] The text evaluation device 500 may further include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 5 The illustrated structure of the text evaluation device does not limit the text evaluation device, and the device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0096] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to execute the steps of the text evaluation method.

[0097] The present invention also provides a text evaluation device, which includes a memory and a processor. The memory stores instructions, and when the instructions are executed by the processor, the processor executes the steps of the text evaluation method in the above embodiments.

[0098] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0099] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0100] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0101] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0102] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text evaluation method, characterized in that: The text evaluation method comprises: Obtaining a target text to be evaluated and a reference text for evaluating the target text; wherein the target text is pre-output by a text generation model; Determining a mapping relationship between the target text and the reference text based on the text features of the target text and the text features of the reference text; wherein the mapping relationship includes: a confidence level between characters in the target text and the reference text, or a confidence level between characters in the reference text and the target text; the confidence level between characters in the target text and the reference text is determined by comparing the similarity between the character features of each character in the target text and the character features of the reference text characters; Based on the mapping relationship, an evaluation result of the target text is generated.

2. The text evaluation method according to claim 1, characterized in that The step of determining a mapping relationship between the target text and the reference text based on the text features of the target text and the text features of the reference text includes: Obtaining a character feature of each character in the target text and a character feature of each character in the reference text; For each character in the target text, the character feature of the current character is compared with the character feature of each character in the reference text to obtain similarity; the maximum similarity is determined as the confidence of the current character; The confidence of each character of the target text is determined as a mapping relationship between the target text and the reference text.

3. The text evaluation method according to claim 1, characterized in that The step of determining a mapping relationship between the target text and the reference text based on the text features of the target text and the text features of the reference text includes: Splicing the target text and the reference text to obtain a spliced ​​text; Inputting the concatenated text into a pre-trained language representation model, and outputting text features of the concatenated text; Input the text features into a preset multi-layer perceptron layer and output an intermediate result; The intermediate result is input into a preset activation function to obtain a mapping relationship between the target text and the reference text.

4. The text evaluation method according to claim 3, characterized in that: The language representation model is obtained by training a preset training sample set; the training sample set includes multiple sets of training sample pairs; the sample labels of the training sample pairs are used to indicate the confidence between the text samples in the training sample pairs; If the training sample pair includes a sample text and a sample summary of the sample text, the confidence corresponding to the training sample pair is one; if the training sample pair includes a sample text and a text unrelated to the sample text, the confidence corresponding to the training sample pair is zero.

5. The text evaluation method according to claim 1, characterized in that: The reference text includes a standard text and an original text of the target text; The step of generating an evaluation result of the target text based on the mapping relationship includes: If the target text is a compressed version of the original text, obtaining a first mapping relationship between the target text and the original text, and a second mapping relationship between the standard text and the target text; Calculating a first average value of each confidence level in the first mapping relationship and a second average value of each confidence level in the second mapping relationship; The product of the first average value and the second average value is determined as the evaluation result of the target text.

6. The text evaluation method according to claim 1, characterized in that: The step of generating an evaluation result of the target text based on the mapping relationship includes: If the target text is a rewritten text of the reference text, obtaining a first mapping relationship between the target text and the reference text, and a second mapping relationship between the reference text and the target text; Calculating a first average value of each confidence level in the first mapping relationship and a second average value of each confidence level in the second mapping relationship; The product of the first average value and the second average value is determined as the evaluation result of the target text.

7. The text evaluation method according to claim 1, characterized in that: The step of generating an evaluation result of the target text based on the mapping relationship includes: If the target text is the answer text of the reference text, obtaining a first mapping relationship between the target text and the reference text; wherein the reference text includes: an original text and a question text; The sum of the confidence levels in the first mapping relationship is determined as the evaluation result of the target text.

8. A text evaluation device, characterized in that: The device comprises: A first acquisition module is used to acquire a target text to be evaluated and a reference text for evaluating the target text; wherein the target text is output in advance by a text generation model; A first determination module is configured to determine a mapping relationship between the target text and the reference text based on text features of the target text and text features of the reference text; wherein the mapping relationship includes: a confidence level between characters in the target text and the reference text, or a confidence level between characters in the reference text and the target text; the confidence level between characters in the target text and the reference text is determined by performing a similarity comparison between the character features of each character in the target text and the character features of the reference text characters; The first generating module is used to generate an evaluation result of the target text based on the mapping relationship.

9. A text evaluation device, characterized in that: The text evaluation device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instructions in the memory to enable the text evaluation device to execute the text evaluation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the text evaluation method according to any one of claims 1 to 7 is implemented.

Images