Method, device and apparatus for text generation quality assessment

Abstract

The present application relates to the field of artificial intelligence technology, and discloses a method for text generation quality assessment, including: obtaining a reference text and a generated text, wherein the generated text is obtained based on the reference text; inputting the reference text and the generated text into a preset evaluation model to obtain evaluation indicators; the evaluation model is obtained based on sample text with a topic similarity label and a generated sentence recognition label; and the text generation quality is assessed based on the evaluation indicators. Compared with the current evaluation method, the method for text generation quality assessment of the present application, in addition to evaluating the topic similarity between the reference text and the corresponding generated text, also considers whether the generated text is easily identified as model-generated text, thereby improving the reliability of text generation quality assessment and improving the reliability of text generation quality assessment. The present application also discloses a device and equipment for text generation quality assessment.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and to a method, device and apparatus for text generation quality assessment. Background Art

[0002] At present, text generation is an important research field in natural language processing technology. Due to its wide application scenarios, research on text generation is endless. A major bottleneck of text generation is how to formulate a reasonable evaluation index to objectively and accurately evaluate the quality of generated text.

[0003] In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related art:

[0004] Currently, most text generation quality assessments only focus on the similarity of semantic topics between the reference text and the corresponding generated text, without paying attention to whether the generated text can be easily identified as model-generated text. The evaluation indicators are single and the reliability of the evaluation results is low. Summary of the invention

[0005] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0006] The embodiments of the present disclosure provide a method, an apparatus and a device for text generation quality assessment, which can improve the reliability of text generation quality assessment.

[0007] In some embodiments, the method comprises:

[0008] Obtain reference text and generated text, where the generated text is obtained based on the reference text;

[0009] Input the reference text and the generated text into a preset evaluation model to obtain evaluation indicators; the evaluation model is obtained based on sample text with topic similarity labels and generated sentence recognition labels;

[0010] Evaluate the quality of text generation based on evaluation indicators.

[0011] In some embodiments, the device includes a processor and a memory storing program instructions, and the processor is configured to perform the above-mentioned method for text generation quality assessment when executing the program instructions.

[0012] In some embodiments, the device includes the above-mentioned apparatus for text generation quality assessment.

[0013] The text generation quality evaluation method, apparatus and device provided by the embodiments of the present disclosure can achieve the following technical effects: by training an evaluation model with sample texts having topic similarity labels and generated sentence recognition labels, the trained evaluation model can comprehensively evaluate the text generation quality through two aspects: topic similarity and generated sentence recognition. Compared with the current evaluation method, in addition to evaluating the topic similarity between the reference text and the corresponding generated text, it also considers whether the generated text is easy to be identified as model-generated text, thereby improving the reliability of text generation quality evaluation.

[0014] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0016] Figure 1 is a schematic diagram of a method for text generation quality provided by an embodiment of the present disclosure;

[0017] Figure 2 It is a schematic diagram of a device for improving text generation quality provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0019] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0020] Unless otherwise stated, the term "plurality" means two or more.

[0021] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0022] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0023] Combination Figure 1 As shown, the present disclosure provides a method for text generation quality assessment, including:

[0024] Step S101, obtaining a reference text and a generated text, wherein the generated text is obtained based on the reference text;

[0025] Step S102, inputting the reference text and the generated text into a preset evaluation model to obtain evaluation indicators; the evaluation model is obtained based on sample text with topic similarity labels and generated sentence recognition labels;

[0026] Step S103, evaluating the quality of text generation according to the evaluation index.

[0027] A method for evaluating text generation quality according to an embodiment of the present disclosure is adopted. An evaluation model is trained by using sample texts with topic similarity labels and generated sentence recognition labels. The trained evaluation model can comprehensively evaluate the text generation quality through two aspects: topic similarity and generated sentence recognition. Compared with the current evaluation method, in addition to evaluating the topic similarity between the reference text and the corresponding generated text, it also considers whether the generated text is easy to be identified as model-generated text, thereby improving the reliability of text generation quality evaluation.

[0028] Optionally, an evaluation model is obtained based on sample text with topic similarity labels and generated sentence recognition labels, including: obtaining sample text; obtaining topic similarity labels for the sample text; obtaining generated sentence recognition labels for the sample text; and training a preset neural network model using sample text with topic similarity labels and generated sentence recognition labels to obtain an evaluation model.

[0029] In this way, an evaluation model is obtained through collaborative training of sample texts with topic similarity labels and generated sentence recognition labels, so that the trained evaluation model can comprehensively evaluate the text generation quality through two aspects: topic similarity and generated sentence recognition, thereby improving the reliability of text generation quality evaluation.

[0030] Optionally, the neural network model includes a BERT (Bidirectional Encoder Representation from Transformers) pre-trained model. In some embodiments, the neural network model is trained using sample texts with topic similarity labels and generated sentence recognition labels to obtain an evaluation model. Compared with traditional statistical evaluation models, the evaluation model based on neural network model training can analyze the topic features and structural features of the text, evaluate the topic similarity of the text by analyzing the topic features and structural features, and the evaluation of the topic similarity between texts is more reliable.

[0031] Optionally, the sample text includes a first text pair and a second text pair; obtaining the sample text includes: obtaining a reference sample text and a generated sample text, wherein the generated sample text is generated based on the reference sample text; combining the reference sample text with the generated sample text corresponding to the reference sample text to obtain a first text pair; and combining two different reference sample texts to obtain a second text pair.

[0032] Optionally, obtaining the topic similarity label of the sample text includes: obtaining a first similarity between the reference sample text and the generated sample text in the first text pair, and obtaining the topic similarity label of the first text pair according to the first similarity; obtaining a second similarity between the reference sample texts in the second text pair, and obtaining the topic similarity label of the second text pair according to the second similarity. In this way, the sample text has a topic similarity label, and the evaluation model trained based on the sample text with the topic similarity label can generate a topic similarity index based on the topic similarity label, and the topic similarity between the texts is evaluated according to the topic similarity index, thereby improving the accuracy of the evaluation model in evaluating the quality of text generation.

[0033] Optionally, obtaining a first similarity between a reference sample text and a generated sample text in a first text pair includes: inputting the reference sample text and the generated sample text into a preset topic model for topic analysis, obtaining a first probability distribution vector and a second probability distribution vector corresponding to the reference sample text and the generated sample text on a number of preset topics, respectively; obtaining a first KL distance (Kullback-Leibler Divergence) between the first probability distribution vector and the second probability distribution vector, and determining the first KL distance as the first similarity.

[0034] Optionally, by calculating Obtain the first KL distance between the first probability distribution vector and the second probability distribution vector, where p(x i ) is the first probability distribution vector, q(x i ) is the second probability distribution vector, D KL (p||q) is the first probability distribution vector p(x i) and the second probability distribution vector q(x i ), N is the first probability distribution vector p(x i ) and the second probability distribution vector q(x i ), where N is a positive integer.

[0035] Optionally, the first KL distance D KL The smaller the value of (p||q), the smaller the first probability distribution vector p(x i ) and the second probability distribution vector q(x i ) the higher the similarity between them.

[0036] Optionally, obtaining the second similarity between the reference sample texts in the second text pair includes: inputting the two reference sample texts into a preset topic model for topic analysis to obtain a third probability distribution vector and a fourth probability distribution vector corresponding to the two reference sample texts on a number of preset topics respectively; obtaining a second KL distance between the third probability distribution vector and the fourth probability distribution vector; and determining the second KL distance as the second similarity.

[0037] Optionally, by calculating Obtain the second KL distance between the third probability distribution vector and the fourth probability distribution vector, where p'(x i ) is the third probability distribution vector, q'(x i ) is the fourth probability distribution vector, D' KL (p'||q') is the third probability distribution vector p'(x i ) and the fourth probability distribution vector q'(x i ), N' is the third probability distribution vector p'(x i ) and the fourth probability distribution vector q'(x i ), N' is a positive integer.

[0038] Optionally, the second KL distance D' KL The smaller the value of (p'||q'), the greater the probability distribution vector p'(x i ) and the fourth probability distribution vector q'(x i ) the higher the similarity between them.

[0039] Optionally, obtaining a topic similarity label for the first text pair based on the first similarity includes: when the first similarity satisfies a first preset condition, determining the topic similarity label for the first text pair as topic similarity; and, when the first similarity does not satisfy the first preset condition, determining the topic similarity label for the first text pair as topic dissimilarity.

[0040] Optionally, when the first similarity satisfies a first preset condition, the topic similarity label of the first text pair is determined as topic similarity, including: when the first similarity of the first text pair is less than or equal to a preset first threshold, the topics of the reference sample text and the generated text sample in the first text pair are similar, and the topic similarity label of the first text pair is determined as topic similarity.

[0041] Optionally, when the first similarity does not satisfy a first preset condition, the topic similarity label of the first text pair is determined as topic dissimilarity, including: when the first similarity of the first text pair is greater than a preset second threshold, the topics of the reference sample text and the generated text sample in the first text pair are dissimilar, and the topic similarity label of the first text pair is determined as topic dissimilarity.

[0042] Optionally, obtaining a topic similarity label for the second text pair based on the second similarity includes: determining the topic similarity label for the second text pair as topic similarity when the second similarity satisfies a second preset condition; and determining the topic similarity label for the second text pair as topic dissimilarity when the second similarity does not satisfy the second preset condition.

[0043] Optionally, when the second similarity satisfies a second preset condition, the topic similarity label of the second text pair is determined as topic similarity, including: when the second similarity of the second text pair is less than or equal to a preset third threshold, the topics between the reference sample texts in the second text pair are similar, and the topic similarity label of the second text pair is determined as topic similarity.

[0044] Optionally, when the second similarity does not satisfy the second preset condition, the topic similarity label of the second text pair is determined as topic dissimilarity, including: when the second similarity of the second text pair is greater than a preset fourth threshold, the topics between the reference sample texts in the second text pair are dissimilar, and the topic similarity label of the second text pair is determined as topic dissimilarity.

[0045] Optionally, obtaining the generated sentence identification label of the sample text includes: when the sample text includes the generated sample text, determining the generated sentence identification label of the sample text as having the generated sample text; when the sample text does not include the generated sample text, determining the generated sentence identification label of the sample text as not having the generated sentence. In this way, the sample text has the generated sentence identification label, and the evaluation model trained based on the sample text with the generated sentence identification label can output the generated sentence recognition index, and the possibility of the evaluation text being identified as the model-generated text is improved according to the generated sentence recognition index, thereby improving the accuracy of the evaluation model in evaluating the text generation quality.

[0046] Optionally, a preset neural network model is trained using sample texts with topic similarity labels and generated sentence recognition labels to obtain an evaluation model, including: obtaining a first loss value corresponding to the topic similarity label in the trained neural network model; obtaining a second loss value corresponding to the generated sentence recognition label in the trained neural network model; and calculating L all =L dist +λL topic Get the overall loss value, where L all is the overall loss value, L topic is the first loss value, L dist is the second loss value, λ is a hyperparameter, λ≤1. Optionally, the larger λ is, the greater the influence of the topic evaluation index corresponding to the topic similarity label generated by the evaluation model on the evaluation index when evaluating the text generation quality.

[0047] Optionally, obtaining a first loss value corresponding to a topic similarity label in a trained neural network model includes: passing a reference sample text in a first text pair through a trained neural network model to obtain a first sample hidden layer output vector; passing a generated sample text in the first text pair through a trained neural network model to obtain a second sample hidden layer output vector; passing different reference sample texts in the second text pair through a trained neural network model to obtain a third sample hidden layer output vector and a fourth sample hidden layer output vector, respectively; taking an arithmetic mean of a combination of the first sample hidden layer output vector and the third sample hidden layer output vector to obtain a first sample arithmetic mean; and performing arithmetic calculation on the generated sample text in the first text pair. The arithmetic mean of the combination of the hidden layer output vector and the fourth sample hidden layer output vector is taken to obtain the second sample arithmetic mean; the first sample arithmetic mean is transformed to the output dimension corresponding to the topic similarity label through a multi-layer perceptron (Muti-LayerPerception, MLP) to obtain the fifth sample hidden layer output vector; the second sample arithmetic mean is transformed to the output dimension corresponding to the topic similarity label through a multi-layer perceptron MLP to obtain the sixth sample hidden layer output vector; the fifth sample hidden layer output vector and the sixth sample hidden layer output vector are concatenated to obtain a sample concatenation vector; the sample concatenation vector is linearly transformed through a fully connected layer (Fully Connected Layer, FC) to obtain the sample topic similarity feature corresponding to the sample concatenation vector; the sample topic similarity feature is binary classified through a SIGMOID function to obtain a sample topic similarity function corresponding to the topic similarity label; the loss of the sample topic similarity function is calculated according to a cross entropy loss function to obtain a first loss value.

[0048] Optionally, obtaining a second loss value corresponding to a generated sentence recognition label in a trained neural network includes: obtaining a sample semantic representation symbol generated by the trained neural network in an input layer; passing the semantic representation symbol through the trained neural network to obtain a sample semantic output vector; transforming the sample semantic output vector to an output dimension corresponding to the generated sentence recognition label through a multi-layer perceptron MLP to obtain a sample semantic vector; performing binary classification processing on the sample semantic vector through a SIGMIOD function to obtain a sample generated sentence recognition function corresponding to the generated sentence recognition label; calculating the loss of the sample generated sentence recognition function according to a cross entropy loss function to obtain a second loss value.

[0049] Optionally, the preset neural network model is trained using sample texts with topic similarity labels and generated sentence recognition labels to obtain an evaluation model, including: inputting the sample texts with topic similarity labels and generated sentence recognition labels into the preset neural network model for training, recording the overall loss value of the training model in each preset cycle; obtaining the lowest value among the recorded overall loss values; when the overall loss value of the training model for M consecutive preset cycles is not lower than the lowest value among the overall loss values, it is determined that the accuracy of the neural network model is no longer improved; stopping the model training, and determining the trained model as the evaluation model; wherein M is a positive integer. Optionally, M is greater than or equal to 10.

[0050] Optionally, the reference text and the generated text are input into a preset evaluation model to obtain an evaluation index, including: passing the reference text through the evaluation model to obtain a first hidden layer output vector; passing the generated text through the evaluation model to obtain a second hidden layer output vector; obtaining the semantic representation symbol generated by the evaluation model at the input layer; passing the semantic representation symbol through the evaluation model to obtain a semantic output vector; taking the arithmetic mean of the first hidden layer output vector to obtain a first arithmetic mean; taking the arithmetic mean of the second hidden layer output vector to obtain a second arithmetic mean; transforming the first arithmetic mean to an output dimension corresponding to a topic similarity label through a multi-layer perceptron MLP to obtain a third hidden layer output vector; transforming the second arithmetic mean to an output dimension corresponding to a topic similarity label through a multi-layer perceptron MLP to obtain a fourth hidden layer output vector; splicing the third hidden layer output vector and the fourth hidden layer output vector to obtain a spliced ​​vector; The concatenated vector is linearly transformed through the fully connected layer FC to obtain the topic similarity feature corresponding to the concatenated vector; the topic similarity feature is binary-classified through the SIGMOID function to obtain the topic similarity function corresponding to the topic similarity label; the first confidence of the topic similarity function on the positive example corresponding to the topic similarity label is obtained; the first confidence is determined as the topic similarity index; the semantic output vector is transformed to the output dimension corresponding to the generated sentence recognition label through the multi-layer perceptron MLP to obtain the semantic vector; the semantic vector is binary-classified through the SIGMIOD function to obtain the generated sentence recognition function corresponding to the generated sentence recognition label; the second confidence of the generated sentence recognition function on the positive example corresponding to the generated sentence recognition label is obtained; the second confidence is determined as the generated sentence recognition index; the evaluation index is obtained according to the topic similarity index and the generated sentence recognition index.

[0051] In one embodiment, a reference text and a generated text are input into an evaluation model based on a BERT pre-trained model, which generates input data at an input layer, and the input data includes a [CLS] symbol, a reference text, and a generated text, wherein the [CLS] symbol represents the semantic features of the reference text and the generated text; a [CLS] output vector after the [CLS] symbol passes through the evaluation model is obtained, and the [CLS] output vector is determined as a semantic output vector.

[0052] Optionally, obtaining the evaluation index according to the topic similarity index and the generated sentence recognition index includes: obtaining the evaluation index by multiplying the topic similarity index and the generated sentence recognition index.

[0053] Optionally, the text generation quality is evaluated according to the evaluation index, including: when the evaluation index is greater than or equal to a first set threshold, the text generation quality is determined to be excellent; when the evaluation index is greater than or equal to a second set threshold and less than the first set threshold, the text generation quality is determined to be good; when the evaluation index is less than the second set threshold, the text generation quality is determined to be poor; wherein the second set threshold is less than the first set threshold. Optionally, the first set threshold is 0.7. Optionally, the second set threshold is 0.4.

[0054] In one embodiment, the reference text is "This is a mobile phone with a very high cost performance. I love it so much, the price is so affordable, the cost performance is super high, and it is suitable for everyone. Especially for the elderly, this page is clear, simple, and elegant, and the elderly are not dazzled. The screen is clear and smooth, sliding and unlocking are convenient, and the sound quality is excellent. The tone is pleasant. The photo taking is excellent, and it is the best among the thousand-yuan phones." According to the reference text, a generated text is obtained by using the text generation model to be evaluated. The generated text is "This mobile phone has a really high cost performance. I love it so much, the cost performance is super high, and it is suitable for use in all scenarios, especially for the elderly. The screen page display is clear, the sliding is smooth, the unlocking is fast, and the sound quality and tone are excellent. The photo taking is the best, and it is the best among the thousand-yuan phones." The reference text and the generated text are input into the evaluation model; the topic similarity of the topic similarity label is determined as a positive example of the topic similarity function, and the higher the topic similarity between the reference text and the generated text, the higher the value of the topic similarity index; the topic similarity index output by the evaluation model is 0.852; the generated sentence without the generated sentence identification label is determined as a positive example of the generated sentence identification function, and the higher the possibility that the generated text is identified as the text generated by the model, the higher the value of the generated sentence identification index; the generated sentence identification index output by the evaluation model is 0.831; the evaluation index obtained by multiplying the topic similarity index and the generated sentence identification index is 0.708; then the text generation quality of the text generation model to be evaluated is determined to be excellent.

[0055] Combination Figure 2 As shown, an embodiment of the present disclosure provides a device for text generation quality assessment, including a processor (processor) 100 and a memory (memory) 101 storing program instructions. Optionally, the device may also include a communication interface (Communication Interface) 102 and a bus 103. The processor 100, the communication interface 102, and the memory 101 may communicate with each other through the bus 103. The communication interface 102 may be used for information transmission. The processor 100 may call the program instructions in the memory 101 to execute the method for text generation quality assessment of the above embodiment.

[0056] In addition, the program instructions in the above-mentioned memory 101 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0057] The memory 101 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 100 executes the function application and data processing by running the program instructions / modules stored in the memory 101, that is, the method for text generation quality assessment in the above embodiment is implemented.

[0058] The memory 101 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 101 may include a high-speed random access memory and may also include a non-volatile memory.

[0059] By adopting the device for text generation quality assessment provided by the embodiment of the present disclosure, an assessment model is trained by using sample texts with topic similarity labels and generated sentence recognition labels, so that the trained assessment model can comprehensively assess the text generation quality through two aspects: topic similarity and generated sentence recognition. Compared with the current assessment method, in addition to assessing the topic similarity between the reference text and the corresponding generated text, it also considers whether the generated text is easy to be identified as model-generated text, thereby improving the reliability of text generation quality assessment.

[0060] The disclosed embodiment provides a device, comprising the above-mentioned apparatus for text generation quality assessment, which trains an assessment model by using sample texts with topic similarity labels and generated sentence recognition labels, so that the trained assessment model can comprehensively assess the text generation quality through two aspects: topic similarity and generated sentence recognition. Compared with the current assessment method, in addition to assessing the topic similarity between a reference text and a corresponding generated text, it also considers whether the generated text is easy to be identified as model-generated text, thereby improving the reliability of text generation quality assessment.

[0061] Optionally, the device includes a computer, a smart phone, a tablet computer, etc.

[0062] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned method for text generation quality assessment.

[0063] An embodiment of the present disclosure provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned method for text generation quality assessment.

[0064] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0065] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0066] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0067] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0068] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0069] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for evaluating the quality of text generation, characterized in that: include: Acquire a reference text and a generated text, wherein the generated text is obtained based on the reference text; Inputting the reference text and the generated text into a preset evaluation model to obtain evaluation indicators; The evaluation model is obtained based on sample text with topic similarity labels and generated sentence identification labels; Evaluate the quality of text generation according to the evaluation indicators; Evaluate the model based on sample text with topic similarity labels and generated sentence identification labels, including: Get sample text; Obtaining a topic similarity label of the sample text; obtaining a generated sentence identification label of the sample text; Using the sample text with the topic similarity label and the generated sentence identification label to train a preset neural network model to obtain an evaluation model; The sample text includes a first text pair and a second text pair; and obtaining the sample text includes: Acquire a reference sample text and generate a sample text, wherein the generated sample text is generated according to the reference sample text; Combining the reference sample text with the generated sample text corresponding to the reference sample text to obtain a first text pair; combining two different reference sample texts to obtain a second text pair; The step of obtaining a generated sentence identification label of the sample text includes: In the case where the sample text includes generated sample text, determining the generated sentence identification label of the sample text as having generated sample text; In the case where the sample text does not include generated sample text, determining the generated sentence identification label of the sample text as having no generated sentence; The step of inputting the reference text and the generated text into a preset evaluation model to obtain evaluation indicators includes: Passing the reference text through the evaluation model to obtain a first hidden layer output vector; Passing the generated text through the evaluation model to obtain a second hidden layer output vector; Obtaining the semantic representation symbol generated by the evaluation model at the input layer; passing the semantic representation symbol through the evaluation model to obtain a semantic output vector; Taking the arithmetic mean of the first hidden layer output vector to obtain a first arithmetic mean; taking the arithmetic mean of the second hidden layer output vector to obtain a second arithmetic mean; transforming the first arithmetic mean to the output dimension corresponding to the topic similarity label through a multi-layer perceptron MLP to obtain a third hidden layer output vector; transforming the second arithmetic mean to the output dimension corresponding to the topic similarity label through a multi-layer perceptron MLP to obtain a fourth hidden layer output vector; concatenating the third hidden layer output vector and the fourth hidden layer output vector to obtain a concatenated vector; performing a linear transformation on the concatenated vector through a fully connected layer FC to obtain a topic similarity feature corresponding to the concatenated vector; performing a binary classification process on the topic similarity feature through a SIGMOID function to obtain a topic similarity function corresponding to the topic similarity label; obtaining a first confidence of the topic similarity function on the positive example corresponding to the topic similarity label; determining the first confidence as a topic similarity index; The semantic output vector is transformed into the output dimension corresponding to the generated sentence identification label by a multi-layer perceptron MLP to obtain a semantic vector; the semantic vector is subjected to binary classification processing by a SIGMIOD function to obtain a generated sentence identification function corresponding to the generated sentence identification label; a second confidence of the generated sentence identification function on the positive example corresponding to the generated sentence identification label is obtained; and the second confidence is determined as a generated sentence identification index; An evaluation index is obtained according to the topic similarity index and the generated sentence recognition index.

2. The method according to claim 1, characterized in that The step of obtaining the topic similarity label of the sample text includes: Obtaining a first similarity between the reference sample text and the generated sample text in the first text pair, and obtaining a topic similarity label of the first text pair according to the first similarity; A second similarity between the reference sample texts in the second text pair is obtained, and a topic similarity label of the second text pair is obtained according to the second similarity.

3. The method according to claim 2, characterized in that The step of obtaining a topic similarity label of the first text pair according to the first similarity includes: In the case where the first similarity satisfies a first preset condition, determining the topic similarity label of the first text pair as topic similarity; and, When the first similarity does not satisfy a first preset condition, the topic similarity label of the first text pair is determined as topic dissimilarity.

4. The method according to claim 2, characterized in that The step of obtaining a topic similarity label of the second text pair according to the second similarity includes: In the case where the second similarity satisfies a second preset condition, determining the topic similarity label of the second text pair as topic similarity; and, When the second similarity does not satisfy a second preset condition, the topic similarity label of the second text pair is determined as topic dissimilarity.

5. A device for evaluating the quality of text generation, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to perform the method for text generation quality assessment according to any one of claims 1 to 4 when executing program instructions.

6. A device, characterized in that: Comprising the apparatus for text generation quality assessment as described in claim 5.

Images