Evaluation methods, devices, electronic equipment, and storage media for the continuation of the text

Abstract

This disclosure provides a method for evaluating a continuation text, which may include: acquiring a target text containing the preceding text and the continuation text; evaluating the quality of the continuation text across multiple dimensions using a continuation text evaluation model, and obtaining a dimension score corresponding to each dimension, wherein the dimensions include at least fluency, relevance, diversity, and logicality; and integrating the multiple dimension scores to generate a comprehensive score, wherein the comprehensive score is used to characterize the overall fit of the continuation text to each dimension. This disclosure also provides a device, electronic device, and storage medium for evaluating a continuation text.

Description

Technical Field

[0001] This disclosure relates to a method, apparatus, electronic device, and storage medium for evaluating a continuation. Background Technology

[0002] When creating text, creators may encounter creative blocks or other reasons that prevent them from producing complete text content. An incomplete text renders the initial creative work meaningless. Alternatively, readers may feel unsatisfied with the ending and crave further content to satisfy their reading interest and imagination. Therefore, a continuation writing model is often used to supplement or extend the completed parts of the text, making it readable and complete.

[0003] However, the quality of current continuation writing models varies greatly, making it difficult to meet people's needs for supplementing and extending text. Furthermore, the optimization of a continuation writing model depends on the evaluation parameters for the continuation content. Related technologies typically evaluate only one aspect of the continuation text, such as fluency or diversity, resulting in a limited evaluation perspective and making it difficult to comprehensively measure the quality of the continuation text. Therefore, one-sided evaluation results cannot generate comprehensive optimization parameters, nor can they identify the weaknesses of the continuation text from multiple evaluation perspectives, making it even more difficult to generate a continuation writing model that balances fluency, relevance, diversity, and logical coherence. Summary of the Invention

[0004] To address at least one of the aforementioned technical problems, this disclosure provides a method, apparatus, electronic device, and storage medium for evaluating a continuation of a work.

[0005] One aspect of this disclosure provides a method for evaluating a sequel, which may include: obtaining a target text containing the preceding text and the sequel; evaluating the quality of the sequel across multiple dimensions using a sequel evaluation model and obtaining a dimension score corresponding to each dimension, wherein the dimensions include at least fluency, relevance, diversity, and logicality; and integrating the multiple dimension scores to generate a comprehensive score, wherein the comprehensive score is used to characterize the overall fit of the sequel to each dimension.

[0006] In some implementations, a continuation evaluation model is used to evaluate the quality of the continuation in multiple dimensions and obtain the corresponding dimension scores for each dimension, including: calculating the confusion ratio and the mean repetition of the continuation, and using the sum of the two as the fluency score.

[0007] In some implementations, calculating the perplexity ratio and the average repetition rate of the continuation text separately includes at least: extracting a continuation text segment with the same number of characters as the preceding text; performing masking processing on each continuation character in the continuation text segment to obtain character gaps corresponding to each character; determining the probability of the continuation character appearing in each character gap using a perplexity model, and calculating the perplexity of the continuation text based on each probability of appearance; and calculating the ratio between the perplexity of the preceding text and the perplexity of the continuation text to obtain the perplexity ratio.

[0008] In some implementations, the perplexity ratio and the mean repetition of the continuation text are calculated separately, including at least: dividing the continuation text into multiple continuation text sentences by using the segmentation number of the continuation text as the segmentation point; calculating the number of repeated phrases or short sentences between two adjacent continuation text sentences using a repetition model; and calculating the mean of all repeated numbers in the continuation text to obtain the mean repetition.

[0009] In some implementations, a continuation evaluation model is used to evaluate the quality of the continuation in multiple dimensions and obtain the corresponding dimension scores for each dimension, including: calculating the internal relevance value of the continuation and the connection relevance value between the continuation and the preceding text, and using the sum of the two as the relevance score.

[0010] In some implementations, calculating the internal relevance value of the continuation text and the connection relevance value between the continuation text and the preceding text separately includes at least: dividing the continuation text into multiple continuation text statements by using the segmentation number of the continuation text as the segmentation point; rearranging each continuation text statement using a sorting model to obtain an arrangement result that can represent the sequential order of each continuation text statement; comparing the arrangement result with the writing order of each continuation text statement in the continuation text, and calculating the matching degree between two adjacent continuation text statements in the arrangement result and the writing order; and calculating the mean of each matching degree to obtain the internal relevance value.

[0011] In some implementations, calculating the internal relevance value of the continuation text and the connection relevance value between the continuation text and the preceding text separately includes at least: extracting the action subjects of each continuation paragraph in the continuation text and the domains of each named entity in the continuation text; calculating the ratio of the number of occurrences of the target subject to the total number of occurrences of the action subjects, and using the ratio as the narrative object score, wherein the target subject is the action subject that appeared in the preceding text; calculating the ratio of the number of named entities corresponding to the conflict domain to the total number of named entities, and using the ratio as the theme score, wherein the conflict domain is the domain that does not overlap with the domains of other named entities in the continuation text; and calculating the sum of the narrative object score and the theme score, and using the sum as the connection relevance value.

[0012] In some implementations, a continuation evaluation model is used to evaluate the quality of the continuation in multiple dimensions and obtain dimensional scores corresponding to each dimension. This includes: dividing the continuation into words to obtain multiple word segments and calculating the word count of each segment; classifying each word segment into categories based on the meaning of the word segments to obtain the number of clusters of each category; and calculating the ratio of the number of clusters to the number of words, and using the ratio as the diversity score of the continuation.

[0013] In some implementations, a continuation evaluation model is used to evaluate the quality of the continuation across multiple dimensions and obtain dimensional scores corresponding to each dimension. This includes: extracting the action subjects and the actions performed by the action subjects in the continuation; constructing a role graph to represent the relationships between each action subject and the actions; identifying contradictory relationships in the role graph and calculating the frequency of occurrence of contradictory relationships, where contradictory relationships are those where the role attributes of the action subject do not match the actions; and calculating the ratio between the frequency of contradictory actions and the total number of relationships, using this ratio as a logical score.

[0014] In some implementations, the method further includes: inputting the preceding text multiple times into the continuation model that generates the continuation text, and determining the continuation similarity of the continuation text output by the continuation model.

[0015] In some implementations, after integrating the scores from multiple dimensions to generate a comprehensive score, the method further includes: optimizing the parameters of the continuation model based on the comprehensive score and the continuation similarity.

[0016] Another aspect of this disclosure provides a device for evaluating a continuation text, which may include: a target text acquisition module for acquiring target text containing the preceding text and the continuation text; a quality evaluation module for evaluating the quality of the continuation text in multiple dimensions using a continuation text evaluation model and obtaining dimension scores corresponding to each dimension, wherein the dimensions include at least fluency, relevance, diversity, and logicality; and a score generation module for integrating the multiple dimension scores to generate a comprehensive score, wherein the comprehensive score is used to characterize the overall fit of the continuation text to each dimension.

[0017] Another aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the evaluation method as described in any of the above embodiments.

[0018] Another aspect of this disclosure provides a readable storage medium storing a computer program adapted for loading by a processor to perform the evaluation method as described in any of the above embodiments. Attached Figure Description

[0019] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0020] Figure 1 This is a flowchart of an evaluation method following an exemplary embodiment of this disclosure.

[0021] Figure 2 This is a block diagram of an evaluation apparatus according to an exemplary embodiment of the present disclosure.

[0022] Explanation of reference numerals in the attached figures

[0023] Evaluation device for 1000 continuations

[0024] 1002 Target Text Acquisition Module

[0025] 1004 Quality Evaluation Module

[0026] 1006 Rating Generation Module

[0027] 1100 bus

[0028] 1200 processor

[0029] 1300 memory

[0030] 1400 Other Circuits Detailed Implementation

[0031] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0032] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0034] The terminology used herein is for the purpose of describing particular embodiments and is not restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0035] Figure 1 This is a flowchart of an evaluation method following an exemplary embodiment of this disclosure. The following will be combined with... Figure 1 The evaluation method S100 for the sequel to this red-opened work is explained in detail.

[0036] Step S102: Obtain the target text containing the preceding and following text.

[0037] Here, "previous text" refers to the text content already completed by the creator, while "continuation text" refers to the continuation text obtained by the continuation writing model based on the text content already completed by the creator.

[0038] The target text can be input into the continuation evaluation model by any user, so that the continuation evaluation model can evaluate the continuation in multiple dimensions and give a dimension score, so as to realize the transformation from qualitative to quantitative evaluation results.

[0039] The continuation writing model is a model that writes continuations based on the preceding text. It is trained with a large number of training samples and can accurately output continuations that match the preceding text in at least four dimensions and conform to linguistic logic.

[0040] The continuation evaluation model assesses continuations across at least four dimensions, examining their writing quality in these areas. Based on the quantitative results provided by the model, continuations can be evaluated from multiple perspectives, and the model's weaknesses can be identified, providing data support for its optimization.

[0041] Step S104: Use the continuation evaluation model to evaluate the quality of the continuation in multiple dimensions and obtain the dimension scores corresponding to each dimension.

[0042] The dimensions include at least fluency, relevance, diversity, and logical coherence. While related technologies typically only provide a one-sided evaluation of sequels, this disclosure enables a multi-dimensional and comprehensive evaluation of sequels, with the evaluation results for each dimension presented to the user using dimensional scores.

[0043] Dimensional scores are used to characterize the quality of a continuation of a story in a specific dimension. Typically, when evaluating the quality of a continuation, only qualitative descriptions such as "good," "average," and "poor" are used. Even within the category of "good" quality, there are levels like "generally good" or "excellent," and using qualitative descriptions makes it difficult to discern the actual differences between continuations at the same level. This disclosure generates corresponding dimensional scores for each dimension of the continuation, providing a clear view of the specific quality of the continuation in a particular dimension and offering data support for optimizing the continuation model.

[0044] Specifically, the implementation process of step S104 includes the analysis of the dimensional scores for each dimension, and the method for obtaining the dimensional scores for each dimension is different. The specific methods for obtaining the dimensional scores will be described later.

[0045] Step S106: Integrate the scores of multiple dimensions to generate a comprehensive score, which is used to characterize the overall fit of the continuation to each dimension.

[0046] The overall score is the sum of the scores for each dimension multiplied by their corresponding dimension weights. The specific values ​​of the dimension weights can be set according to the evaluation proportion of each dimension in the continuation evaluation model, and are defined by the user according to their needs; there are no restrictions here.

[0047] When the overall score of the continuation is less than the overall threshold, it indicates that the continuation model needs parameter optimization. In this case, the weaknesses of the continuation model can be identified based on the scores of each dimension for targeted optimization. If the overall score of the continuation is greater than or equal to the overall threshold, it indicates that the continuation generated by the model fits well with each dimension, and no optimization of the continuation model is necessary.

[0048] The following section will elaborate on the calculation process of the dimensional scores for each dimension in step S102.

[0049] In some implementations, calculating the fluency score of the continuation text in the fluency dimension may include: calculating the confusion ratio and the mean repetition of the continuation text respectively, and using the sum of the two as the fluency score.

[0050] The fluency score is used to evaluate the degree of fit between the continuation and conventional expression habits, and also to examine whether there are many repeated phrases and short sentences in the continuation.

[0051] Confusion level refers to the degree of confusion in the text content of a continuation. It is a specific parameter representing the degree of fit between the text and the writing style. The higher the confusion level, the lower the degree of fit, and consequently, the lower the fluency.

[0052] Based on this, the perplexity ratio and the mean repetition rate of the continuation text should be calculated separately.

[0053] Specifically, the calculation method of the perplexity ratio includes: extracting a continuation segment with the same number of characters as the preceding text based on the number of characters in the preceding text; performing masking processing on each continuation character in the continuation segment to obtain the character space corresponding to each character; using the perplexity model to determine the probability of the continuation character appearing in each character space, and calculating the perplexity of the continuation based on each probability of appearance; and calculating the ratio between the perplexity of the preceding text and the perplexity of the continuation to obtain the perplexity ratio.

[0054] More specifically, since the continuation lacks a reference text, the current level of perplexity cannot be calculated based on the reference text. Therefore, the perplexity needs to be converted into a quantitative value. To avoid the cost of automatically constructing a suitable reference text, we make the assumption that the perplexity of the preceding and following texts within the same text should be similar (i.e., fluency). Therefore, we extract a continuation segment with the same number of characters (i.e., text length) as the continuation segment, calculate the perplexity of the continuation segment of both the preceding and following texts, and use the ratio of the two as the corresponding score for perplexity, i.e., the perplexity ratio.

[0055] The perplexity is usually obtained by the perplexity calculation formula, which can be expressed as:

[0056]

[0057] The perplexity formula can be simplified to:

[0058]

[0059] Where S represents any sentence, N represents the number of characters in sentence S (i.e., the length of sentence S), and w i Represents the i-th character (i can be 1 to N), P(w i P(w) represents the probability of the i-th character appearing. i | w1w2…w i-1The expression represents the probability of the i-th character appearing based on the first to (i-1)th characters. Since continuation models typically generate text based on GPT (Generative Pre-Training) models that predict words with high probabilities through autoregression, the BERT (Bidirectional Encoder Representations from Transformer) model is usually used to calculate perplexity. In other words, each continuation character in the continuation segment is masked to generate corresponding empty spaces. The BERT model predicts the character corresponding to each empty space based on the context and compares the predicted character with the continuation character to determine the probability of the continuation character appearing in the empty space. If the probability of the continuation character appearing in the same empty space is high, it proves that the empty space corresponds to the continuation character, indicating that the perplexity of the continuation is low when the continuation character is in that empty space.

[0060] Specifically, the mean repetition rate is calculated as follows: the paragraph number of the continuation text is used as the dividing point to divide the text into multiple continuation text sentences; the repetition rate model is used to calculate the number of repeated phrases or short sentences between two adjacent continuation text sentences; and the mean of all repeated sentences in the continuation text is calculated to obtain the mean repetition rate.

[0061] The segmentation number can be a punctuation mark or other segmentation marker in the continuation text.

[0062] Specifically, the repetition model is usually a self-bleu (self-bilingual evaluation understudy) model. It uses the repetition model to calculate the pairwise repetition of multiple continuation sentences after sentence segmentation, determines the average repetition value (i.e. the number of repetitions of repeated phrases and short sentences), and then calculates the average of all repetitions as the repetition mean.

[0063] In some implementations, calculating the relevance score of the continuation text in the relevance dimension may include: calculating the internal relevance value of the continuation text and the connection relevance value between the continuation text and the preceding text, and using the sum of the two as the relevance score.

[0064] Among them, the internal relevance value represents the degree of relevance within the continuation text. The coherence relevance value represents the relevance between the continuation text and the preceding text.

[0065] Since the calculation methods for internal correlation values ​​and coherence correlation values ​​are different, they will be explained separately below.

[0066] Specifically, calculating the internal relevance value of the continuation text may include: dividing the text into multiple continuation sentences by using the segmentation number as the dividing point; rearranging each continuation sentence using a sorting model to obtain an arrangement result that can represent the sequential order of each continuation sentence; comparing the arrangement result with the writing order of each continuation sentence in the continuation text, and calculating the matching degree between two adjacent continuation sentences in the arrangement result and the writing order; and calculating the mean of each matching degree to obtain the internal relevance value.

[0067] In other words, calculating the matching degree between two adjacent continuation sentences and the continuation order essentially involves using the BERT model to determine whether the connection between two adjacent continuation sentences in the continuation is appropriate. Based on this, the original continuation sentences are shuffled and reordered using a sorting model (i.e., the BERT model) to determine whether the rearranged order is consistent with the continuation order. If they are consistent, it proves that the arrangement of adjacent sentences has a high matching degree. Then, by integrating the matching degrees of each adjacent sentence and calculating their average, the internal correlation value can be obtained.

[0068] Specifically, calculating the connection relevance value between the continuation and the preceding text may include: extracting the action subjects of each continuation paragraph in the continuation and the domains of each named entity in the continuation; calculating the ratio of the number of occurrences of the target subject to the total number of occurrences of the action subjects, and using the ratio as the narrative object score, where the target subject is the action subject that appeared in the preceding text; calculating the ratio of the number of named entities corresponding to the conflict domain to the total number of named entities, and using the ratio as the theme score, where the conflict domain is the domain that does not overlap with the domains of other named entities in the continuation; and calculating the sum of the narrative object score and the theme score, and using the sum as the connection relevance value.

[0069] Among them, the subject of action refers to the protagonist of each continuation paragraph in the sequel. Each subject of action has corresponding character attributes. For example, when the subject of action is a 13-year-old boy, his character attributes are student and boy.

[0070] Named entities refer to words in a continuation paragraph that have a relatively clear domain indication.

[0071] More specifically, the preceding text usually describes one or several action subjects (i.e., target subjects). To ensure consistency between the continuation and the preceding text in terms of the subject of the narrative, the proportion of the target subject in the continuation can be determined by comparing the number of times the target subject appears in each paragraph of the continuation with the total number of times all action subjects appear in the continuation. A larger proportion indicates a higher degree of consistency between the continuation and the preceding text in terms of the subject of the narrative, while a smaller proportion indicates a lower degree of consistency.

[0072] Furthermore, since some named entities have specific domain characteristics and only appear in fixed domains, if a large number of named entities in the continuation sentences have no overlap in domain, it proves that named entities have shifted during the continuation process. Based on this, we can examine the ratio of named entities with domain conflicts to all named entities in the continuation. A smaller ratio indicates a smaller shift, and thus a higher topic score for the continuation; a larger ratio indicates a larger shift, and thus a lower topic score for the continuation.

[0073] In some implementations, calculating the diversity score of the continuation text in the diversity dimension may include: dividing the continuation text into words to obtain multiple word segments and calculating the word count of each word segment; classifying each word segment into categories according to the meaning of each word segment to obtain the number of clusters of each category; and calculating the ratio of the number of clusters to the number of words, and using the ratio as the diversity score of the continuation text.

[0074] In some implementations, calculating the logicality score of the continuation text in the logicality dimension may include: extracting the action subjects and the actions performed by the action subjects in the continuation text; constructing a role graph to represent the relationship between each action subject and the action; identifying contradictory relationships in the role graph and calculating the number of occurrences of contradictory relationships, wherein a contradictory relationship is a relationship where the role attributes of the action subject do not match the action; and calculating the ratio between the number of occurrences of contradictory actions and the total number of relationships, and using the ratio as the logicality score.

[0075] Specifically, logical consistency primarily examines whether the continuation contains factual contradictions, which are more apparent during automatic continuation writing. In the creative process, the author typically focuses on the actions of the main character; that is, the main character will only perform actions consistent with their role attributes. For example, a character with the attributes of a student or teenager will only perform actions such as going to school, attending training classes, or playing. However, after writing a certain length of text, the continuation model often ignores the main character and their attributes from the previous text, writing actions that don't match their attributes. For instance, a character with the attributes of a student or teenager might mistakenly be written as performing actions like going to work, which logically contradicts the previous text.

[0076] Based on this, this disclosure proposes to use a character graph to associate each action subject in the continuation with its corresponding action, thereby obtaining the relationship between the action subject and the action. Furthermore, by determining the frequency of contradictory relationships where the action subject's character attributes do not match the action, and calculating the ratio of the frequency of contradictory relationships to the total number of relationships in the character graph, a logicality score can be obtained.

[0077] In some implementations, the evaluation method S100 for the continuation text further includes: inputting the preceding text multiple times into the continuation writing model that generates the continuation text, and judging the continuation similarity of the continuation text output by the continuation writing model.

[0078] Diversity primarily considers whether the continuation model possesses diverse expressions; therefore, this step mainly evaluates the diversity of the continuation model. Specifically, the same preceding text is input into the continuation model multiple times, and the consistency of the resulting continuation text is examined. If they are consistent or have minimal differences, it indicates poor expressive diversity in the continuation model; if they are inconsistent, it indicates rich expressive diversity. Continuation similarity is not used to characterize the diversity of the continuation text, but it can serve as data support for optimizing the continuation model.

[0079] In some implementations, after step S106, the method further includes: optimizing the parameters of the continuation model based on the comprehensive score and the continuation similarity.

[0080] In other words, if the overall score of the continuation is less than the overall threshold, it indicates that the continuation model needs parameter optimization. The weaknesses of the continuation model can then be identified based on the scores of each dimension for targeted optimization. If the overall score of the continuation is greater than or equal to the overall threshold, it means that the continuation generated by the model fits well with each dimension, and no optimization of the model is necessary.

[0081] The evaluation method for continuation articles proposed in this disclosure evaluates continuation articles from multiple dimensions and gives dimensional scores for each dimension. This can quantitatively and intuitively demonstrate the quality of continuation writing in each dimension and provide data support for the optimization of continuation writing models.

[0082] Figure 2 This is a block diagram of an evaluation apparatus according to an exemplary embodiment of the present disclosure.

[0083] like Figure 2 As shown, another aspect of this disclosure provides a continuation evaluation device 1000, which may include: a target text acquisition module 1002, used to acquire target text containing the preceding text and the continuation text; a quality evaluation module 1004, used to perform multi-dimensional quality evaluation on the continuation text using a continuation text evaluation model, and obtain dimension scores corresponding to each dimension, wherein the dimensions include at least fluency, relevance, diversity and logicality; and a score generation module 1006, used to integrate the multiple dimension scores to generate a comprehensive score, wherein the comprehensive score is used to characterize the overall fit of the continuation text to each dimension.

[0084] The various modules in the continuation evaluation device 1000 disclosed herein are proposed to implement the various steps of the continuation evaluation method. The execution principle and steps of each module can be referred to the previous text and will not be repeated here.

[0085] The device 1000 may include corresponding units that perform one or more steps in the flowchart described above. Therefore, each or more steps in the flowchart can be performed by a corresponding unit, and the device 1000 may include one or more of these units. A unit may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable storage medium for implementation by a processor, or implemented through some combination thereof.

[0086] This hardware architecture can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and bus design constraints. Bus 1100 will connect various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0087] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but it does not imply that there is only one bus or one type of bus.

[0088] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0089] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0090] The continuation evaluation device proposed in this disclosure evaluates the continuation from multiple dimensions and gives the dimensional score of the continuation in each dimension. It can quantitatively and intuitively show the continuation quality in each dimension and provide data support for the optimization of the continuation model.

[0091] For the purposes of this specification, "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use in or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0092] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0093] Those skilled in the art will understand that all or part of the steps of the methods described above can be implemented by a program instructing related hardware, and the program can be stored in a readable storage medium. When executed, the program includes one or a combination of the steps of the method implementation.

[0094] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0095] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0096] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0097] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method of evaluating a continuation, characterized by, The method comprises: obtaining a target text comprising a preceding text and a continuation text; evaluating the quality of the continuation text in multiple dimensions by using a continuation text evaluation model, and obtaining a dimension score corresponding to each dimension, wherein the dimensions at least include fluency, relevance, diversity and logic, and the diversity is the expression diversity of the continuation text; and integrating the multiple dimension scores to generate a comprehensive score, wherein the comprehensive score is used to represent the overall fit degree of the continuation text to each dimension, wherein the perplexity ratio of the continuation text and the average value of the repetition degree of the continuation text are calculated respectively, and the sum of the two is taken as the fluency score of the fluency; wherein the internal correlation value of the continuation text and the coherence correlation value between the continuation text and the preceding text are calculated respectively, and the sum of the two is taken as the relevance score of the relevance; the calculation of the internal correlation value comprises: dividing the continuation text into multiple continuation sentences by taking the segmentation mark in the continuation text as the segmentation point; rearranging each continuation sentence by using a sorting model to obtain a rearrangement result representing the order of each continuation sentence; comparing the rearrangement result with the writing order of each continuation sentence in the continuation text to calculate the matching degree of the order of two adjacent continuation sentences in the rearrangement result and the writing order of the continuation text; and calculating the average value of each matching degree to obtain the internal correlation value; the calculation of the coherence correlation value comprises: extracting the action subject of each continuation paragraph in the continuation text and the domain of each named entity in the continuation text; calculating the ratio of the number of target subjects in the action subjects to the total number of action subjects, and taking the ratio as the narrative object score, wherein the target subject is the action subject appearing in the preceding text; calculating the ratio of the number of named entities corresponding to the conflict domain to the total number of named entities, and taking the ratio as the theme score, wherein the conflict domain is a domain that does not intersect with other named entities in the continuation text; and calculating the sum of the narrative object score and the theme score, and taking the sum as the coherence correlation value; wherein the action subject in the continuation text and the action performed by the action subject are extracted; a role graph is constructed to represent the association between each action subject and the action; a contradiction relationship is identified in the role graph, and the number of occurrences of the contradiction relationship is calculated, wherein the contradiction relationship is a relationship in which the role attribute of the action subject does not match the action; and the ratio between the number of occurrences of the contradiction relationship and the total number of association relationships is calculated, and the ratio is taken as the logic score.

2. The method of claim 1, wherein The calculation of the perplexity ratio of the continuation text and the average value of the repetition degree of the continuation text at least comprises: According to the number of characters in the preceding text, a continuation text segment with the same number of characters is intercepted in the continuation text; each continuation character in the continuation text segment is masked one by one to obtain a character space corresponding to each character; the appearance probability of each character space appearing the continuation character is determined by using a perplexity model, and the perplexity of the continuation text is calculated according to each appearance probability; and calculating a ratio of the perplexity of the preceding text and the perplexity of the continuation text to obtain a perplexity ratio.

3. The method of claim 1, wherein The perplexity ratio and the repetition average of the continuation text are obtained by at least: dividing the continuation text into multiple continuation sentences by taking the segment label of the continuation text as a segmentation point; calculating the repetition number of repeated phrases or repeated sentences between two adjacent continuation sentences by using a repetition model; and calculating an average of all repetition numbers in the continuation text to obtain the repetition average.

4. The method of claim 1, wherein The quality of the continuation text is evaluated in multiple dimensions by using a continuation evaluation model, and a dimension score corresponding to each dimension is obtained, including: dividing the continuation text into multiple words to obtain multiple word pieces, and calculating the number of words; dividing each word piece into a category according to the meaning of the word piece to obtain the number of clusters in the category; and calculating a ratio of the number of clusters and the number of words, and taking the ratio as a diversity score of the continuation text.

5. An evaluation device for a continuation, characterized by including: a target text acquisition module configured to acquire a target text including a preceding text and a continuation text; a quality evaluation module configured to evaluate the quality of the continuation text in multiple dimensions by using a continuation evaluation model, and obtain a dimension score corresponding to each dimension, wherein the dimensions include at least fluency, relevance, diversity and logic, and the diversity is the expression diversity of the continuation text; and a score generation module configured to integrate multiple dimension scores to generate a comprehensive score, wherein the comprehensive score is used to represent the overall fit degree of the continuation text to each dimension, wherein the quality evaluation module obtains a perplexity ratio and a repetition average of the continuation text, and takes the sum of the two as a fluency score of the fluency. The quality evaluation module calculates an internal correlation value of the continuation text and a coherence correlation value between the continuation text and the preceding text, and takes the sum of the two as a correlation score of the correlation. The calculation of the internal correlation value includes: dividing the continuation text into multiple continuation sentences by taking the segmentation label in the continuation text as a segmentation point; rearranging each continuation sentence by using a ranking model to obtain a rearrangement result representing the order of each continuation sentence; comparing the rearrangement result with the writing order of each continuation sentence in the continuation text to calculate the matching degree of the order of two adjacent continuation sentences in the rearrangement result and the writing order of the continuation text; and obtaining the average of each matching degree to obtain the internal correlation value. The calculation of the coherence correlation value includes: extracting the action subject of each continuation paragraph in the continuation text and the domain of each named entity in the continuation text; calculating the ratio of the number of target subjects in the action subject to the total number of action subjects, and taking the ratio as a narrative object score, wherein the target subject is an action subject appearing in the preceding text; calculating the ratio of the number of named entities corresponding to the conflict domain to the total number of named entities, and taking the ratio as a theme score, wherein the conflict domain is a domain that does not intersect with other named entities in the continuation text; and obtaining the sum of the narrative object score and the theme score, and taking the sum as the coherence correlation value. The method includes: extracting the action subject in the continuation text and the action performed by the action subject; constructing a role graph for representing the association relationship between each action subject and the action; identifying a contradictory relationship in the role graph and calculating the number of occurrences of the contradictory relationship, wherein the contradictory relationship is a relationship in which the role attribute of the action subject does not match the action; and calculating the ratio between the number of occurrences of the contradictory relationship and the total number of association relationships, and taking the ratio as a logical score.

Images