Text quality evaluation method and apparatus

Abstract

This disclosure provides a method and apparatus for evaluating text quality, belonging to the field of network technology. The technical solution provided in this disclosure transforms the problem of appropriately evaluating text into a three-step deterministic process of identifying candidate problem items, verifying them, and obtaining a quality score. Specifically, when identifying candidate problem items in the text, identification is performed from multiple quality evaluation dimensions, and text segments are forcibly output, avoiding a first agent making a generalized judgment. Further verification, including a first and second verification, reduces problems such as illusions and comprehension biases in the first agent. Furthermore, when obtaining the text quality score, no agent is used; instead, it is obtained based on scoring rules and verified target problem items. The degree of deviation is no longer a vague textual concept freely interpreted by the agent, but rather a label corresponding to a fixed score. Based on the coordination of the above steps, the accuracy of the text quality score is improved.

Description

Technical Field

[0001] This disclosure relates to the field of network technology, and in particular to a method and apparatus for evaluating the quality of text. Background Technology

[0002] With the increasing use of LLM (Large Language Model) to generate text, such as novels, the quality score of a text is often an important reference when recommending texts, making it crucial to obtain the quality score of a text.

[0003] In related technologies, LLM is usually used to evaluate text based on text and evaluation criteria. LLM outputs a quality score in one go. In the above evaluation process, LLM tends to maintain logical consistency and subconsciously reconcile contradictory viewpoints found in the evaluation process, which makes the quality score tend to be a compromise. This leads to the problem of quality score collapse in the middle, that is, the accuracy of the quality score is low. Therefore, how to improve the accuracy of text quality scores is an important research direction. Summary of the Invention

[0004] This disclosure provides a method and apparatus for evaluating the quality of text, which improves the accuracy of text quality scores.

[0005] According to one aspect of the present disclosure, a method for evaluating the quality of text is provided. The method includes: identifying candidate problem items in the text based on the text to be evaluated and identification prompts, wherein the identification prompts include evaluation descriptions of multiple quality evaluation dimensions, and the candidate problem item includes a text segment in the text, a quality evaluation dimension of the text segment, and a first degree of deviation between the text segment and the evaluation description of the corresponding quality evaluation dimension; performing a first verification and a second verification on the candidate problem item by a second agent, wherein the first verification refers to verifying whether the text segment in the candidate problem item exists in the text, and the second verification refers to verifying whether the quality evaluation dimension of the text segment in the candidate problem item is correct; and obtaining a quality score of the text based on a target problem item and a scoring rule, wherein the target problem item is a candidate problem item that passes both the first verification and the second verification, and the scoring rule indicates the score of the target problem item under different degrees of deviation.

[0006] According to another aspect of the present disclosure, a text quality assessment apparatus is provided, the apparatus comprising: The identification module is configured to identify candidate problem items in the text based on the text to be evaluated and identification prompts by a first intelligent agent. The identification prompts include evaluation descriptions of multiple quality evaluation dimensions. The candidate problem items include a text segment in the text, the quality evaluation dimension of the text segment, and a first degree of deviation between the text segment and the evaluation description of the corresponding quality evaluation dimension. The first verification module is configured to perform a first verification and a second verification on the candidate question item through a second intelligent agent. The first verification refers to verifying whether the text segment in the candidate question item exists in the text, and the second verification refers to verifying whether the quality evaluation dimension of the text segment in the candidate question item is correct. The first acquisition module is configured to acquire the quality score of the text based on the target issue item and the scoring rule. The target issue item is a candidate issue item that passes both the first and second checks. The scoring rule indicates the score of the target issue item under different degrees of deviation.

[0007] In some embodiments, the identification module is configured to identify candidate question items in the text based on the text and the identification prompt words in a first session by a target agent; the first verification module is configured to perform the first verification on the candidate question items based on the text and the candidate question items in a second session by the target agent; and to perform the second verification on the candidate question items based on the evaluation descriptions of the multiple quality evaluation dimensions and the candidate question items in a third session by the target agent.

[0008] In some embodiments, the identification module is configured to, in each session, identify prompt words based on the text and one dimension, respectively, and obtain the question items of the text under each quality evaluation dimension, wherein each dimension identification prompt word includes evaluation descriptions of multiple sub-dimensions corresponding to one quality evaluation dimension; the first verification module is configured to, in each session, perform the first verification on the candidate question items of the text under each quality evaluation dimension based on the text and the candidate question items of the text under each quality evaluation dimension, respectively; and perform the second verification on the candidate question items of the text under each quality evaluation dimension based on the evaluation descriptions of each quality evaluation dimension and the candidate question items of the text under each quality evaluation dimension, respectively, in each session.

[0009] In some embodiments, the quality evaluation dimensions include at least two of the following: setting dimension, narrative dimension, and writing style dimension. The setting dimension indicates the underlying logic and basic information of the text, the narrative dimension indicates the plot and rhythm of the text, and the writing style dimension indicates the expression of the text. The sub-dimensions corresponding to the setting dimension include at least one of the following: the setting logic, setting completeness, and setting innovation of the text. The sub-dimensions corresponding to the narrative dimension include at least one of the following: the narrative coherence, narrative rhythm, plot consistency, and plot appeal of the text. The sub-dimensions corresponding to the writing style perspective include at least one of the following: the fluency of the text's language expression, the three-dimensionality of the character portrayal, the fit of the atmosphere creation, the ability to convey emotional resonance, and the appeal of the three golden chapters.

[0010] In some embodiments, the first verification module is configured to perform the first verification on the candidate question item based on the text and the candidate question item by a first verification agent; and to perform the second verification on the candidate question item based on the evaluation descriptions of the plurality of quality evaluation dimensions and the candidate question item by a second verification agent.

[0011] In some embodiments, the recognition module is configured to use multiple recognition agents in a session to recognize prompt words based on the text and a dimension, respectively, and to obtain candidate question items for the text under each quality evaluation dimension. Each first agent is used to recognize candidate question items for the text under a quality evaluation dimension.

[0012] In some embodiments, the identification prompt includes at least two thought chain texts, with different thought chain texts indicating different identification logics for question items in the text; the identification module is configured to identify the text based on each thought chain text by the first agent to obtain the thought chain question item of the text under each thought chain text; and to obtain candidate question items of the text by the first agent performing consensus voting based on the thought chain question items of the text corresponding to at least two thought chain texts.

[0013] In some embodiments, the apparatus further includes an adding module; the adding module is configured to obtain a plurality of representative texts from the plurality of historical evaluation texts based on the text category of each historical evaluation text, the historical evaluation texts indicating texts that have undergone quality evaluation, the plurality of representative texts including at least one historical evaluation text under each text category; and to add the text category, a first target question item, and the score of the first target question item based on the scoring rules of each representative text as examples to the recognition prompt words, the first target question item being the target question item obtained by the plurality of representative texts during quality evaluation.

[0014] In some embodiments, the candidate question item may further include at least one of the position of the text segment in the text and the reason why the text segment does not conform to the corresponding evaluation description.

[0015] In some embodiments, the first acquisition module is configured to: acquire a question item score for the target question item based on the scoring rules and a first degree of deviation between the text segment of the target question item and the evaluation description of the corresponding quality evaluation dimension; acquire a sub-dimension score for the text in each sub-dimension based on the scoring rules, the sub-dimensions of the target question item, and the question item score of the target question item; acquire a score for the text in each quality evaluation dimension based on the scoring rules and the sub-dimension scores of the text; and acquire a quality score for the text based on the scoring rules and the scores of the text in each quality evaluation dimension.

[0016] In some embodiments, the apparatus further includes a second acquisition module, an evaluation module, a third acquisition module, an output module, and a first adjustment module; The second acquisition module is configured to acquire a first benchmark set, the first benchmark set including a plurality of first sample texts, each first sample text having a label, the label including the label quality score of the first sample text; The evaluation module is configured to evaluate the quality of the plurality of first sample texts based on the first sample recognition prompt words, and obtain a first quality score for each first sample text; The third acquisition module is configured to acquire a first correlation coefficient between the first quality score and the labeled quality score of each first sample text. The output module is configured to output the first sample recognition prompt word as the recognition prompt word if the first correlation coefficient is greater than a preset correlation coefficient; The first adjustment module is configured to adjust the identification prompt word of the first sample if the first correlation coefficient is less than or equal to a preset correlation coefficient, until the adjusted correlation coefficient is greater than the preset correlation coefficient, and then output the adjusted correlation coefficient as the identification prompt word.

[0017] In some embodiments, the first adjustment module is configured as a first acquisition submodule, a second acquisition submodule, and a modification submodule; The first acquisition submodule is configured to acquire deviation points based on the quality evaluation result of each first sample text and the annotation, wherein each deviation point is an explanation of the difference between the quality evaluation result and the annotation, and the quality evaluation result indicates the result obtained by performing quality evaluation on the multiple first sample texts based on the first sample recognition prompt words; The second acquisition submodule is configured to obtain the cause of the deviation point by analyzing the agent based on the deviation point; The modification submodule is configured to modify the sample recognition prompt words based on the cause of the deviation point.

[0018] In some embodiments, the second acquisition submodule is configured to perform attribution analysis based on the deviation point through the analysis agent to obtain the cause of the deviation point.

[0019] In some embodiments, the second acquisition module is configured to acquire a plurality of first texts, wherein the first texts are texts with a perfect quality score; rewrite the plurality of first texts by generating an agent based on evaluation descriptions of multiple quality evaluation dimensions to obtain a plurality of second texts; and store the plurality of first texts, the plurality of second texts and corresponding annotations in the first benchmark set.

[0020] In some embodiments, the first correlation coefficient includes at least one of Spearman correlation coefficient, Pearson correlation coefficient, and Kendall correlation coefficient.

[0021] In some embodiments, the apparatus further includes a second adjustment module; the second adjustment module is configured to: acquire a second benchmark set, the second benchmark set including a plurality of second sample texts, the second sample texts being published texts, the annotation ranking of the plurality of second sample texts being related to the comments of each second sample text; acquire the quality score of the plurality of second sample texts, and based on the quality score of each second sample text, obtain a first evaluation ranking of the plurality of second sample texts; acquire a third correlation coefficient between the first evaluation ranking and the annotation ranking of the second sample texts; and adjust the identification prompt words if the third correlation coefficient is less than or equal to a preset correlation coefficient.

[0022] In some embodiments, the device further includes a second adjustment module; the second adjustment module is configured to: acquire a third benchmark set, the second benchmark set including a plurality of third sample texts, the third sample texts being published texts, the annotation ranking of the plurality of third sample texts being related to the delivery performance index of each second sample text; acquire the quality score of the plurality of third sample texts, and based on the quality score of each third sample text, obtain a second evaluation ranking of the plurality of third sample texts; acquire a fourth correlation coefficient between the second evaluation ranking and the annotation ranking of the plurality of third sample texts; and adjust the identification prompt words if the fourth correlation coefficient is less than or equal to a preset correlation coefficient.

[0023] In some embodiments, the apparatus further includes a second verification module; the second verification module is configured to, through the second agent, obtain a second degree of deviation between a text segment of a candidate question item and the evaluation description of the corresponding quality evaluation dimension based on the evaluation description of the plurality of quality evaluation dimensions and the candidate question item; and to perform a third verification on the candidate question item through the second agent, wherein the third verification refers to verifying the deviation between the second degree of deviation of the candidate question item and the first degree of deviation; wherein the deviation between the second degree of deviation of the target question item and the first degree of deviation is less than or equal to a preset deviation.

[0024] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: one or more processors; a memory for storing processor-executable program code; wherein the processor is configured to execute the program code to implement the text quality evaluation method as described above.

[0025] According to another aspect of the present disclosure, a computer-readable storage medium is provided that, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the quality evaluation method described above.

[0026] According to another aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the above-described text quality evaluation method.

[0027] The technical solution provided in this disclosure transforms the problem of evaluating text into a three-step deterministic process of identifying candidate problem items, verifying them, and obtaining a quality score. Specifically, when identifying candidate problem items in the text, multiple quality evaluation dimensions are used, and text segments are forcibly output, avoiding a generalized judgment by the first agent. Furthermore, first and second verifications are performed, reducing issues such as illusions and comprehension biases by the first agent. Moreover, when obtaining the text's quality score, no agent is used; instead, it is obtained based on scoring rules and verified target problem items. The degree of deviation is no longer a vague textual concept freely interpreted by the agent, but rather a label corresponding to a fixed score. Based on the coordination of these steps, the accuracy of the text's quality score is improved.

[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0030] Figure 1 This is a schematic diagram illustrating an implementation environment according to an exemplary embodiment.

[0031] Figure 2 This is a flowchart illustrating a text quality assessment method according to an exemplary embodiment.

[0032] Figure 3 This is a schematic diagram illustrating a text quality evaluation method according to an exemplary embodiment.

[0033] Figure 4 This is another schematic diagram illustrating a text quality assessment method according to an exemplary embodiment.

[0034] Figure 5 This is a flowchart illustrating another method for evaluating the quality of text according to an exemplary embodiment.

[0035] Figure 6 This is a schematic diagram illustrating the acquisition principle of a first reference set according to an exemplary embodiment.

[0036] Figure 7 This is a schematic diagram illustrating three correlation coefficients according to an exemplary embodiment.

[0037] Figure 8 This is a schematic diagram illustrating a multi-reference set according to an exemplary embodiment.

[0038] Figure 9 This is a block diagram illustrating a text quality evaluation device according to an exemplary embodiment.

[0039] Figure 10 This is a block diagram illustrating a terminal according to an exemplary embodiment.

[0040] Figure 11 This is a block diagram illustrating a server according to an exemplary embodiment. Detailed Implementation

[0041] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0042] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0043] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0044] ICC (Intraclass Correlation Coefficient): is a statistic that measures the similarity of observations within the same group or the consistency and reliability between different measurements. It is defined as the proportion of the variance between groups to the total variance.

[0045] Z-Score (Z-Score / Standard Score): is the process of dividing the difference between a number and the mean by the standard deviation.

[0046] NLG (Natural Language Generation) refers to the process of automatically converting structured data or internal representations into human-readable text.

[0047] CoT (Chain of Thought) is a problem-solving method that guides a model to generate a series of intermediate reasoning steps. It mimics the human problem-solving process, breaking it down into multiple coherent logical steps to gradually derive the final conclusion. The core of this method is that the model not only provides the answer directly but also demonstrates its thought process, thereby reducing the possibility of inaccurate results.

[0048] ToT (Tree of Thoughts): An upgraded version of COT (Content-Oriented Thinking), it models the reasoning process as a tree, where each node represents a partial solution, and the branches of the tree represent different exploration paths. The advantage of TOT is its ability to consider multiple possible solutions simultaneously and to correct errors through backtracking.

[0049] SFT (Supervised Fine-Tuning) refers to the process of further training a pre-trained large model using a high-quality dataset with labeled input and output, enabling it to learn specific task formats, instruction compliance, or domain knowledge, thereby transforming general language capabilities into targeted execution capabilities.

[0050] DPO (Direct Preference Optimization) is a large language model alignment technique that optimizes policy models by directly utilizing paired human preference data (such as preferred answers vs. non-preferred answers).

[0051] Benchmark (Benchmark Dataset): A set of standardized test tasks, questions, and datasets used to measure and compare the performance of different AI models.

[0052] ROI (Return On Investment): refers to the ratio of net income obtained from investment to total investment, reflecting the economic return a project obtains from investment activities.

[0053] AI (Artificial Intelligence) is a new technological science that studies, develops, and applies theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence.

[0054] Figure 1 This is a schematic diagram illustrating an implementation environment according to an exemplary embodiment. (Refer to...) Figure 1 The implementation environment specifically includes: terminal 101 and server 102. Terminal 101 and server 102 can be connected directly or indirectly through wired or wireless communication, which is not limited herein.

[0055] Terminal 101 is at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, and mobile computer. Terminal 101 runs an application that supports text quality assessment methods. Users can log in to this application through Terminal 101 to access the services provided by the application. For example, users can perform text quality assessments through this application on Terminal 101.

[0056] Terminal 101 generally refers to one of a plurality of terminals; this embodiment uses terminal 101 as an example. Those skilled in the art will understand that the number of terminals can be more or less. For example, there may be several terminals, or dozens or hundreds of terminals, or even more. This disclosure does not limit the number of terminals or the type of device.

[0057] Server 102 is at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 102 provides backend services for applications supporting text quality assessment methods. Server 102 can connect to terminal 101 via a wireless or wired network. Server 102 can provide the terminal with database services related to text quality assessment methods, and of course, other types of services. In some embodiments, the number of servers may be more or fewer, and this disclosure does not limit this. Of course, server 102 may also include other functional servers to provide more comprehensive and diversified services.

[0058] In related technologies, texts generated through LLM (taking novels as an example) are being widely adopted in online literature platforms, digital publishing institutions, and literary award review and recommendation systems. As a content form characterized by long texts, strong narrative structures, and reader retention, the quality evaluation of these novels is crucial for LLM iteration, recommendation ranking, and publishing decisions. However, current evaluation standards in the novel field are fragmented and lack scientific verification mechanisms: different platforms and evaluation institutions use significantly different evaluation standards; the LLM evaluation engine used as a "referee" suffers from systemic biases such as mid-range score collapse, attention decay in long texts, illusory scoring, and a lack of traceable evidence; and the optimization of the referee model itself has long relied on manual prompt trial and error, which is inefficient and unreproducible. Therefore, "how to make the referee model scoring both efficient and accurate" has become a core problem that the industry urgently needs to solve.

[0059] Among related technologies, the first evaluation method used to evaluate novels is purely subjective human evaluation, in which editors, critics, or target readers subjectively score the novel after reading it. The advantage is that it is close to the real reader's feelings, but the disadvantages are that the evaluation standards are not uniform, personal preferences vary greatly, efficiency is low, and it is not reproducible.

[0060] Among related technologies, the second evaluation method used to evaluate novels is single objective index evaluation, which uses word count, lexical richness, sentence complexity, grammatical accuracy, keyword coverage, or general NLG indicators such as BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation). These methods can only reflect the surface features of the text and are powerless to address higher-order dimensions such as setting, narrative, and character development.

[0061] Among related technologies, the third evaluation method for novels is to use an LLM (Limited Learning Model) to directly obtain the novel's quality score. This approach has four typical problems: First, mid-score collapse, with quality scores highly concentrated around 3-4 points, resulting in poor differentiation; second, illusory scoring, where the LLM's scoring reasons may be unrelated to the original text and cannot be traced back to specific paragraphs; third, attention decay in long texts, with the quality of scores for the later sections of long novels significantly lower than those for the earlier sections.

[0062] In summary, the related technologies suffer from the following problems: a lack of systematic, multi-dimensional, detailed, and verifiable evaluation standards for text; LLM scoring is direct and untraceable, with unreliable and unidentifiable reasons for deductions; optimization of prompt words relies on manual trial and error, resulting in low efficiency and non-reproducibility; quality evaluation of a single text can take hours or even days, making it impossible to support minute-level quality evaluation, with low automation and significant efficiency bottlenecks; they often focus on a single dimension, failing to consider a more comprehensive quality evaluation; and the rationality of the quality score itself lacks quantitative verification methods, making the credibility of the quality score untraceable. To address the problems existing in the above-mentioned evaluation methods, this disclosure proposes a text quality evaluation method.

[0063] Figure 2 This is a flowchart illustrating a text quality assessment method according to an exemplary embodiment, with reference to... Figure 2 In this embodiment, the method is executed by a terminal, and the method includes the following steps: In step 201, the first agent identifies candidate problem items in the text based on the text to be evaluated and the identification prompts. The identification prompts include evaluation descriptions of multiple quality evaluation dimensions. The candidate problem items include text segments in the text, the quality evaluation dimensions of the text segments, and the first degree of deviation between the text segments and the evaluation descriptions of the corresponding quality evaluation dimensions.

[0064] In this step, the first agent can be any large language model, such as the Qwen3 series of large language models. The text to be evaluated is a textual work, such as a novel. The recognition prompts indicate how to identify candidate question items from the text to be evaluated according to the evaluation descriptions of multiple quality evaluation dimensions. Quality evaluation dimensions refer to the dimensions used to evaluate the text, such as at least one of setting, narrative, and writing style dimensions, or at least one of character, plot, and emotion dimensions. Evaluation descriptions refer to the explanatory text for the quality evaluation dimensions. The text segment is a portion of the original text. The first deviation level is the degree of deviation between the text segment and the evaluation description of the corresponding quality evaluation dimension, such as slight, moderate, or severe deviation.

[0065] For example, the identification prompts include: You are an expert in identifying issues across multiple quality assessment dimensions; the input is the text to be evaluated and evaluation descriptions for multiple quality assessment dimensions; the task is to identify candidate issues only within the specified range of multiple quality assessment dimensions; each candidate issue outputs at least three tuples: ② quality assessment dimension (belonging to the assessment dimension), ③ first degree of deviation (slight, moderate, or severe), ④ text segment (the original text within the text ≤ 20 characters), with constraints that it cannot cross boundaries into other quality assessment dimensions, the text segment must be the original text, and no quality score is assigned to the text; the output is a list consisting of at least one candidate issue. Through the above constraints and the output format of each candidate issue, the first agent is doubly constrained, ensuring that it only performs the task of "identifying candidate issues" and that these issues fall within a fixed quality assessment dimension. This avoids the "self-justification bias" caused by a single agent both identifying and scoring. When the first agent only outputs the identified candidate issues, does not output scores, and the quality assessment dimensions are limited, this self-justification is naturally eliminated.

[0066] To facilitate faster location of text segments during the first verification, candidate question items may include the position of the text segment within the text. To improve the convenience of the second verification, candidate question items may also include the reason why the text segment does not conform to the corresponding evaluation description. Accordingly, in some embodiments, candidate question items may also include at least one of the following: the position of the text segment within the text and the reason why the text segment does not conform to the corresponding evaluation description. In this step, the position of the text segment within the text can be: which chapter and paragraph the text segment is in, providing an anchor point in the original text for subsequent verification. During the first verification, the second agent can quickly locate the corresponding position in the text based on the position of the text segment, improving the speed of the first verification. During the second verification, based on the reason why the text segment does not conform to the corresponding evaluation description, it is easier to verify whether the quality evaluation dimension of the text segment is correct, and this reason also guides further modifications to the text, solving the problems of lack of evaluation basis, untraceable quality scores, and lack of guiding significance in LLM quality evaluation.

[0067] For example, candidate question items might be: Candidate Problem 1: Chapter 3, paragraphs 47-52; logical inconsistency, serious; "Character 1 pushed a five-ton bronze cauldron" - violates basic physics.

[0068] Candidate issue 2: Chapter 7, paragraph 12; logical inconsistency, serious; character background inconsistent with the setting in Chapter 1—attribute clipping.

[0069] Candidate issue 3: Chapter 11, paragraph 3; minor logical inconsistency; slight timeline misalignment—the date description in Chapter 5 is incorrect.

[0070] In some embodiments, the quality evaluation dimensions include at least two of the following: setting dimension, narrative dimension, and writing style dimension. The setting dimension indicates the underlying logic and basic information of the text; the narrative dimension indicates the plot and rhythm of the text; and the writing style dimension indicates the expression of the text. The sub-dimensions corresponding to the setting dimension include at least one of the following: logical setting, complete setting, and innovative setting. The sub-dimensions corresponding to the narrative dimension include at least one of the following: narrative coherence, narrative rhythm, plot consistency, and plot appeal. The sub-dimensions corresponding to the writing style dimension include at least one of the following: fluency of language expression, three-dimensionality of character development, appropriateness of atmosphere creation, power of emotional resonance, and appeal of the "golden three chapters." Specifically, the setting dimension refers to the solidity and originality of the worldview foundation in the text; the narrative dimension refers to the causal chain and rhythm control of the plot progression in the text; and the writing style dimension refers to the quality of language, character portrayal, and emotional rendering in the text. The quality evaluation dimensions can be continuously iterated with reference to industry standards (such as online literature evaluation systems). For example, multiple quality evaluation dimensions and their corresponding sub-dimensions are shown in Table 1, which is a table explaining the evaluation of the quality evaluation dimensions and their corresponding sub-dimensions.

[0071]

[0072] Table 1 In related technologies, LLM relies on free reasoning when performing quality assessment, resulting in strong randomness and poor reproducibility of the assessment results. To address this issue, the identification prompts can include thought chain text representing the identification logic for the question items in the text. Accordingly, in some embodiments, the identification prompts include at least two thought chain texts, with different thought chain texts indicating different identification logics for the question items in the text. Step 201 includes: a first agent identifying the text based on each thought chain text to obtain the thought chain question items for the text under each thought chain text; and the first agent performing consensus voting based on the thought chain question items corresponding to at least two thought chain texts to obtain candidate question items for the text. In this step, the thought chain text forces the first agent to reason and identify according to the identification logic of the thought chain text, eliminating the subjective oversight of the first agent and improving reproducibility. Considering that single thought chain texts suffer from path dependence and are prone to bias, such as focusing only on logical flaws while neglecting emotional resonance, this disclosure introduces a thought tree structure (TOT). This involves at least two thought chain texts, with recognition performed in parallel on the same sub-dimension based on these texts. Consensus voting is conducted on the thought chain question items corresponding to the texts of these at least two thought chain texts to obtain candidate question items for the text. This avoids the bias and one-sided perspective problems associated with single thought chain texts. For example, the thought chain texts can include recognition logics for three personality types: "target reader profile," "senior editor," and "literary critic." By injecting multiple thought chain texts into the recognition prompts, the self-interaction of the recognition logics corresponding to multiple personalities naturally creates differences in perspectives during the recognition of candidate question items, thereby improving the comprehensiveness of the quality evaluation.

[0073] For example, each thought chain text includes the following identification logic: the text to be evaluated is broken down item by item according to each sub-dimension, and the observation points of each sub-dimension are listed, without allowing cross-sub-dimension mixing (multi-dimensional deconstruction); for each observation point, a specific paragraph anchor point is forcibly located in the text and the text segment is extracted, and the observation point of the found text segment is output as a candidate question item (evidence tracing); the candidate question item is output in the following format: the position of the text segment in the text, the text segment in the text, the quality evaluation dimension of the text segment, and the evaluation description of the text segment and the corresponding quality evaluation dimension (structured output). Among them, the identification logic of different thought chain texts focuses on different aspects. For example, for target readers (pleasure-oriented), the focus is on the reader's reading experience, the break of pleasure points, and the emotional incongruity; for senior editors (business-oriented), the focus is on the commercial compliance, retention hooks, traffic shortcomings, and publishing suitability; for literary critics (literature-oriented), the focus is on the narrative techniques, character arcs, logical narrative, and writing aesthetics.

[0074] To improve the accuracy of the first agent's recognition, texts that have already undergone quality evaluation can be added as examples to the recognition prompts. Accordingly, in some embodiments, the method further includes: obtaining multiple representative texts from multiple historical evaluation texts based on the text category of each historical evaluation text, where the historical evaluation texts indicate texts that have undergone quality evaluation, and the multiple representative texts include at least one historical evaluation text under each text category; adding the text category, the first target question item, and the score of the first target question item based on the scoring rules of each representative text as examples to the recognition prompts, where the first target question item is the target question item obtained from the multiple representative texts during quality evaluation. If only the evaluation descriptions of multiple quality evaluation dimensions are relied upon, the first agent's understanding of the boundaries and deviations of each quality evaluation dimension is inevitably biased. By adding the above content to the recognition prompts and using the above content as examples, the first agent can more clearly define the boundaries of each quality evaluation dimension and the boundaries of the deviations, thereby improving the accuracy of the candidate question items recognized by the first agent. Furthermore, in this step, multiple representative texts include at least one historical evaluation text under each text category, so that there are examples under each text category, enabling the first agent to have high recognition accuracy for texts in each text category.

[0075] In step 202, the second agent performs a first verification and a second verification on the candidate question item. The first verification refers to verifying whether the text segment in the candidate question item exists in the text, and the second verification refers to verifying whether the quality evaluation dimension of the text segment in the candidate question item is correct.

[0076] In this step, the second agent can be any large language model. The first check verifies whether the text segment in the candidate question item exists in the text. If the check fails, the text segment is considered a hallucination of the first agent. The second check verifies whether the quality evaluation dimensions of the text segment in the candidate question item are correct. If the check fails, it is considered that the first agent has a biased understanding of the quality evaluation dimensions. Through these two types of checks, the impact of the first agent's hallucinations (such as citing text segments that do not exist in the text, or treating text segments that do not match the quality evaluation dimensions as candidate question items) and comprehension biases (such as correctly identifying the text segment but misclassifying it according to the quality evaluation dimensions) on the quality evaluation is reduced.

[0077] In this step, the second agent compares the text segment in the candidate question item with the text. If the match fails, the text segment does not exist in the text, the candidate question item is a phantom, and the candidate question item is discarded. The second agent independently determines whether the text segment in the candidate question item truly matches the quality evaluation dimensions output by the first agent. For example, if a candidate question item includes a text segment that belongs to the setting logic question, the second agent rereads the text segment and verifies it according to the evaluation descriptions of multiple quality evaluation dimensions to obtain the verification result, such as "this is indeed setting logic" or "it actually belongs to narrative coherence". If the second agent's verification of the quality evaluation dimensions of the text segment in the candidate question item is incorrect, the candidate question item is discarded.

[0078] When both the first and second agents are target agents, in some embodiments, step 201 includes: identifying candidate question items in the text based on text and recognition prompts in the first session by the target agent; step 202 includes: performing a first verification on the candidate question items based on text and candidate question items in the second session by the target agent; and performing a second verification on the candidate question items based on evaluation descriptions of multiple quality evaluation dimensions and candidate question items in the third session by the target agent. When the identification and verification process is based on the same target agent, it avoids the situation of split understanding of evaluation descriptions of quality evaluation dimensions. Furthermore, by isolating identification and verification through different sessions, the verification process starts from a completely new blank session, so the target agent cannot see the memories from the identification process and will not engage in self-rationalization for the sake of logical consistency.

[0079] When both the first and second agents are target agents, candidate issue items can be identified and verified in a single session based on the evaluation descriptions for each quality evaluation dimension. Accordingly, in some embodiments, step 201 includes: in each session, the target agent identifies prompt words based on the text and one dimension to obtain the issue items for the text under each quality evaluation dimension. Each dimension's prompt word includes evaluation descriptions for multiple sub-dimensions corresponding to one quality evaluation dimension. Step 202 includes: in each session, the target agent performs a first verification on the candidate issue items for the text under each quality evaluation dimension based on the text and the candidate issue items for the text under each quality evaluation dimension; in each session, the target agent performs a second verification on the candidate issue items for the text under each quality evaluation dimension based on the evaluation descriptions for each quality evaluation dimension and the candidate issue items for the text under each quality evaluation dimension. In this step, a single session focuses on the identification or verification of only one quality evaluation dimension, without needing to simultaneously identify or verify multiple quality evaluation dimensions. This reduces the omission of candidate issue items, improves calibration accuracy, and avoids confusion between different quality evaluation dimensions.

[0080] The first and second verifications can be performed by different verification agents. Accordingly, in some embodiments, step 202 includes: performing a first verification on candidate question items based on text and candidate question items by a first verification agent; and performing a second verification on candidate question items based on evaluation descriptions and candidate question items across multiple quality evaluation dimensions by a second verification agent. In this step, the first and second verifications employ different agents, each agent performs only one type of verification task, and the first and second verifications do not interfere with each other. Each verification agent is not distracted by another verification task, resulting in higher verification accuracy and avoiding homogeneity bias when the same agent performs verification.

[0081] For different quality assessment dimensions, different recognition agents can be used to identify corresponding candidate question items. Accordingly, in some embodiments, step 201 includes: using multiple recognition agents in the session to identify prompt words based on the text and one dimension respectively, and obtaining candidate question items for the text under each quality assessment dimension. Each first agent is used to identify candidate question items for the text under one quality assessment dimension. If the same recognition agent is used to identify candidate question items under all quality assessment dimensions, then the recognition agent needs to memorize the recognition logic of all quality assessment dimensions at the same time. When dealing with texts with a large number of words, it is easy to miss candidate question items. In this step, different recognition agents are used to identify candidate question items under different quality assessment dimensions. The thinking scope of each recognition agent is narrowed according to the quality assessment dimension, making it less likely to miss candidate question items during recognition, and avoiding the problem of recognition agents confusing different quality assessment dimensions, thus improving the accuracy of the quality assessment dimensions of the text segment. The recognition of multiple quality assessment dimensions can be executed in parallel, improving the processing speed.

[0082] For candidate question items under different quality evaluation dimensions, first and second verification agents can be used to perform first and second verifications respectively. Accordingly, in some embodiments, step 202 includes: using multiple first verification agents in a session, respectively based on the text and the candidate question items under a quality evaluation dimension, to perform first verification on the candidate question items under a quality evaluation dimension; using multiple second verification agents in a session, respectively based on the evaluation description of a quality evaluation dimension and the candidate question items under a quality evaluation dimension, to perform second verification on the candidate question items under a quality evaluation dimension. (Refer to...) Figure 3In this step, for each quality evaluation dimension, three agents work together: an identification agent, a first verification agent, and a second verification agent. The multiple agents work against each other, which reduces the comprehension bias of a single agent and completely isolates the identification and verification of different quality evaluation dimensions, thus avoiding confusion across quality evaluation dimensions.

[0083] In step 203, the quality score of the text is obtained based on the target issue item and the scoring rules. The target issue item is the candidate issue item that passes both the first and second checks. The scoring rules include the scores of the target issue item under different degrees of deviation.

[0084] In this step, the target issue item is a candidate issue item whose text segment exists in the text and whose quality evaluation dimension is correct. For example, the scoring rules include a preset maximum score for each sub-dimension and a deduction score corresponding to the first degree of deviation. For instance, the preset maximum score for a sub-dimension is 5 points; for example, a slight deviation results in a deduction score of 0.2, a moderate deviation results in a deduction score of 0.6, and a severe deviation results in a deduction score of 1.0. Based on this step, the existing technology of directly evaluating quality and giving quality scores through intelligent agents is upgraded to having an intelligent agent provide target problem items and obtain quality scores based on scoring rules. The final quality score is determined entirely by two factors: the target problem item and the scoring rules. It no longer relies on the subjective judgment of any intelligent agent, which improves the interpretability of the quality score. It is clear at a glance which target problem item each point is deducted from, and the list of target problem items naturally serves as "scoring evidence." It also improves traceability, as the quality score results can be traced back to their original positions in the text, facilitating manual review. Furthermore, it improves reproducibility, eliminating the randomness of quality evaluation through subjective judgment in LLM, ensuring that the same list of target problem items always receives the same score.

[0085] In some embodiments, step 203 includes: obtaining a question item score for the target question item based on scoring rules and a first degree of deviation between the text segment of the target question item and the evaluation description of the corresponding quality evaluation dimension; obtaining a sub-dimension score for the text in each sub-dimension based on scoring rules, the sub-dimensions of the target question item, and the question item score of the target question item; obtaining a score for the text in each quality evaluation dimension based on scoring rules and the sub-dimension scores of the text; and obtaining a quality score for the text based on scoring rules and the scores of the text in each quality evaluation dimension. Wherein, refer to Figure 4Taking the scoring rules, which include the preset full score and the deduction score corresponding to the first deviation degree for each sub-dimension, as an example, the sub-dimension score is max(0, 5 - total deduction), where the total deduction score refers to the sum of the deduction scores of all target problem items in the sub-dimension. By max(0,...), it is ensured that the quality score will not become negative due to too many target problem items, so that the quality score is capped at 0. Of course, the preset full score and the deduction score corresponding to the first deviation degree can also be other scores, and this embodiment does not specifically limit them. The scoring rules can also include the weights of different sub-dimensions and the weights of different quality evaluation dimensions. For example, if the weight of completeness is set at 27%, the weight of logic at 27%, the weight of innovation at 27%, and the weight of social value at 19%, then the score for each dimension is: (completeness sub-dimension score × 27%) + (logic sub-dimension score × 27%) + (innovation sub-dimension score × 27%) + (social value sub-dimension score × 19%). For example, if the weights of each dimension are 36%, the weight of the narrative dimension is 36%, and the weight of the writing style dimension is 28%, then the quality score of the text is: (dimensional score × 36%) + (narrative dimension score × 36%) + (writing style dimension score × 28%).

[0086] For example, taking a text quality score of 5 as the maximum, the text quality descriptions corresponding to different quality scores are shown in Table 2. Table 2 is the text quality description (or generation level description).

[0087]

[0088] Table 2 The method can also verify the first degree of deviation. Accordingly, in some embodiments, the method further includes: obtaining a second degree of deviation between the text segment of a candidate question item and the evaluation description of the corresponding quality evaluation dimension based on the evaluation description and candidate question items of multiple quality evaluation dimensions by a second intelligent agent; performing a third verification on the candidate question item by the second intelligent agent, wherein the third verification refers to verifying the deviation between the second degree of deviation and the first degree of deviation of the candidate question item; wherein the deviation between the second degree of deviation and the first degree of deviation of the target question item is less than or equal to a preset deviation. In this step, the deviation is the level of difference between the degrees of deviation. For example, when the degree of deviation includes mild, moderate and severe, the preset deviation can be level one. That is, if the first degree of deviation is severe and the second degree of deviation is moderate, the deviation between the second degree of deviation and the first degree of deviation is level one, and the deviation is equal to the preset deviation, then the verification passes; if the first degree of deviation is severe and the second degree of deviation is mild, the deviation between the second degree of deviation and the first degree of deviation is level two, and the deviation is greater than the preset deviation, then the verification fails.

[0089] The technical solution provided in this disclosure transforms the problem of evaluating text into a three-step deterministic process of identifying candidate problem items, verifying them, and obtaining a quality score. Specifically, when identifying candidate problem items in the text, multiple quality evaluation dimensions are used, and text segments are forcibly output, avoiding a generalized judgment by the first agent. Furthermore, first and second verifications are performed, reducing issues such as illusions and comprehension biases by the first agent. Moreover, when obtaining the text's quality score, no agent is used; instead, it is obtained based on scoring rules and verified target problem items. The degree of deviation is no longer a vague textual concept freely interpreted by the agent, but rather a label corresponding to a fixed score. Based on the coordination of these steps, the accuracy of the text's quality score is improved.

[0090] Figure 5 This is a flowchart illustrating another text quality assessment method according to an exemplary embodiment, with reference to... Figure 5 In this embodiment, the method is executed by an electronic device. The following description uses an electronic device provided as a terminal as an example. The method includes the following steps: In step 501, a first benchmark set is obtained, which includes multiple first sample texts, each of which has annotations, including the annotation quality score of the first sample text.

[0091] In this step, the labeled quality score refers to the quality score of the first sample text obtained through manual annotation. The first benchmark set includes multiple labeled first sample texts. Therefore, based on the first benchmark set, the consistency between the quality score obtained by the text quality evaluation method in this embodiment of the disclosure and the labeled quality score can be verified.

[0092] In some embodiments, the annotation further includes annotation issue items, which include the annotated text segment, the annotation quality evaluation dimension (or annotation sub-dimension) of the annotated text segment, the position of the annotated text segment in the text, the degree of annotation deviation between the annotated text segment and the evaluation description of the corresponding quality evaluation dimension, the annotation reason why the annotated text segment does not conform to the corresponding evaluation description, and the score of the annotation issue item (e.g., the score is the deducted score). This ensures that the content items included in the annotation are consistent with the content items obtained by the text quality evaluation method in the embodiments of this disclosure.

[0093] Below, refer to Figure 6 The process of obtaining the first reference set is illustrated by way of example, and step 501 above includes steps 5011 to 5013 below.

[0094] In step 5011, multiple first texts are obtained, and the first texts are texts with a quality score of full marks.

[0095] In this step, the first text can be an industry-recognized perfect text, such as "And Then There Were None," "The Little Prince," and "The Old Man and the Sea."

[0096] In step 5012, multiple first texts are rewritten by generating evaluation descriptions based on multiple quality evaluation dimensions of the intelligent agent to obtain multiple second texts.

[0097] In this step, the generated agent is any large language model, used to summarize defects (Bad Cases) based on evaluation descriptions of multiple quality assessment dimensions, and also based on academic theoretical basis (such as setting the logic sub-dimensional based on Forster's *Aspects of the Novel*, the narrative rhythm sub-dimensional based on Richardson's *On the Narrative Rhythm of Modern Novels*, etc.). Defects refer to negative textual patterns that lower the quality score of the first text under the corresponding sub-dimensional. For example, in setting the logic sub-dimensional, defects could be: key plot points lack causal transitions, events occur out of nowhere; rewriting multiple first texts to varying degrees, such as rewriting 3 to 5 defects in 25% to 33% of the chapters of the first text, or rewriting more than 6 defects in 41% to 50% of the chapters of the first text, to generate multiple second texts, and the instruction scores of the multiple second texts are different, then the multiple first texts and multiple second texts can form a benchmark set with a clear quality score gradient.

[0098] In step 5013, multiple first texts, multiple second texts, and their corresponding annotations are stored in the first reference set.

[0099] In this step, the annotation of each first sample text (first text or second text) is obtained by manually annotating the first sample text. The manual annotation includes annotation by industry experts and annotation by crowdsourcing personnel. Crowdsourcing personnel refers to unspecified part-time workers who distribute the annotation tasks of the first and second texts to the general public through the Internet platform.

[0100] In some embodiments, the manual annotation of the first sample text is processed to obtain annotations, including: obtaining multiple manual annotations for the first sample text (each manual annotation is performed by one annotator), each manual annotation including initial problem items and initial quality scores of the first sample text (obtained based on initial problem items and scoring rules); obtaining the ICC (Intraclass Correlation) of the multiple initial quality scores of the first sample text. Coefficient (intra-group correlation coefficient). If ICC < 0.5, re-acquire the manual annotation of the first sample text until ICC ≥ 0.5. Perform Z-Score scaling normalization on each initial quality score of the first sample text, calculate the average value, and obtain the annotation quality score. Cluster all initial question items of the first sample text according to the position and similarity of the annotated text segments in the initial question items. That is, cluster the initial question items whose positions are in the same paragraph and whose similarity meets the preset value into the same question cluster. For the quality evaluation dimension (or sub-dimension) and deviation degree of the annotated text segments in each question cluster, take the mode to obtain the annotation quality evaluation dimension (or annotation sub-dimension) and annotation deviation degree. Refine the reasons in the manual annotations that include the corresponding annotation quality evaluation dimension (or annotation sub-dimension) and annotation deviation degree to obtain the annotation reasons that the annotated text segments do not conform to the corresponding evaluation description, and thus obtain the annotation question items. In this step, an ICC < 0.5 indicates significant differences in the leniency or strictness of the annotations among multiple annotators regarding the first sample text, necessitating complete re-annotation. An ICC ≥ 0.5 indicates minimal disagreement among annotators regarding the overall evaluation of the first sample text, making manual annotation acceptable. Z-Score normalization transforms each annotator's initial quality score into a distribution with a mean of 0 and a standard deviation of 1, converting subjective and ambiguous individual judgments into objective and uniform quantitative data, eliminating biases in the leniency or strictness of different annotators. The annotation quality score better reflects the annotator's true overall evaluation; therefore, the correlation coefficient is calculated based on the annotation quality score obtained in this step.

[0101] The ICC (2,1) value measures the consistency of quality scores given by different annotators when evaluating the quality of the same batch of first-sample texts. The ICC value ranges from 0 to 1, with higher values ​​indicating higher consistency. A ICC close to 1 indicates very high consistency among annotators, meaning that different annotators give very similar quality scores, and the differences between the first-sample texts can be well distinguished. A ICC close to 0 indicates very low consistency among annotators, meaning that the quality scores given by annotators are random and cannot reliably distinguish the first-sample texts.

[0102] For example, the ICC is calculated using the following formula: Formula 1:

[0103] Where n is the number of text samples in the first sample; k is the number of annotators; The row mean square is the mean square of the initial quality scores of the n first sample texts, with the first sample text as the row. The column mean square is calculated with the annotators as the column, which is the mean square of the initial quality scores among the k annotators. The mean square of the error.

[0104] For example, the Z-Score is calculated using the following Formula 2: Formula 2:

[0105] Where z is the standardized score of a first sample text at an annotator's scale; x is the initial quality score given to the first sample text by the annotator; μ is the average of all initial quality scores given by the annotator; and σ is the standard deviation of all initial quality scores given by the annotator. Subtracting the average score from the initial quality score eliminates centering bias, directly eliminating both loose and strict biases. Will In addition to its standard deviation, the score scales of different annotators are "stretched" or "compressed" to the same variable range.

[0106] Of course, the above is merely an exemplary description of obtaining the first benchmark set. The first benchmark set can also be any benchmark set with annotations, and this disclosure does not specifically limit this.

[0107] In step 502, the quality of multiple first sample texts is evaluated based on the first sample recognition prompt words to obtain a first quality score for each first sample text.

[0108] In this step, the first sample identification prompt includes evaluation descriptions for multiple quality assessment dimensions.

[0109] The process of quality evaluation of multiple first sample texts is described below by way of example. Step 502 above includes steps 5021 to 5023.

[0110] In step 5021, the first intelligent agent identifies candidate question items in the first sample text based on the first sample text and sample identification prompts. The sample identification prompts include evaluation descriptions of multiple quality evaluation dimensions. The candidate question items include sample text segments in the first sample text, the quality evaluation dimensions of the sample text segments, and the sample deviation degree between the sample text segments and the evaluation descriptions of the corresponding quality evaluation dimensions.

[0111] In step 5022, the second agent performs a first verification and a second verification on the sample candidate question item. The first verification refers to verifying whether the sample text segment in the sample candidate question item exists in the first sample text, and the second verification refers to verifying whether the quality evaluation dimension of the text segment in the sample candidate question item is correct.

[0112] In step 5023, based on the target question item and the scoring rules, the first quality score of the first sample text is obtained. The target question item is the candidate question item of the sample that passes both the first and second checks. The scoring rules indicate the score of the target question item under different degrees of deviation.

[0113] The principles of steps 5021 to 5023 are the same as those of steps 201 to 203, and will not be repeated here.

[0114] In step 503, the first correlation coefficient between the first quality score and the labeled quality score of each first sample text is obtained.

[0115] In this step, the first correlation coefficient indicates the consistency between the first quality score and the labeled quality score of each first sample text. The larger the first correlation coefficient, the more similar the ranking of the first sample text based on the first quality score and the ranking based on the labeled quality score are.

[0116] In some embodiments, the first correlation coefficient includes at least one of Spearman correlation coefficient, Pearson correlation coefficient, and Kendall correlation coefficient. ρ (Spearman correlation coefficient) measures the consistency between the ranking corresponding to the first quality score and the ranking corresponding to the labeled quality score; ρ > 0.5 indicates consistent ranking. r (Pearson correlation coefficient) measures the synchronicity between the fluctuation range of the first quality score and the labeled quality score; r > 0.4 indicates a strong linear correlation. τ (Kendall correlation coefficient) measures the consistency ratio of relative superiority / inferiority between any two text pairs; τ > 0.4 indicates consistent pairing. Furthermore, the p (probability value) values ​​of ρ, r, and τ can be limited to < 0.05 (i.e., a confidence level of 95%). The p value is used to measure whether the correlation reflected by the above correlation coefficients is genuine and credible, and whether it is a random coincidence.

[0117] For example, the Spearman correlation coefficient ρ is calculated using the following formulas three and four: Formula 3:

[0118] Formula 4:

[0119] in, Let be the first quality score of the i-th first sample text; Let be the quality score of the annotation of the i-th first sample text; Let i be the rank of the i-th first sample text in the sorting based on the first quality score; Let be the rank of the i-th first sample text in the sorting based on the labeled quality score; ρ is the difference between the ranking based on the first quality score and the ranking based on the labeled quality score for the same first sample text; n is the number of first sample texts. The value of ρ ranges from [-1, 1], where 1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and 0 indicates no correlation.

[0120] Spearman correlation coefficient is used to prove that the ranking of the first sample text based on the first quality score is consistent with the ranking based on the annotation quality score. For example, the annotation quality scores of the three first sample texts are 95, 60 and 55, and their first quality scores are 70, 60 and 50, respectively. Although the scores differ significantly, the rankings are completely consistent, and the Spearman coefficient is still 1.

[0121] For example, the Pearson correlation coefficient r is calculated using the following formula five: Formula 5:

[0122] Where n is the number of first sample texts; i is the i-th first sample text, with a value of [1, n]; Let be the first quality score of the i-th first sample text; The average of the first quality scores of the n first sample texts; Let be the quality score of the annotation of the i-th first sample text; Let r be the average of the labeled quality scores of the n first sample texts. The value of r ranges from [-1, 1], where 1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and 0 indicates no correlation.

[0123] The Pearson correlation coefficient is used to demonstrate that the first quality score of the first sample text fluctuates in the same range relative to the labeled quality score. For example, the labeled quality scores of the three first sample texts are 60, 70, and 80, and their first quality scores are 50, 60, and 70, respectively (although the overall scores are low, the scores of each first sample text differ by 10 points). Because the fluctuation range is completely consistent, the Pearson coefficient is 1.

[0124] For example, the Kendall correlation coefficient τ is calculated using the following formula six: Formula Six:

[0125] Where n is the number of first sample texts; C is the number of co-op pairs, which are first sample text pairs whose relative ranking is consistent based on the first quality score and the labeled quality score; D is the number of contradictory pairs, which are first sample text pairs whose relative ranking is inconsistent based on the first quality score and the labeled quality score; τ ranges from [-1, 1], where 1 represents perfect positive correlation, -1 represents perfect negative correlation, and 0 represents no correlation.

[0126] Kendall's correlation coefficient measures the consistency of the relative ranking when comparing any two first-sample texts. It does not concern itself with the numerical value, but only with the relative magnitude. It is used to demonstrate the consistency between the ranking result based on the first quality score and the ranking result based on the annotation quality score when comparing the superiority or inferiority of any two first-sample texts. For example, the annotation quality scores of three first-sample texts (A, B, and C) are A(95)>B(90)>C(85), and their first quality scores are A(70)>C(60)>B(50), respectively. However, the comparison between B and C will show a bias, with Kendall deducting more points and Spearman deducting fewer points.

[0127] In step 504, if the first correlation coefficient is greater than the preset correlation coefficient, the first sample identification prompt word is output as the identification prompt word.

[0128] In this step, the preset correlation coefficient refers to a preset threshold for the first correlation coefficient. For example, when the first correlation coefficient includes ρ, r, and τ, the preset correlation coefficient corresponding to ρ can be 0.5, the preset correlation coefficient corresponding to r can be 0.4, and the preset correlation coefficient corresponding to τ can be 0.4. If the first correlation coefficient is greater than the preset correlation coefficient, it indicates that the obtained quality score is already quite close to the manually labeled score, meaning that the quality score obtained by the method of this embodiment can replace the manually labeled quality score. At this point, it can be output as a recognition prompt word, and the quality of the text to be evaluated can be assessed based on the recognition prompt word. This step verifies the reliability of the quality evaluation result, i.e., the first quality score.

[0129] The first correlation coefficient can include all three types mentioned above. When all three first correlation coefficients are greater than the corresponding preset correlation coefficients, it indicates that the rankings are consistent, the fluctuation ranges are synchronized, and the relative superiority or inferiority of any two text pairs is highly consistent. Alternatively, you can first use Pearson to check the linear correlation. If the linear correlation is not obvious, select Spearman (n≥30) or Kendall (n<30) according to the sample size (i.e., the number of first sample texts n), where n is a positive integer.

[0130] In step 505, if the first correlation coefficient is less than or equal to the preset correlation coefficient, the identification prompt word of the first sample is adjusted until the adjusted correlation coefficient is greater than the preset correlation coefficient, and the adjusted correlation coefficient is output as the identification prompt word.

[0131] In this step, if the first correlation coefficient is less than or equal to the preset correlation coefficient, it indicates that the obtained quality score is significantly different from the manually labeled one, and further adjustments to the first sample identification prompt words are needed.

[0132] The process of adjusting the identification prompt words of the first sample is described below by way of example. Step 505 includes the following steps 5051 to 5053.

[0133] In step 5051, based on the quality evaluation result and annotation of each first sample text, deviation points are obtained. Each deviation point is an explanation of the difference between the quality evaluation result and the annotation. The quality evaluation result indicates the result obtained by evaluating the quality of multiple first sample texts based on the first sample recognition prompt words.

[0134] In some embodiments, the quality evaluation result includes sample target issues of the first sample text. These sample target issues include sample text segments within the first sample text, the quality evaluation dimensions of the sample text segments, the degree of deviation between the sample text segments and the evaluation descriptions of the corresponding quality evaluation dimensions, the position of the sample text segments within the first sample text, and the reasons why the sample text segments do not conform to the corresponding evaluation descriptions. Deviation points refer to the explanations of the differences between the quality evaluation result and the annotations. There are various types of deviation points, such as omissions (e.g., the annotation includes Chapter 11, but the timeline is misaligned, while the quality evaluation result does not include the above content), misidentifications (e.g., the quality evaluation result includes character conflicts in Chapter 3, but the corresponding paragraph is consistent throughout the original text without any loopholes), dimension classification errors (e.g., for the same sample text segment, the character description is stiff, the annotation classifies it as writing style-character development, but the quality evaluation result incorrectly classifies it as setting logic), and deviation degree errors (e.g., for core worldview conflicts, the annotation indicates severe issues, while the quality evaluation result indicates minor issues). Furthermore, the deviation points include sample text segments in the first sample text, the annotation quality evaluation dimension (or annotation sub-dimension) of the sample text segments, the deviation type, the content related to the deviation points in the annotation, and the content related to the deviation points in the quality evaluation results, which can be analyzed based on the sample text segments.

[0135] In step 5052, the agent analyzes the deviation points to obtain the reasons for the deviation points.

[0136] In this step, the analytical agent can be any large language model.

[0137] In some embodiments, step 5052 includes: performing attribution analysis based on deviation points by an analysis agent to obtain the cause of the deviation points. In this step, performing attribution analysis on deviation points of the same deviation category by the analysis agent includes: performing attribution analysis on each deviation point to obtain the erroneous logic of each deviation point, and grouping deviation points with the same erroneous logic into the same deviation cluster; if the number of deviation points within the same deviation cluster is higher than a preset value, that is, a large number of deviation points have highly similar erroneous logic, then it is a systematic deviation, and the erroneous logic is output as the cause of the deviation point; if the number of deviation points within the same deviation cluster is lower than or equal to the preset value, it is a sporadic deviation point, belonging to random fluctuations, and no optimization is performed on random fluctuations. For example, the erroneous logic may be: failing to identify long-term conflicts across chapters; or: underestimating the negative impact of main narrative logic problems, with a lenient judgment on the degree of deviation.

[0138] In step 5053, the sample recognition prompt words are modified based on the cause of the deviation.

[0139] In this step, modifying the sample recognition prompts can be achieved by adding patch text related to the cause of the deviation. This patch text can include new examples, explanations of quality assessment dimensions, details of the deviation determination, etc. This step involves analyzing the agent's understanding of the cause of the deviation and the sample recognition prompts to modify the prompts. For example, for the error logic of failing to identify cross-chapter long-line setting conflicts, the patch text can consist of multiple text examples. Each text example's manual annotation includes the issue item regarding cross-chapter long-line setting conflicts, demonstrating how to compare setting conflicts before and after identification across paragraphs.

[0140] After the above steps, with the first benchmark set including 779 first sample texts, before optimizing the first sample recognition prompts, the accuracy rate of the text quality assessment method based on this embodiment of the disclosure for scoring text quality is 33%. After optimizing the first sample recognition prompts, the accuracy rate of the text quality assessment method based on this embodiment of the disclosure for scoring text quality is 85%. The method of this embodiment of the disclosure, based on the first benchmark set, yields r=0.9669, ρ=0.9497, and τ=0.8714, all of which far exceed the corresponding preset correlation coefficients, proving that the text quality assessment method in this embodiment of the disclosure is highly aligned with the results of manual evaluation. (Refer to...) Figure 7The diagram illustrates the first quality score and labeled quality score of the first sample text in the first benchmark set, as well as the r, ρ, and τ obtained based on the first quality score and labeled quality score of the first sample text. The Pearson correlation coefficient r = 0.9669 (far exceeding the corresponding preset correlation coefficient of 0.4), the Spearman correlation coefficient ρ = 0.9497 (far exceeding the corresponding preset correlation coefficient of 0.5), and the Kendall correlation coefficient τ = 0.8714 (far exceeding the corresponding preset correlation coefficient of 0.4).

[0141] After outputting the first sample recognition prompt word, the text to be evaluated can be evaluated using the text quality evaluation method of this embodiment. That is, the quality score of the text to be evaluated is obtained through steps 201 to 203. Furthermore, at least one of the second and third benchmark sets can be obtained based on the already published text to further verify the text quality evaluation method of this embodiment. Accordingly, in some embodiments, reference is made to... Figure 8 The method also includes at least one of the following methods.

[0142] The first approach involves: obtaining a second benchmark set, which includes multiple second sample texts. These second sample texts are published texts, and their labeled rankings are correlated with the comments on each text. The approach also involves obtaining quality scores for these texts and calculating a first evaluation ranking based on these scores. A third correlation coefficient is then obtained between the first evaluation ranking and the labeled rankings of the second sample texts. If the third correlation coefficient is less than or equal to a preset correlation coefficient, the identification prompts are adjusted. In this step, the second sample texts can be published texts on text-based platforms. The annotation rankings of the multiple second sample texts are obtained by a summarizing agent based on the comments corresponding to each text. This summarizing agent represents a ranking that better aligns with popular aesthetics and can be any large language model. Since the labeled rankings of the multiple second sample texts primarily reflect the ranking itself, the third correlation coefficient is a coefficient related to ranking consistency. For example, the third correlation coefficient can include at least one of the Spearman correlation coefficient and the Kendall correlation coefficient. If the third correlation coefficient is less than or equal to the preset correlation coefficient, it indicates that the consistency between the first evaluation ranking and the labeled ranking of the second sample text is poor. In this case, the identification prompt words are adjusted so that the quality score obtained by the text quality evaluation method according to the present disclosure embodiment can better reflect the public's aesthetic.

[0143] The second approach involves obtaining a third benchmark set, which includes multiple third sample texts. These third sample texts are published texts, and their labeled rankings are related to the performance metrics of each second sample text. The approach also involves obtaining quality scores for the multiple third sample texts and, based on these scores, obtaining a second evaluation ranking. Finally, it involves obtaining a fourth correlation coefficient between the second evaluation ranking and the labeled rankings of the multiple third sample texts. If the fourth correlation coefficient is less than or equal to a preset correlation coefficient, the identification prompts are adjusted. In this step, the performance metrics include at least one of ROI, text reading time, and click-through rate. Of course, performance metrics can also include more metrics reflecting the effectiveness of text delivery; this embodiment does not specifically limit this. Since the labeled rankings of the multiple third sample texts are related to the performance metrics of each second sample text, the labeled rankings of the multiple third sample texts better reflect the actual delivery effect (i.e., commercial effect). If the fourth correlation coefficient is less than or equal to the preset correlation coefficient, it indicates that the consistency between the second evaluation ranking and the labeled ranking of the third sample text is poor. In this case, the identification prompt words are adjusted so that the quality score obtained by the text quality evaluation method in this embodiment can better reflect the delivery effect.

[0144] By using at least one of the second and third benchmark sets mentioned above, the text quality evaluation method of this disclosure embodiment can be further verified. If the verification fails, the identification prompt words can be adjusted based on the second and third benchmark sets so that the quality score obtained by the text quality evaluation method of this disclosure embodiment has higher consistency with the real data of each text on each text category platform.

[0145] In some embodiments, the method further includes: deleting the benchmark set in response to a deletion operation on any benchmark set; editing the annotations of the sample text in the benchmark set in response to an editing operation on any benchmark set (e.g., editing the sample text segments annotated with question items, the sample sub-dimensions of the sample text segments, the weights of each sub-dimension, the scores of the annotated question items, etc.); displaying detailed information of the benchmark set in response to a viewing operation on any benchmark set, including the sample text and annotations of the sample text in the benchmark set; and importing other selected benchmark sets or datasets in response to an import operation, wherein the imported benchmark set is an annotated benchmark set and the imported dataset is the dataset to be evaluated.

[0146] In some embodiments, the method further includes: in response to a trigger operation for agent management, displaying a management page, selecting an agent for identification and verification based on the management page, and configuring prompts for the agent; the management page can also configure a calling interface, such as configuring an RPC (Remote Procedure Call) interface.

[0147] In some embodiments, the method further includes: in response to a new evaluation task operation, displaying an evaluation task creation page, on which task name, task description, selection method of text corresponding to the task (dataset or uploaded text), selection of dataset, etc. can be set; and in response to a confirmation operation of the evaluation task, creating the evaluation task.

[0148] In some embodiments, the method further includes: displaying multiple evaluation tasks in response to an evaluation task management operation; and displaying detailed information of the evaluation task in response to a triggering operation on any evaluation task. The detailed information includes task details, data details, and report results. The task details include task progress, total number of texts, number of texts that have been evaluated, number of texts that have failed to be evaluated, completion rate, dataset, etc. The data details include the dialogue content between each text and the agent in the task, as well as the output content of the agent. The report results include the quality score of each text, target question item, deducted score for the target question item, sub-dimension score of sub-dimension, etc.

[0149] When using a single agent, this embodiment identifies and verifies the text based on a separate session. When using multiple agents, different agents perform identification and verification separately. This mechanism eliminates the self-justification that can occur when a single agent directly provides a text score, avoiding the bias of "scores converging towards the center" and solving the problem of score collapse in the middle. By forcing the first agent to identify text segments within the text, combined with subsequent verification, the misjudgment of fabricated text segments and sub-dimensions by the first agent is reduced, minimizing the impact of agent illusions on quality evaluation. The list of target question items directly serves as scoring evidence, improving traceability. The quality score is entirely determined by the target question items and scoring rules; the same list of target question items consistently receives the same score, improving reproducibility. The scoring rules and the weights of each sub-dimension can be continuously adjusted without retraining the agent or readjusting the recognition prompts, resulting in better evolution. The optimization of prompt word recognition has been upgraded from "trial and error based on human experience" to "data-driven closed loop for deviation points," significantly improving the accuracy and stability of the evaluation. Previously, evaluating the quality of a single text manually required at least one day per person. However, the method of this disclosure reduces the time required for evaluating the quality of a single text to the minute level, improving efficiency by more than an order of magnitude compared to purely manual quality evaluation.

[0150] This disclosure proposes a text quality assessment method, constructing a unified evaluation system from three core perspectives: setting, narrative, and writing style, which is also an evaluation description of the quality assessment dimensions. Through a three-step evidence chain scoring algorithm (i.e., identification, verification, and scoring) + multi-agent collaborative implementation (or single agents performing identification and verification in single-round conversations), the method of this disclosure transforms black-box scoring into white-box traceability. Through closed-loop optimization of identification prompts based on a first benchmark set, the optimization of identification prompts shifts from being driven by human experience to being data-driven. Through verification using ICC + Z-Score + correlation coefficient, alignment between the quality score obtained by the method of this disclosure and the manually labeled quality score is achieved.

[0151] Figure 9 This is a block diagram illustrating a text quality evaluation device according to an exemplary embodiment. (Refer to...) Figure 9 The device includes: The identification module 901 is configured to identify candidate problem items in the text based on the text to be evaluated and identification prompt words by a first intelligent agent. The identification prompt words include evaluation descriptions of multiple quality evaluation dimensions. The candidate problem items include a text segment in the text, the quality evaluation dimension of the text segment, and a first degree of deviation between the text segment and the evaluation description of the corresponding quality evaluation dimension. The first verification module 902 is configured to perform a first verification and a second verification on the candidate question item through a second intelligent agent. The first verification refers to verifying whether the text segment in the candidate question item exists in the text, and the second verification refers to verifying whether the quality evaluation dimension of the text segment in the candidate question item is correct. The first acquisition module 903 is configured to acquire the quality score of the text based on the target issue item and the scoring rule. The target issue item is a candidate issue item that passes both the first verification and the second verification. The scoring rule indicates the score of the target issue item under different degrees of deviation.

[0152] In some embodiments, the identification module is configured to identify candidate question items in the text based on the text and the identification prompt words in a first session by a target agent; the first verification module is configured to perform the first verification on the candidate question items based on the text and the candidate question items in a second session by the target agent; and to perform the second verification on the candidate question items based on the evaluation descriptions of the multiple quality evaluation dimensions and the candidate question items in a third session by the target agent.

[0153] In some embodiments, the identification module is configured to, in each session, identify prompt words based on the text and one dimension, respectively, and obtain the question items of the text under each quality evaluation dimension, wherein each dimension identification prompt word includes evaluation descriptions of multiple sub-dimensions corresponding to one quality evaluation dimension; the first verification module is configured to, in each session, perform the first verification on the candidate question items of the text under each quality evaluation dimension based on the text and the candidate question items of the text under each quality evaluation dimension, respectively; and perform the second verification on the candidate question items of the text under each quality evaluation dimension based on the evaluation descriptions of each quality evaluation dimension and the candidate question items of the text under each quality evaluation dimension, respectively, in each session.

[0154] In some embodiments, the quality evaluation dimensions include at least two of the following: setting dimension, narrative dimension, and writing style dimension. The setting dimension indicates the underlying logic and basic information of the text, the narrative dimension indicates the plot and rhythm of the text, and the writing style dimension indicates the expression of the text. The sub-dimensions corresponding to the setting dimension include at least one of the following: the setting logic, setting completeness, and setting innovation of the text. The sub-dimensions corresponding to the narrative dimension include at least one of the following: the narrative coherence, narrative rhythm, plot consistency, and plot appeal of the text. The sub-dimensions corresponding to the writing style perspective include at least one of the following: the fluency of the text's language expression, the three-dimensionality of the character portrayal, the fit of the atmosphere creation, the ability to convey emotional resonance, and the appeal of the three golden chapters.

[0155] In some embodiments, the first verification module is configured to perform the first verification on the candidate question item based on the text and the candidate question item by a first verification agent; and to perform the second verification on the candidate question item based on the evaluation descriptions of the plurality of quality evaluation dimensions and the candidate question item by a second verification agent.

[0156] In some embodiments, the recognition module is configured to use multiple recognition agents in a session to recognize prompt words based on the text and a dimension, respectively, and to obtain candidate question items for the text under each quality evaluation dimension. Each first agent is used to recognize candidate question items for the text under a quality evaluation dimension.

[0157] In some embodiments, the identification prompt includes at least two thought chain texts, with different thought chain texts indicating different identification logics for question items in the text; the identification module is configured to identify the text based on each thought chain text by the first agent to obtain the thought chain question item of the text under each thought chain text; and to obtain candidate question items of the text by the first agent performing consensus voting based on the thought chain question items of the text corresponding to at least two thought chain texts.

[0158] In some embodiments, the apparatus further includes an adding module; the adding module is configured to obtain a plurality of representative texts from the plurality of historical evaluation texts based on the text category of each historical evaluation text, the historical evaluation texts indicating texts that have undergone quality evaluation, the plurality of representative texts including at least one historical evaluation text under each text category; and to add the text category, a first target question item, and the score of the first target question item based on the scoring rules of each representative text as examples to the recognition prompt words, the first target question item being the target question item obtained by the plurality of representative texts during quality evaluation.

[0159] In some embodiments, the candidate question item may further include at least one of the position of the text segment in the text and the reason why the text segment does not conform to the corresponding evaluation description.

[0160] In some embodiments, the first acquisition module is configured to: acquire a question item score for the target question item based on the scoring rules and a first degree of deviation between the text segment of the target question item and the evaluation description of the corresponding quality evaluation dimension; acquire a sub-dimension score for the text in each sub-dimension based on the scoring rules, the sub-dimensions of the target question item, and the question item score of the target question item; acquire a score for the text in each quality evaluation dimension based on the scoring rules and the sub-dimension scores of the text; and acquire a quality score for the text based on the scoring rules and the scores of the text in each quality evaluation dimension.

[0161] In some embodiments, the apparatus further includes a second acquisition module, an evaluation module, a third acquisition module, an output module, and a first adjustment module; The second acquisition module is configured to acquire a first benchmark set, the first benchmark set including a plurality of first sample texts, each first sample text having a label, the label including the label quality score of the first sample text; The evaluation module is configured to evaluate the quality of the plurality of first sample texts based on the first sample recognition prompt words, and obtain a first quality score for each first sample text; The third acquisition module is configured to acquire a first correlation coefficient between the first quality score and the labeled quality score of each first sample text. The output module is configured to output the first sample recognition prompt word as the recognition prompt word if the first correlation coefficient is greater than a preset correlation coefficient; The first adjustment module is configured to adjust the identification prompt word of the first sample if the first correlation coefficient is less than or equal to a preset correlation coefficient, until the adjusted correlation coefficient is greater than the preset correlation coefficient, and then output the adjusted correlation coefficient as the identification prompt word.

[0162] In some embodiments, the first adjustment module is configured as a first acquisition submodule, a second acquisition submodule, and a modification submodule; The first acquisition submodule is configured to acquire deviation points based on the quality evaluation result of each first sample text and the annotation, wherein each deviation point is an explanation of the difference between the quality evaluation result and the annotation, and the quality evaluation result indicates the result obtained by performing quality evaluation on the multiple first sample texts based on the first sample recognition prompt words; The second acquisition submodule is configured to obtain the cause of the deviation point by analyzing the agent based on the deviation point; The modification submodule is configured to modify the sample recognition prompt words based on the cause of the deviation point.

[0163] In some embodiments, the second acquisition submodule is configured to perform attribution analysis based on the deviation point through the analysis agent to obtain the cause of the deviation point.

[0164] In some embodiments, the second acquisition module is configured to acquire a plurality of first texts, wherein the first texts are texts with a perfect quality score; rewrite the plurality of first texts by generating an agent based on evaluation descriptions of multiple quality evaluation dimensions to obtain a plurality of second texts; and store the plurality of first texts, the plurality of second texts and corresponding annotations in the first benchmark set.

[0165] In some embodiments, the first correlation coefficient includes at least one of Spearman correlation coefficient, Pearson correlation coefficient, and Kendall correlation coefficient.

[0166] In some embodiments, the apparatus further includes a second adjustment module; the second adjustment module is configured to: acquire a second benchmark set, the second benchmark set including a plurality of second sample texts, the second sample texts being published texts, the annotation ranking of the plurality of second sample texts being related to the comments of each second sample text; acquire the quality score of the plurality of second sample texts, and based on the quality score of each second sample text, obtain a first evaluation ranking of the plurality of second sample texts; acquire a third correlation coefficient between the first evaluation ranking and the annotation ranking of the second sample texts; and adjust the identification prompt words if the third correlation coefficient is less than or equal to a preset correlation coefficient.

[0167] In some embodiments, the device further includes a second adjustment module; the second adjustment module is configured to: acquire a third benchmark set, the second benchmark set including a plurality of third sample texts, the third sample texts being published texts, the annotation ranking of the plurality of third sample texts being related to the delivery performance index of each second sample text; acquire the quality score of the plurality of third sample texts, and based on the quality score of each third sample text, obtain a second evaluation ranking of the plurality of third sample texts; acquire a fourth correlation coefficient between the second evaluation ranking and the annotation ranking of the plurality of third sample texts; and adjust the identification prompt words if the fourth correlation coefficient is less than or equal to a preset correlation coefficient.

[0168] In some embodiments, the apparatus further includes a second verification module; the second verification module is configured to, through the second agent, obtain a second degree of deviation between a text segment of a candidate question item and the evaluation description of the corresponding quality evaluation dimension based on the evaluation description of the plurality of quality evaluation dimensions and the candidate question item; and to perform a third verification on the candidate question item through the second agent, wherein the third verification refers to verifying the deviation between the second degree of deviation of the candidate question item and the first degree of deviation; wherein the deviation between the second degree of deviation of the target question item and the first degree of deviation is less than or equal to a preset deviation.

[0169] It should be noted that the text quality evaluation device provided in the above embodiments is only illustrated by the division of the above functional units when performing text quality evaluation. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the text quality evaluation device and the text quality evaluation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0170] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0171] When an electronic device is provided as a terminal, Figure 10 This is a block diagram illustrating a terminal according to an exemplary embodiment. The diagram shows a structural block diagram of a terminal 1000 provided in an exemplary embodiment of this disclosure. The terminal 1000 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 1000 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0172] Typically, terminal 1000 includes a processor 1001 and a memory 1002.

[0173] Processor 1001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1001 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1001 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0174] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 are used to store at least one computer program, which is executed by the processor 1001 to implement the method provided in the method embodiments of this application.

[0175] In some embodiments, the terminal 1000 may also optionally include a peripheral device interface 1003 and at least one peripheral device. The processor 1001, memory 1002, and peripheral device interface 1003 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1003 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, and a power supply 1008.

[0176] Peripheral device interface 1003 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1001 and memory 1002. In some embodiments, processor 1001, memory 1002 and peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1001, memory 1002 and peripheral device interface 1003 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0177] The radio frequency (RF) circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1004 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. In some embodiments, the RF circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1004 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1004 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0178] Display screen 1005 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1005 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1001 for processing. In this case, display screen 1005 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1005, disposed on the front panel of terminal 1000; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1000 or in a folded design; in still other embodiments, display screen 1005 may be a flexible display screen, disposed on a curved or folded surface of terminal 1000. Furthermore, display screen 1005 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1005 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0179] The camera assembly 1006 is used to acquire images or videos. In some embodiments, the camera assembly 1006 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1006 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash is a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0180] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1001 for processing, or input to the radio frequency circuit 1004 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1000. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1007 may also include a headphone jack.

[0181] The power supply 1008 is used to power the various components in the terminal 1000. The power supply 1008 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 1008 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0182] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on terminal 1000 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0183] The aforementioned electronic devices can also be implemented as servers. The structure of a server is described below: Figure 11 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1100 can vary considerably due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1101 and one or more memories 1102. The one or more memories 1102 store at least one computer program, which is loaded and executed by the one or more processors 1101 to implement the methods provided in the above-described method embodiments. Of course, the server 1100 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1100 may also include other components for implementing device functions, which will not be elaborated here.

[0184] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program that can be executed by a processor to perform the methods in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0185] In an exemplary embodiment, a computer program product or computer program is also provided, which includes program code stored in a computer-readable storage medium. A processor of an electronic device reads the program code from the computer-readable storage medium and executes the program code, causing the electronic device to perform the method described above.

[0186] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0187] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for evaluating the quality of text, characterized in that, The method includes: The first intelligent agent identifies candidate problem items in the text based on the text to be evaluated and the recognition prompt words. The recognition prompt words include evaluation descriptions of multiple quality evaluation dimensions. The candidate problem items include a text segment in the text, the quality evaluation dimension of the text segment, and a first degree of deviation between the text segment and the evaluation description of the corresponding quality evaluation dimension. The second intelligent agent performs a first verification and a second verification on the candidate question item. The first verification refers to verifying whether the text segment of the candidate question item exists in the text, and the second verification refers to verifying whether the quality evaluation dimension of the text segment of the candidate question item is correct. Based on the target question item and the scoring rules, the quality score of the text is obtained. The target question item is a candidate question item that passes both the first and second checks. The scoring rules include the scores of the target question item under different degrees of deviation.

2. The text quality evaluation method according to claim 1, characterized in that, The step of identifying candidate question items in the text based on the text to be evaluated and the recognition prompt words by the first intelligent agent includes: In the first session, the target agent identifies candidate question items in the text based on the text and the recognition prompts; The step of performing the first and second checks on the candidate question items through the second intelligent agent includes: In the second session, the target agent performs the first verification on the candidate question item based on the text and the candidate question item; The target agent performs the second verification on the candidate question items based on the evaluation descriptions of the multiple quality evaluation dimensions and the candidate question items in the third session.

3. The text quality evaluation method according to claim 1, characterized in that, The step of identifying candidate question items in the text based on the text to be evaluated and the recognition prompt words by the first intelligent agent includes: In each session, the target agent identifies prompt words based on the text and one dimension, respectively, and obtains the question items of the text under each quality evaluation dimension. Each dimension-identified prompt word includes the evaluation description of multiple sub-dimensions corresponding to one quality evaluation dimension. The step of performing the first and second checks on the candidate question items through the second intelligent agent includes: In each session, the target agent performs the first verification on the candidate question items of the text under each quality evaluation dimension, based on the text and the candidate question items of the text under each quality evaluation dimension. In each session, the target agent performs the second verification on the candidate question items of the text under each quality evaluation dimension, based on the evaluation description of each quality evaluation dimension and the candidate question items of the text under each quality evaluation dimension.

4. The text quality evaluation method according to claim 3, characterized in that, The quality evaluation dimensions include at least two of the following: setting dimension, narrative dimension, and writing style dimension. The setting dimension indicates the underlying logic and basic information of the text, the narrative dimension indicates the plot and rhythm of the text, and the writing style dimension indicates the expression of the text. Among them, the sub-dimensions corresponding to the setting dimension include at least one of the following: the setting logic, setting completeness, and setting innovation of the text; the sub-dimensions corresponding to the narrative dimension include at least one of the following: the narrative coherence, narrative rhythm, plot consistency, and plot appeal of the text; and the sub-dimensions corresponding to the writing perspective include at least one of the following: the fluency of the text's language expression, the three-dimensionality of the character portrayal, the fit of the atmosphere creation, the ability to convey emotional resonance, and the appeal of the three golden chapters.

5. The text quality evaluation method according to claim 1, characterized in that, The step of performing the first and second checks on the candidate question items through the second intelligent agent includes: The first verification agent performs the first verification on the candidate question item based on the text and the candidate question item; The second verification agent performs the second verification on the candidate question items based on the evaluation descriptions of the multiple quality evaluation dimensions and the candidate question items.

6. The text quality evaluation method according to claim 1, characterized in that, The step of identifying candidate question items in the text based on the text to be evaluated and the recognition prompt words by the first intelligent agent includes: Multiple intelligent agents are used in the conversation to identify prompt words based on the text and a dimension, respectively, to obtain candidate question items for the text under each quality evaluation dimension. Each first agent is used to identify candidate question items for the text under a quality evaluation dimension.

7. The text quality evaluation method according to claim 1, characterized in that, The identification prompts include at least two thought chain texts, with different thought chain texts indicating different identification logics for the question items in the texts; The step of identifying candidate question items in the text based on the text to be evaluated and the recognition prompt words by the first intelligent agent includes: The first intelligent agent identifies the text based on each thought chain text to obtain the thought chain question item of the text under each thought chain text; Candidate question items for the text are obtained by the first intelligent agent performing consensus voting based on the thought chain question items corresponding to at least two thought chain texts.

8. The text quality evaluation method according to claim 1, characterized in that, The method further includes: Based on the text category of each historical evaluation text, multiple representative texts are obtained from the multiple historical evaluation texts, wherein the historical evaluation texts indicate texts that have undergone quality evaluation, and the multiple representative texts include at least one historical evaluation text under each text category; The text category, first target question item, and score of the first target question item based on the scoring rules for each representative text are added as examples to the recognition prompt words, wherein the first target question item is the target question item obtained by the plurality of representative texts during quality evaluation.

9. The text quality evaluation method according to claim 1, characterized in that, The candidate question items also include at least one of the following: the position of the text segment in the text and the reason why the text segment does not conform to the corresponding evaluation description.

10. The text quality evaluation method according to claim 1, characterized in that, The process of obtaining the quality score of the text based on the target question items and scoring rules includes: Based on the scoring rules and the first degree of deviation between the text segment of the target question item and the evaluation description of the corresponding quality evaluation dimension, the question item score of the target question item is obtained; Based on the scoring rules, the sub-dimensions of the target question item, and the question item score of the target question item, obtain the sub-dimension score of the text in each sub-dimension; Based on the scoring rules and the scores of each sub-dimension of the text, obtain the score of the text in each quality evaluation dimension; Based on the scoring rules and the scores of the text in each quality evaluation dimension, the quality score of the text is obtained.

11. The text quality assessment method according to claim 1, characterized in that, The method further includes: Obtain a first benchmark set, which includes multiple first sample texts, each of which has a label, the label including the label quality score of the first sample text; Based on the first sample recognition prompt words, the quality of the multiple first sample texts is evaluated to obtain a first quality score for each first sample text; Obtain the first correlation coefficient between the first quality score and the labeled quality score of each first sample text; If the first correlation coefficient is greater than the preset correlation coefficient, the first sample identification prompt word will be output as the identification prompt word; If the first correlation coefficient is less than or equal to the preset correlation coefficient, the identification prompt word of the first sample is adjusted until the adjusted correlation coefficient is greater than the preset correlation coefficient, and the adjusted correlation coefficient is output as the identification prompt word.

12. The text quality assessment method according to claim 11, characterized in that, The adjustment of the identification prompt words of the first sample includes: Based on the quality evaluation result of each first sample text and the annotation, deviation points are obtained. Each deviation point is an explanation of the difference between the quality evaluation result and the annotation. The quality evaluation result indicates the result obtained by evaluating the quality of the multiple first sample texts based on the first sample recognition prompt words. By analyzing the agent's analysis of the deviation points, the causes of the deviation points can be obtained; Based on the cause of the deviation, the sample recognition prompt words are modified.

13. The text quality evaluation method according to claim 12, characterized in that, The step of analyzing the agent based on the deviation point to obtain the cause of the deviation point includes: The analytical agent performs attribution analysis based on the deviation points to obtain the causes of the deviation points.

14. The text quality assessment method according to claim 11, characterized in that, The acquisition of the first reference set includes: Obtain multiple first texts, where each first text is a text with a perfect quality score; By generating evaluation descriptions based on multiple quality evaluation dimensions by an intelligent agent, the multiple first texts are rewritten to obtain multiple second texts; The plurality of first texts, the plurality of second texts, and the corresponding annotations are stored in the first reference set.

15. The text quality assessment method according to claim 11, characterized in that, The first correlation coefficient includes at least one of Spearman correlation coefficient, Pearson correlation coefficient and Kendall correlation coefficient.

16. The text quality evaluation method according to claim 1, characterized in that, The method further includes: Obtain a second benchmark set, which includes multiple second sample texts, which are published texts, and the annotation ranking of the multiple second sample texts is related to the comments of each second sample text; Obtain the quality scores of the plurality of second sample texts, and based on the quality scores of each second sample text, obtain the first evaluation ranking of the plurality of second sample texts; Obtain the third correlation coefficient between the first evaluation ranking and the labeled ranking of the second sample text; If the third correlation coefficient is less than or equal to the preset correlation coefficient, the identification prompt word is adjusted.

17. The text quality evaluation method according to claim 1, characterized in that, The method further includes: Obtain a third benchmark set, which includes multiple third sample texts, which are published texts, and the annotation ranking of the multiple third sample texts is related to the delivery performance index of each second sample text; Obtain the quality scores of the plurality of third sample texts, and based on the quality scores of each third sample text, obtain a second evaluation ranking of the plurality of third sample texts; Obtain the fourth correlation coefficient between the second evaluation ranking and the labeled ranking of the plurality of third sample texts; If the fourth correlation coefficient is less than or equal to the preset correlation coefficient, the identification prompt word is adjusted.

18. The text quality evaluation method according to claim 1, characterized in that, The method further includes: The second intelligent agent obtains the second degree of deviation between the text segment of the candidate question item and the evaluation description of the corresponding quality evaluation dimension based on the evaluation description of the multiple quality evaluation dimensions and the candidate question item; The third verification is performed on the candidate question item by the second intelligent agent. The third verification refers to verifying the deviation between the second deviation degree and the first deviation degree of the candidate question item. Wherein, the deviation between the second deviation degree of the target problem item and the first deviation degree is less than or equal to a preset deviation degree.

19. A text quality evaluation device, characterized in that, The device includes: The identification module is configured to identify candidate problem items in the text based on the text to be evaluated and identification prompts by a first intelligent agent. The identification prompts include evaluation descriptions of multiple quality evaluation dimensions. The candidate problem items include a text segment in the text, the quality evaluation dimension of the text segment, and a first degree of deviation between the text segment and the evaluation description of the corresponding quality evaluation dimension. The first verification module is configured to perform a first verification and a second verification on the candidate question item through a second intelligent agent. The first verification refers to verifying whether the text segment in the candidate question item exists in the text, and the second verification refers to verifying whether the quality evaluation dimension of the text segment in the candidate question item is correct. The first acquisition module is configured to acquire the quality score of the text based on the target issue item and the scoring rule. The target issue item is a candidate issue item that passes both the first and second checks. The scoring rule indicates the score of the target issue item under different degrees of deviation.

20. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory used to store the executable program code of the processor; The processor is configured to execute the program code to implement the text quality assessment method as described in any one of claims 1 to 18.

21. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the text quality assessment method as described in any one of claims 1 to 18.

22. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the text quality evaluation method according to any one of claims 1 to 18.

Images