Large model agent generation content quality evaluation method based on double baselines

Abstract

A large model agent generation content quality evaluation method based on double baselines comprises the steps that data generated by human beings is acquired, preprocessing and semantic segmentation are carried out, and data samples after semantic segmentation are formed; performing theme extraction on the data samples after data segmentation to form a theme list; the content of the data forming the theme list is generated through an intelligent agent workflow and the content of the data forming the theme list is directly generated through a large model, and an evaluated sample and a contrast sample are obtained; comparing and scoring the data sample after semantic segmentation, the evaluated sample and the contrast sample; selecting a sorting algorithm to carry out score conversion on the graded and compared samples; and performing statistical verification and result analysis. According to the method, a comprehensive, reliable and efficient solution is provided for quality evaluation of the generation content of the large model agent. The method not only can accurately evaluate the content quality, but also provides powerful support for continuous optimization of an intelligent agent system.

Description

Technical Field

[0001] The present invention relates to a content quality assessment method, and more particularly to a content quality assessment method for large model intelligent agent generation based on dual baselines. Background Art

[0002] With the rapid development of artificial intelligence and natural language processing technology, large models, including large language models and multimodal large models, and intelligent agent systems built on them have shown great potential in the field of content generation. These systems can generate various types of content, including text, code, image descriptions, etc., and play an increasingly important role in academic research, commercial applications, and daily life. However, with the widespread application of these technologies, how to accurately and objectively evaluate the quality of content generated by intelligent agents has become a key issue that needs to be solved urgently.

[0003] Traditional content evaluation methods mainly rely on manual judgment, which is not only time-consuming and labor-intensive, but also often difficult to ensure consistency and objectivity in the face of large-scale and diverse content. In recent years, some studies have attempted to use machine learning methods to automate the content quality evaluation process, but these methods usually only target specific types of content or specific evaluation dimensions, lacking versatility and comprehensiveness.

[0004] On the other hand, as large language models have demonstrated outstanding capabilities in various tasks, some studies have begun to explore the use of large models to evaluate content quality. This approach shows promising prospects because large models have powerful language understanding and analysis capabilities and can comprehensively judge content from multiple dimensions. However, this approach also faces some challenges, such as how to ensure the consistency and reliability of the evaluation, and how to avoid the impact of the large model's own bias on the evaluation results.

[0005] In addition, existing evaluation methods often lack effective benchmark comparisons. When evaluating agent-generated content, we need to know not only the absolute quality of the content, but also how it performs relative to human-generated content and simple AI-generated content. This multi-faceted comparison can provide more valuable guidance for the improvement of agent systems.

[0006] Therefore, there is an urgent need for a content quality assessment method that can comprehensively utilize the capabilities of large language models while overcoming the limitations of existing methods. This method should be applicable to various types of content, conduct comprehensive assessments from multiple dimensions, ensure the consistency and reliability of the assessment, provide a comparative benchmark relative to human creation and simple AI-generated content, and be able to quantify the assessment results and perform statistical verification. Summary of the invention

[0007] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a large-model intelligent agent generation content quality assessment method based on dual baselines, which can not only accurately evaluate content quality but also provide strong support for the continuous optimization of the intelligent agent system.

[0008] The technical solution adopted by the present invention is: a method for evaluating the quality of content generated by a large model agent based on dual baselines, comprising the following steps:

[0009] 1) Obtain human-generated data, perform preprocessing and semantic segmentation, and form semantically segmented data samples;

[0010] 2) Extract topics from the data samples after data segmentation to form a topic list;

[0011] 3) Using the agent workflow to generate content and using the large model to directly generate content for the data constituting the topic list, respectively, to obtain evaluated samples and control samples;

[0012] 4) Compare and score the semantically segmented data samples, evaluated samples, and control samples;

[0013] 5) Select the sorting algorithm to convert the scores of the scored and compared samples;

[0014] 6) Statistical verification and result analysis.

[0015] The beneficial effects of the dual-baseline large model agent generated content quality assessment method of the present invention are as follows:

[0016] First, the method of the present invention demonstrates excellent comprehensiveness and versatility. By using a large model for topic extraction, content generation and scoring, it can be applied to various types of content, including text, code and image descriptions. The scoring criteria cover multiple dimensions of content quality, ensuring the comprehensiveness of the evaluation and providing a solid foundation for the comprehensive evaluation of content quality.

[0017] Secondly, the innovation of the method of the present invention lies in the introduction of a dual baseline comparison mechanism. By using human-generated content and large model-directed content as dual baselines, the method of the present invention not only evaluates the absolute quality of the content generated by the agent, but also provides a comparative benchmark relative to the human level and the simple AI generation level. This multi-faceted comparison provides more valuable guidance for the improvement of the agent system and helps to accurately locate the performance level of the agent system.

[0018] In terms of consistency and reliability of evaluation, the method of the present invention adopts a strategy of multiple generation, multiple evaluation and cross-evaluation. This design effectively reduces random errors and positional deviations in the evaluation process, and significantly improves the consistency and reliability of the evaluation results. At the same time, by using a sorting algorithm to convert relative scores into standardized absolute scores, the present invention achieves quantification of the evaluation results. Combined with the statistical verification step, the scientificity and reliability of the evaluation method are further ensured, providing solid data support for the evaluation results.

[0019] Another major advantage of the method of the present invention is its flexibility and scalability. The modular design allows each step to be adjusted and optimized according to specific needs. For example, the scoring criteria can be adjusted according to the characteristics of different fields, or different large models can be used to adapt to specific types of content. This flexibility enables the method of the present invention to adapt to the evaluation needs of different scenarios and has a wide range of application potential.

[0020] In terms of efficiency, compared with traditional manual evaluation methods, this method greatly improves the efficiency of evaluation by utilizing the capabilities of large language models. It can complete the quality evaluation of large-scale content in a short period of time, making it possible for rapid iteration and large-scale application. In addition, although the method of the present invention allows the addition of manual design to the construction of scoring criteria, it mainly relies on the capabilities of large models, which to a certain extent reduces the impact of human bias on the evaluation results and improves the objectivity of the evaluation.

[0021] In summary, the dual-baseline-based large-model agent-generated content quality assessment method of the present invention provides a comprehensive, reliable, and efficient solution for the quality assessment of large-model agent-generated content. It can not only accurately assess the quality of content, but also provide strong support for the continuous optimization of the agent system. This innovative assessment method is expected to play an important role in the field of artificial intelligence content generation and assessment, and promote the advancement of related technologies and the expansion of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of a method for evaluating content quality generated by a large model agent based on dual baselines according to the present invention. DETAILED DESCRIPTION

[0023] The following is a detailed description of a dual-baseline based large model agent generated content quality assessment method of the present invention in conjunction with the embodiments and drawings.

[0024] like Figure 1 As shown, a dual-baseline large model agent generated content quality assessment method of the present invention comprises the following steps:

[0025] 1) Obtain human-generated data, perform preprocessing and semantic segmentation, and form semantically segmented data samples; including:

[0026] (1.1) Obtaining data of the same type as the content generated by the agent to be evaluated to form a reference data set, wherein the data is in the form of text, image, audio, or video, such as scientific papers;

[0027] (1.2) Preprocessing the data in the acquired reference data set, including format unification, deduplication, and cleaning of the data;

[0028] (1.3) The preprocessed data is semantically segmented according to the evaluation requirements to facilitate subsequent topic extraction and evaluation. The segmentation is performed manually, or regular expressions are used to segment according to the chapter titles, or LDA topic modeling technology is used to assist segmentation to obtain semantically segmented data samples.

[0029] 2) Extract topics from the data samples after data segmentation to form a topic list; including:

[0030] (2.1) Apply more than one large model, substitute the semantically segmented data samples into the prompt word template to perform more than one topic extraction, and directly enter (2.3) if only one topic extraction is performed;

[0031] The big model mentioned here refers to a generative artificial intelligence model. There are no requirements on the model type, whether it is multimodal, the size of parameters, the adaptation language, the deployment platform, and the implementation plan; there are no requirements on the content of the prompt words in the process of using the big model.

[0032] Here is a prompt word template, for example:

[0033] For the following text, extract the single topic mentioned in it and return it in JSON format

[0034] {Text data to be extracted}

[0035] Output format:

[0036] {"Topic":{extracted topic}}

[0037] (2.2) After multiple extractions are performed, all the extraction results are spliced ​​together and fused by the large model to obtain the integrated theme and enter (2.3);

[0038] (2.3) After all themes are extracted, the data samples are spliced ​​into themes to form a theme list.

[0039] 3) Using the agent workflow and the large model to generate content for the data constituting the topic list, respectively, to obtain evaluated samples and control samples; wherein:

[0040] The above-mentioned generating content by using the agent workflow is: the agent to be evaluated is based on the topics in the topic list, and the agent autonomously plans and executes tasks, and generates more than one time to obtain structured or unstructured data output as the sample to be evaluated;

[0041] The method for generating intelligent agents is to create content using a large model, without making any requirements on the specific implementation scheme, degree of automation, programming language used, API or library called.

[0042] The direct generation of content using a large model is to use the large model to directly generate content based on the same topic once or multiple times as a control sample.

[0043] 4) Compare and score the semantically segmented data samples, the evaluated samples, and the control samples; including:

[0044] (4.1) Mixing the semantically segmented data samples, the evaluated samples, and the control samples to obtain the data samples to be scored;

[0045] (4.2) Based on the mixed content of the sample, a scoring standard with more than one dimension is constructed manually or by a large model, covering all aspects of content quality, and finally forming a scale containing a scoring standard with more than one dimension;

[0046] (4.3) Using a large model, a pairwise comparison strategy is adopted according to the scoring criteria containing more than one dimension to score the data samples to be scored after the samples are mixed, and more than one comparison and comparison with exchanged positions are performed to reduce the impact of potential position bias and random errors.

[0047] 5) Select the sorting algorithm to convert the scores of the scored and compared samples; including:

[0048] Select Page Rerank sorting algorithm or ELO sorting algorithm to convert the results of pairwise comparison into standardized absolute scores corresponding to the data (such as 0-10 points).

[0049] 6) Statistical verification and results analysis, including:

[0050] (6.1) Based on the results of pairwise comparisons, perform one or more of the following tests: intra-group consistency test and transitivity test.

[0051] (6.2) Based on the standardized absolute scores corresponding to the data, the percentage of the relative score difference between the evaluated samples generated by the agent and the control samples directly generated by the large model relative to the samples after semantic segmentation is calculated, and the gap between the evaluated samples generated by the agent and the control samples directly generated by the large model relative to the samples after semantic segmentation is evaluated;

[0052] The percentage calculation formula is as follows:

[0053] (The score of the evaluated sample generated by the intelligent agent or the control sample directly generated by the large model - the score of the sample after data segmentation) / the score of the sample after data segmentation * 100%.

[0054] Here are some specific examples:

[0055] 1. Data Acquisition

[0056] In this embodiment, the quality of academic reviews is evaluated as an example. First, high-quality human-written review articles are obtained from Google Scholar. The selection criteria include publication in high-impact journals, high citation counts, and authors who are recognized experts in the field.

[0057] 1.1. Obtaining human-generated data: We selected 14 review articles that met the above criteria, covering multiple disciplines such as chemistry, physics, and biology.

[0058] 1.2. Data preprocessing: The obtained review articles were formatted in a unified way, all articles were converted into plain text format, and non-text content such as charts and references were removed. The natural language processing tool NLTK was used for text cleaning, including removing special characters and unifying upper and lower case.

[0059] 1.3. Data segmentation: Each review article is segmented according to its original chapter structure. For articles without clear chapters, the topic modeling technology LDA is used to assist in semantic segmentation to ensure that each segmented paragraph contains complete semantic units.

[0060] 2. Topic Extraction

[0061] 2.1. Extracting topics using a large model: The Claude 3.5Sonnet model was used for topic extraction. For each segmented paragraph, the following prompt words were designed: "Please extract the core topic of this text and summarize it in one sentence." This process was repeated 5 times for each paragraph to increase the stability of the results.

[0062] 2.2. Theme integration: The results of the five extractions were input into Claude 3.5Sonnet and integrated using the following prompt words: "The following are three theme summaries of the same paragraph. Please integrate this information and give the most accurate and concise theme description." In this way, the final theme description of each paragraph was obtained.

[0063] 2.3. Theme splicing: After all themes are extracted, the data samples are spliced ​​to form a theme list.

[0064] 3. Content Generation

[0065] 3.1. Agent workflow to generate content: A complex agent system based on Claude 3.5Sonnet is used to generate review content. The system to be evaluated comes from arxiv:2407.20906. This process is repeated 9 times for each topic to generate 9 versions of the review paragraph.

[0066] 3.2. Large model directly generates content: directly use the Claude 3.5Sonnet model, take the topic description as input, and generate the corresponding review paragraph. The prompt is as follows: "Please generate an academic review paragraph based on the following topic, which requires accurate content, clear logic, and rigorous language." Similarly, we repeat this process 5 times for each topic.

[0067] 4. Rating and comparative analysis

[0068] 4.1. Content mixing: The review paragraphs written by original humans, the review paragraphs generated by the agent, and the review paragraphs directly generated by Claude3.5Sonnet are mixed together to form a content pool to be evaluated.

[0069] 4.2. Constructing the scoring criteria: Combining the general requirements of the academic community for high-quality reviews and the suggestions of Claude 3.5Sonnet, a scoring criteria with 27 dimensions was constructed, and the weight of each dimension was determined by subject experts. The scoring dimensions include: accuracy and completeness of citations, accuracy of information, appropriate use of professional terms, avoidance of plagiarism and appropriate citations, clarity of logical structure, clarity of description of research methods, comprehensiveness of information, conciseness of expression, critical analysis of existing research, feasibility of the proposed suggestions, fluency of language expression, identification and discussion of research limitations, inclusion of the latest research results, innovation of viewpoints, integration of interdisciplinary perspectives, logic of argumentation, multi-angle analysis of problems, potential contribution to the field, practical application potential of research results, proposal of new research directions or problems, quality of introduction and conclusion, rationality of method selection, rationality of chapter arrangement, reflection of current research hotspots, rigor of research design, timeliness and relevance of references, and relevance of the subject.

[0070] 4.3. Paired comparison: Claude 3.5Sonnet was used for paired comparison. For each paragraph to be compared, the following prompt was designed: "Please compare the following two review texts and evaluate them from these aspects [list 27 rating dimensions]. For each dimension, please explain which paragraph performs better and give reasons. Finally, please give an overall judgment on which review has higher quality."

[0071] To reduce position bias, each pair of paragraphs was compared twice, with their positions swapped in the second comparison. In addition, five repeated evaluations were used to increase the stability of the results.

[0072] 5. Score conversion

[0073] The PageRank algorithm is used to convert the relative comparison results into absolute scores and normalize them to a range of 0 to 10. Specifically, we assign 10 points to the paragraph with the highest PageRank value and 0 points to the paragraph with the lowest PageRank value, and the scores of other paragraphs are assigned proportionally.

[0074] 6. Statistical verification and results analysis

[0075] 6.1 Statistical verification:

[0076] Two key statistical methods were used to assess the reliability and consistency of the assessment results: Intraclass Correlation Coefficient (ICC) and Transitive Consistency Ratio (TCR).

[0077] First, ICC is used to evaluate the consistency of multiple evaluation results. Specifically, the ICC value is calculated for the multiple scoring results of each pair of content. A two-way random effects model is selected because the scorers (i.e., repeated Claude 3.5Sonnet) are randomly selected and each scorer scores all content pairs. The ICC value ranges from 0 to 1, where 0 indicates complete inconsistency and 1 indicates complete consistency. Generally, an ICC value greater than 0.75 is considered to be excellent consistency, 0.6 to 0.74 is considered to be good consistency, and 0.4 to 0.59 is considered to be general consistency. In the experiment, it was observed that the ICC values ​​of most scoring dimensions were above 0.75, indicating that the evaluation method of the present invention has a high degree of consistency and reliability.

[0078] Secondly, TCR is calculated to evaluate the transitive consistency of the scoring results. TCR measures the proportion of all possible triple comparisons that meet transitivity. For example, if A is better than B, and B is better than C, then A should be better than C. The value of TCR is also between 0 and 1, and the higher the value, the better the transitive consistency. In the experiment, the TCR values ​​of most scoring dimensions are above 0.75, which further confirms the reliability of the evaluation method of the present invention.

[0079] 6.2. Calculate the relative score difference percentage:

[0080] After calculation, the average relative score difference percentage of Claude3.5Sonnet is 23.63%, the best paragraph score difference percentage is 30.79%, and the model baseline is -89.14%. Through analysis, we believe that the quality of the review generated by the evaluated agent has reached or exceeded the level of manual writing, far exceeding the model baseline, showing the effectiveness of the evaluated agent workflow.

[0081] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

[0082] Although the entirety of the present invention has been illustrated by the description of various embodiments, and although these embodiments have been described in considerable detail, the applicant does not intend that the scope of the appended claims be limited or in any way restricted to such details. Additional advantages and modifications will be readily apparent to those skilled in the art. Therefore, the present invention in its broader aspects is not limited to the specific details, representative devices and methods, and illustrative examples shown and described. Therefore, deviations from such details may be made without departing from the spirit or scope of the overall inventive concept.

Claims

1. A dual-baseline large-model agent generated content quality assessment method, characterized in that: Including the following steps: 1) Obtain human-generated data, perform preprocessing and semantic segmentation, and form semantically segmented data samples; 2) Extract topics from the data samples after data segmentation to form a topic list; 3) Using the agent workflow to generate content and using the large model to directly generate content for the data constituting the topic list, respectively, to obtain evaluated samples and control samples; 4) Compare and score the semantically segmented data samples, evaluated samples, and control samples; 5) Select the sorting algorithm to convert the scores of the scored and compared samples; 6) Statistical verification and result analysis.

2. According to the dual-baseline large model agent generated content quality assessment method according to claim 1, it is characterized in that: Step 1) includes: (1.1) obtaining data of the same type as the content generated by the agent to be evaluated to form a reference data set, wherein the data is in the form of text, image, audio, or video; (1.2) Preprocessing the data in the acquired reference data set, including format unification, deduplication, and cleaning of the data; (1.3) The preprocessed data is semantically segmented according to the evaluation requirements. The segmentation is performed manually, or regular expressions are applied according to the chapter titles, or LDA topic modeling technology is used to assist segmentation to obtain semantically segmented data samples.

3. According to the dual-baseline large model agent generated content quality assessment method of claim 1, it is characterized by: Step 2) includes: (2.1) Apply more than one large model, substitute the semantically segmented data samples into the prompt word template to perform more than one topic extraction, and directly enter (2.3) if only one topic extraction is performed; (2.2) After multiple extractions are performed, all the extraction results are spliced ​​together and fused by the large model to obtain the integrated theme and enter (2.3); (2.3) After all themes are extracted, the data samples are spliced ​​into themes to form a theme list.

4. According to the dual-baseline large model agent generated content quality assessment method of claim 1, it is characterized in that: In step 3): The above-mentioned generating content by using the agent workflow is: the agent to be evaluated is based on the topics in the topic list, and the agent autonomously plans and executes tasks, and generates more than one time to obtain structured or unstructured data output as the sample to be evaluated; The direct generation of content using a large model is to use the large model to directly generate content based on the same topic once or multiple times as a control sample.

5. According to the dual-baseline large model agent generated content quality assessment method of claim 1, it is characterized by: Step 4) includes: (4.1) Mixing the semantically segmented data samples, the evaluated samples, and the control samples to obtain the data samples to be scored; (4.2) Based on the mixed content of the sample, a scoring standard with more than one dimension is constructed manually or by a large model, covering all aspects of content quality, and finally forming a scale containing a scoring standard with more than one dimension; (4.3) Using a large model, based on a scoring criterion containing more than one dimension, a pairwise comparison strategy is adopted to score the data samples to be scored after the samples are mixed, and to compare more than once and compare the positions of the samples in a swapped manner.

6. According to the dual-baseline large model agent generated content quality assessment method of claim 5, it is characterized in that: The aspects of content quality described in step (4.2) include: accuracy and completeness of citations, accuracy of information, appropriate use of professional terms, avoidance of plagiarism and appropriate citations, clarity of logical structure, clarity of description of research methods, comprehensiveness of information, conciseness of expression, critical analysis of existing research, feasibility of proposed suggestions, fluency of language expression, identification and discussion of research limitations, inclusion of the latest research results, innovative ideas, integration of interdisciplinary perspectives, logic of argumentation, multi-angle analysis of problems, potential contribution to the field, practical application potential of research results, proposal of new research directions or problems, quality of introduction and conclusion, rationality of method selection, rationality of chapter arrangement, reflection of current research hotspots, rigor of research design, timeliness and relevance of references, and relevance of the topic.

7. According to the dual-baseline large model agent generated content quality assessment method of claim 1, it is characterized by: Step 5) includes: selecting the Page Rerank sorting algorithm or the ELO sorting algorithm, and converting the results of the pairwise comparison into standardized absolute scores corresponding to the data.

8. According to the dual-baseline large model agent generated content quality assessment method of claim 1, it is characterized by: Step 6) includes: (6.1) Based on the results of pairwise comparisons, perform one or more of the following tests: intra-group consistency test and transitivity test. (6.2) Based on the standardized absolute scores corresponding to the data, the percentage of the relative score difference between the evaluated samples generated by the agent and the control samples directly generated by the large model relative to the samples after semantic segmentation is calculated, and the gap between the evaluated samples generated by the agent and the control samples directly generated by the large model relative to the samples after semantic segmentation is evaluated; The percentage calculation formula is as follows: (The score of the evaluated sample generated by the intelligent agent or the control sample directly generated by the large model - the score of the sample after data segmentation) / the score of the sample after data segmentation * 100%.

Images