A content quality evaluation method for a large model agent based on double baselines

Abstract

A double-baseline-based large model agent generates a content quality evaluation method, comprising: obtaining human-generated data, and preprocessing and semantic segmentation to form a semantic segmented data sample; topic extraction is performed on the data segmented data sample to form a topic list; the data constituting the topic list is respectively used to generate content by an intelligent agent workflow and directly generate content by a large model to obtain an evaluated sample and a control sample; the semantic segmented data sample, the evaluated sample and the control sample are compared and scored; a sorting algorithm is selected to convert the scores of the samples after scoring and comparison; statistical verification and result analysis are performed. The present application provides a comprehensive, reliable and efficient solution for the quality evaluation of large model agent generated content. It not only can accurately evaluate the content quality, but also provides strong support for the continuous optimization of the intelligent agent system.

Description

TECHNICAL FIELD

[0001] The present application relates to a content quality evaluation method. In particular, it relates to a content quality evaluation method for large model intelligent agents based on double baselines. BACKGROUND

[0002] In today's rapidly developing field of artificial intelligence and natural language processing, large models, including large language models and multi-modal large models, and intelligent agent systems based on them have shown great potential in content generation. These systems can generate various types of content, including text, code, image descriptions, etc., and play an increasingly important role in academic research, commercial applications and daily life. However, with the widespread application of these technologies, how to accurately and objectively evaluate the quality of intelligent agent-generated content has become a key problem that needs to be solved.

[0003] Traditional content evaluation methods mainly rely on manual evaluation, which is not only time-consuming and labor-intensive, but also often difficult to ensure consistency and objectivity when faced with large-scale and diverse content. In recent years, some research has attempted to use machine learning methods to automate the content quality evaluation process, but these methods are usually only applicable to specific types of content or specific evaluation dimensions, lacking universality and comprehensiveness.

[0004] On the other hand, with the outstanding performance of large language models in various tasks, some research has begun to explore the use of large models to evaluate content quality. This approach shows promising prospects, as large models have strong language understanding and analysis capabilities, allowing for comprehensive evaluation of content from multiple dimensions. However, this approach also faces some challenges, such as how to ensure the consistency and reliability of the evaluation, and how to avoid the influence of the model's own bias on the evaluation results.

[0005] In addition, existing evaluation methods often lack effective benchmarking. When evaluating intelligent agent-generated content, we not only need to know the absolute quality of the content, but also need to understand how it performs relative to human-created content and simple AI-generated content. This multi-dimensional comparison can provide more valuable guidance for the improvement of intelligent agent systems.

[0006] Therefore, there is an urgent need for a content quality evaluation method that can take advantage of the capabilities of large language models while overcoming the limitations of existing methods. This method should be applicable to multiple types of content, provide comprehensive evaluation from multiple dimensions, ensure the consistency and reliability of the evaluation, provide a comparative benchmark relative to human-created and simple AI-generated content, and be able to quantify the evaluation results and conduct statistical verification. SUMMARY

[0007] The technical problem to be solved by the present application is to overcome the shortcomings of the prior art and provide a big model agent generated content quality evaluation method based on double baseline, which can not only accurately evaluate content quality but also provide strong support for the continuous optimization of intelligent agent systems.

[0008] The technical solution adopted by the present application is: a big model agent generated content quality evaluation method based on double baseline, comprising the following steps:

[0009] 1) obtaining human-generated data, preprocessing and semantic segmentation, and constructing semantic segmented data samples;

[0010] 2) extracting topics from the data segmented data samples to construct a topic list;

[0011] 3) generating content using an intelligent agent workflow and directly generating content using a big model for the data constituting the topic list to obtain the evaluated samples and control samples;

[0012] 4) comparing and scoring the semantic segmented data samples, the evaluated samples and the control samples;

[0013] 5) selecting a sorting algorithm to convert the scores of the samples after scoring and comparison;

[0014] 6) statistical verification and result analysis.

[0015] The beneficial effects of the big model agent generated content quality evaluation method based on double baseline of the present application are as follows:

[0016] Firstly, the method of the present application exhibits excellent comprehensiveness and universality. By using a big model for topic extraction, content generation and scoring, it can be applied to various types of content, including text, code and image description, etc. The scoring criteria cover multiple dimensions of content quality, ensuring the comprehensiveness of the evaluation and providing a solid foundation for comprehensive evaluation of content quality.

[0017] Secondly, the innovation of the method of the present application lies in the introduction of a double baseline comparison mechanism. By taking human-generated content and big model directly generated content as double baseline, the method of the present application not only evaluates the absolute quality of intelligent agent generated content, but also provides a comparative benchmark relative to human level and simple AI generation level. This multi-dimensional comparison provides more valuable guidance for the improvement of intelligent agent systems and helps to accurately locate the performance level of intelligent agent systems.

[0018] In terms of consistency and reliability, the method of the present application adopts the strategy of multiple generation, multiple evaluation and cross-evaluation. This design effectively reduces random errors and position bias in the evaluation process, significantly improving the consistency and reliability of the evaluation results. At the same time, by using the sorting algorithm to convert the relative score into a standardized absolute value, the present application realizes the quantification of the evaluation results. Combined with the statistical verification step, it further ensures the scientificity and reliability of the evaluation method, providing a solid data support for the evaluation results.

[0019] Another major advantage of the method of the present application is its flexibility and scalability. The modular design allows each step to be adjusted and optimized according to specific needs. For example, the scoring criteria can be adjusted according to the characteristics of different fields, or different large models can be used to adapt to specific types of content. This flexibility enables the method of the present application to adapt to the evaluation needs of different scenarios, with broad application potential.

[0020] In terms of efficiency, compared with traditional manual evaluation methods, the present application greatly improves the efficiency of evaluation by utilizing the capabilities of large language models. It can complete large-scale content quality evaluation in a short time, making rapid iteration and large-scale application possible. In addition, although the method of the present application allows human design to be added in the construction of scoring criteria, it mainly relies on the capabilities of large models, which to some extent reduces the influence of human bias on the evaluation results, improving the objectivity of the evaluation.

[0021] In summary, the method of the present application provides a comprehensive, reliable and efficient solution for the quality evaluation of large model agent generated content. It not only can accurately evaluate the quality of content, but also provides strong support for the continuous optimization of agent systems. This innovative evaluation method is expected to play an important role in the field of artificial intelligence content generation and evaluation, promoting the progress and expansion of related technologies. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of the method of the present application. DETAILED DESCRIPTION

[0023] The method of the present application will be described in detail below in conjunction with the embodiments and drawings.

[0024] As shown in Figure 1 , the method of the present application includes the following steps:

[0025] 1) Obtain human-generated data, and pre-process and semantically segment it to form a semantically segmented data sample; including:

[0026] (1.1) Obtain data of the same type as the content generated by the agent to be evaluated to form a reference data set, and the data form is text or image or audio or video, such as a scientific paper, etc.

[0027] (1.2) Pre-process the data in the obtained reference data set, and the pre-processing includes operations such as format unification, deduplication, and cleaning of the data.

[0028] (1.3) Semantically segment the pre-processed data according to the evaluation requirements to facilitate subsequent topic extraction and evaluation, and the segmentation is performed manually, or according to chapter titles using regular expressions, or using LDA topic modeling technology to assist in segmentation, to obtain a semantically segmented data sample.

[0029] 2) Topic extraction on the segmented data sample to form a topic list; including:

[0030] (2.1) Apply more than one large model to the semantically segmented data sample to perform more than one topic extraction using a prompt word template, and only one topic extraction directly enters (2.3).

[0031] The large model refers to a generative artificial intelligence model, and the model type, whether it is multi-modal, parameter size, language adaptation, deployment platform, and implementation scheme are not required. The prompt word content in the process of using the large model is not required.

[0032] The prompt word template described herein is, for example:

[0033] For the following text, extract the single topic described therein and return it in JSON format

[0034] {to be extracted text data}

[0035] Output format:

[0036] {"topic": {extracted topic}}

[0037] (2.2) When multiple extractions are performed, concatenate all the extraction results and integrate them by the large model to obtain integrated topics that enter (2.3).

[0038] (2.3) Concatenate all the topics extracted from the data sample to form a topic list.

[0039] 3) Use the agent workflow generated content and the large model directly generated content to form a topic list to obtain the evaluated sample and the control sample; wherein:

[0040] The content generated by the intelligent agent workflow is: based on the topics in the topic list, the intelligent agent to be evaluated generates structured or unstructured data output by autonomously planning and executing tasks more than once, as the sample to be evaluated;

[0041] The intelligent agent generation method is content creation using a large model, and does not require specific implementation schemes, automation levels, programming languages, APIs or libraries.

[0042] The content generated directly by the large model is the direct generation of content by the large model based on the same topic one or more times, as the control sample.

[0043] 4) Compare and score the semantically segmented data sample, the evaluated sample and the control sample; including:

[0044] (4.1) Mix the semantically segmented data sample, the evaluated sample and the control sample to obtain the data sample to be scored;

[0045] (4.2) According to the content of the sample mixing, an artificial or large model constructs a scoring standard of more than one dimension, covering various aspects of content quality, and finally forms a scale containing a scoring standard of more than one dimension;

[0046] (4.3) Using a large model, according to the scoring standard containing more than one dimension, using a pairwise comparison strategy, scoring the sample-mixed data sample to be scored and more than once comparison and comparison of position to reduce the influence of potential position bias and random error.

[0047] 5) Select a sorting algorithm to convert the scores and compared samples; including:

[0048] Select Page Rerank sorting algorithm or ELO sorting algorithm to convert the results of pairwise comparison into standardized absolute scores (such as 0-10) corresponding to the data.

[0049] 6) Statistical verification and result analysis, including:

[0050] (6.1) According to the results of pairwise comparison, one or more of the within-group consistency test and transitivity test are performed.

[0051] (6.2) According to the standardized absolute scores corresponding to the data, the relative score difference percentage of the intelligent agent generated evaluated sample and the large model directly generated control sample relative to the semantically segmented sample is calculated, respectively, to evaluate the gap between the intelligent agent generated evaluated sample and the large model directly generated control sample relative to the semantically segmented sample;

[0052] The percentage calculation formula is as follows:

[0053] (Score of the sample generated by the agent or the control sample directly generated by the large model - sample score after data splitting) / sample score after data splitting * 100%.

[0054] Specific examples are given below:

[0055] 1. Data acquisition

[0056] In this embodiment, the quality of academic reviews is evaluated as an example. First, high-quality human-written review articles are obtained from Google Scholar. The selection criteria include being published in high-impact factor journals, having a high number of citations, and the authors being recognized experts in the field.

[0057] 1.1, Obtain human-generated data: 14 review articles that meet the above criteria are selected, covering multiple disciplines such as chemistry, physics, biology, etc.

[0058] 1.2, Data preprocessing: The obtained review articles are processed for format unification, converting all articles to pure text format, removing charts, references, and other non-text content. Use natural language processing tool NLTK for text cleaning, including removing special characters, unifying case, etc.

[0059] 1.3, Data segmentation: Each review article is segmented according to its original chapter structure. For articles without clear chapters, use the topic modeling technique LDA to assist in semantic segmentation, ensuring that each segmented paragraph contains complete semantic units.

[0060] 2. Topic extraction

[0061] 2.1, Extract topics using large models: Claude 3.5Sonnet model is selected for topic extraction. For each segmented paragraph, the following prompt is designed: "Please extract the core topic of this text and summarize it in one sentence." Repeat this process 5 times for each paragraph to increase the stability of the results.

[0062] 2.2, Topic integration: Input the results of the 5 extractions into Claude 3.5Sonnet and use the following prompt for integration: "The following are three topic summaries for the same text. Please integrate this information and give a most accurate and concise topic description." In this way, the final topic description for each paragraph is obtained.

[0063] 2.3, Topic splicing: Splice all topic extraction data samples to form a topic list.

[0064] 3. Content generation

[0065] 3.1, Agent Workflow Generation Content: Using a complex agent system built on Claude 3.5 Sonnet to generate review content. The system to be evaluated comes from arxiv:2407.20906, and this process is repeated 9 times for each topic, generating 9 versions of review paragraphs.

[0066] 3.2, Direct Generation of Large Models: Directly using Claude 3.5 Sonnet model, taking the topic description as input, to generate the corresponding review paragraph. The prompt is as follows: "Based on the following topic, generate an academic review paragraph, requiring accurate content, clear logic, and precise language." Similarly, we repeat this process 5 times for each topic.

[0067] 4. Scoring and Comparative Analysis

[0068] 4.1, Content Mixing: Mix the original human-written review paragraphs, agent-generated review paragraphs, and Claude 3.5 Sonnet directly generated review paragraphs together to form the content pool to be evaluated.

[0069] 4.2, Building Scoring Criteria: Based on the general requirements of the academic community for high-quality reviews and the suggestions of Claude 3.5 Sonnet, a scoring standard containing 27 dimensions is constructed, and the weight of each dimension is determined by subject experts. Scoring dimensions include: accuracy and completeness of citations, accuracy of information, appropriateness of professional terminology, avoidance of plagiarism and proper citation, clarity of logical structure, clarity of research method description, comprehensiveness of information, conciseness of expression, critical analysis of existing research, feasibility of proposed suggestions, fluency of language expression, identification and discussion of research limitations, inclusion of latest research findings, innovation of viewpoints, integration of interdisciplinary perspectives, logicality of argument, multi-angle analysis of problems, potential contribution to the field, potential practical application of research findings, proposal of new research directions or problems, quality of introduction and conclusion, rationality of method selection, rationality of chapter arrangement, reflection of current research hotspots, rigor of research design, timeliness and relevance of references, topic relevance.

[0070] 4.3, Pairwise Comparison: Using Claude 3.5 Sonnet for pairwise comparison. For each pair of paragraphs to be compared, the following prompt is designed: "Please compare the following two review texts, evaluate them from [list 27 scoring dimensions]. For each dimension, explain which paragraph performs better and give reasons. Finally, give an overall judgment on which review has higher quality."

[0071] To reduce positional bias, each pair of passages is compared twice, the second time with their positions swapped. In addition, a 5-fold cross-validation is used to increase the stability of the results.

[0072] 5. Score conversion

[0073] The relative comparison results are converted into absolute scores using the PageRank algorithm and normalized to the range of 0-10 scores. Specifically, we assign the highest PageRank value to the passage with a score of 10 and the lowest to the passage with a score of 0, and other passages are assigned scores in proportion.

[0074] 6. Statistical verification and result analysis

[0075] 6.1. Statistical verification:

[0076] Two key statistical methods are used to assess the reliability and consistency of the evaluation results: Intraclass Correlation Coefficient (ICC) and Transitive Consistency Ratio (TCR).

[0077] First, ICC is used to assess the consistency of multiple evaluation results. Specifically, the ICC value is calculated for each pair of content based on multiple scoring results. A two-way random effects model is chosen because the raters (i.e., repeated Claude 3.5 Sonnet) are randomly selected, and each rater scores all content pairs. The ICC value ranges from 0 to 1, where 0 indicates complete inconsistency and 1 indicates complete consistency. Generally, an ICC value greater than 0.75 is considered excellent consistency, 0.6 to 0.74 is considered good consistency, and 0.4 to 0.59 is considered average consistency. In the experiment, it was observed that the ICC values of most scoring dimensions were above 0.75, indicating that the evaluation method of the invention has high consistency and reliability.

[0078] Second, TCR is calculated to assess the transitive consistency of the scoring results. TCR measures the proportion of transitive consistency in all possible three-way comparisons. For example, if A is better than B, and B is better than C, then A should be better than C. The value of TCR is also between 0 and 1, and the higher the value, the better the transitive consistency. In the experiment, the TCR values of most scoring dimensions were above 0.75, further confirming the reliability of the evaluation method of the invention.

[0079] 6.2. Calculate the relative score difference percentage:

[0080] The average relative score difference percentage of Claude 3.5 Sonnet is 23.63%, the optimal paragraph score difference percentage is 30.79%, and the model baseline is -89.14%. Through analysis, we believe that the review generation content quality of the evaluated agent has reached or exceeded the level of artificial writing, far exceeding the model baseline, and showing the effectiveness of the workflow of the agent to be evaluated.

[0081] The above examples are only for illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made in accordance with the spirit and essence of the present application shall be covered within the protection scope of the present application.

[0082] While the application has been illustrated by a description of various embodiments, and while these embodiments have been described in considerable detail, it is not the intention that the application be limited or restricted to such detail. Additional advantages and modifications will readily occur to those skilled in the art. The application in its broader aspects is therefore not limited to the specific details, representative apparatus and method, and illustrative examples shown and described. Accordingly, departures can be made from such details without departing from the spirit or scope of the general inventive concept.

Claims

1. A method for generating content quality evaluation based on a double-baseline large model agent, characterized in that, Comprising, for example, the following steps: 1) Obtain human-generated data in the form of text or images or audio or video, and pre-process and semantically segment the data to form a semantically segmented data sample; 2) Extract topics from the semantically segmented data sample to form a topic list; comprising: (2.1) Apply one or more large models to the semantically segmented data sample in a prompt word template to extract topics one or more times, and only extract topics once to go directly to (2.3); (2.2) When multiple extractions are performed, splice all extraction results and integrate them by a large model to obtain integrated topics that go to (2.3); (2.3) Splice all topic-extracted data samples to form a topic list; 3) Generate content using an intelligent agent workflow based on the topics in the topic list and directly generate content using a large model to obtain evaluated samples and control samples; wherein: The content generated by the intelligent agent workflow is: based on the topics in the topic list, the intelligent agent to be evaluated plans and executes tasks autonomously, generates one or more times, and obtains structured or unstructured data output as the evaluated sample; The content directly generated by the large model is one or more times of directly generating content based on the same topic using a large model as a control sample; 4) Compare and score the semantically segmented data sample, the evaluated sample and the control sample; comprising: (4.1) Mix the semantically segmented data sample, the evaluated sample and the control sample to obtain a data sample to be scored; (4.2) According to the content of the sample mixture, construct one or more dimensions of scoring standards by artificial or large model, cover all aspects of content quality, and finally form a scale containing one or more dimensions of scoring standards; (4.3) Using a large model, according to the scoring standards containing one or more dimensions, using a pairwise comparison strategy, scoring the sample-mixed data sample to be scored and comparing and exchanging positions one or more times 5) Select a sorting algorithm to convert the scores and comparisons of the samples; comprising: selecting Page Rerank sorting algorithm or ELO sorting algorithm to convert the results of pairwise comparison into standardized absolute scores corresponding to the data; 6) Statistical verification and result analysis; comprising: (6.1) According to the results of pairwise comparison, one or more of the following tests are performed: inter-group consistency test and transitivity test; (6.2) According to the standardized absolute scores corresponding to the data, calculate the relative score difference percentage of the evaluated sample generated by the intelligent agent and the control sample directly generated by the large model relative to the semantically segmented sample, and evaluate the gap between the evaluated sample generated by the intelligent agent and the control sample directly generated by the large model relative to the semantically segmented sample; The percentage calculation formula is as follows: (The score of the evaluated sample generated by the intelligent agent or the control sample directly generated by the large model - the score of the data segmented sample) / the score of the data segmented sample * 100%.

2. The method of claim 1, wherein, Step 1) comprises: (1.1) Obtain a reference dataset of the same type of content generated by the agent to be evaluated, in the form of text or image or audio or video; (1.2) Preprocess the data in the obtained reference dataset, including format unification, deduplication, and cleaning operations; (1.3) Perform semantic segmentation on the preprocessed data according to the evaluation requirements, either manually or using regular expressions based on chapter titles, or using LDA topic modeling techniques to assist in segmentation, to obtain semantic segmented data samples.

3. The method of claim 1, wherein, The aspects of content quality covered in step (4.2) include: accuracy and completeness of citations, accuracy of information, appropriateness of professional terminology use, avoidance of plagiarism and proper citation, clarity of logical structure, clarity of research method description, comprehensiveness of information, conciseness of expression, critical analysis of existing research, feasibility of proposed suggestions, fluency of language expression, identification and discussion of research limitations, inclusion of latest research findings, innovation of viewpoints, integration of interdisciplinary perspectives, logic of argumentation, multi-angle analysis of problems, potential contribution to the field, potential practical application of research results, proposal of new research directions or problems, quality of introduction and conclusion, reasonableness of method selection, reasonableness of chapter arrangement, reflection of current research hotspots, rigor of research design, timeliness and relevance of references, thematic relevance.

Images