Multi-stage content quality evaluation method and system based on large language model

Abstract

The invention provides a multi-stage content quality evaluation method and system based on a large language model, and relates to the technical field of natural language processing. The method comprises the steps of obtaining a to-be-evaluated content data set and an evaluation index corresponding to a data type of the to-be-evaluated content data set; based on the evaluation indexes, utilizing a first large language model to carry out batch multi-dimensional coarse screening on the content data set, obtaining evaluation scores of all content data, sorting the evaluation scores, and filtering and retaining a plurality of content data with the top scores; an improved ELO dynamic scoring mechanism is introduced, a second large language model is used for conducting dynamic pairwise comparison sorting on the filtered content data, final ELO scores of all the content data are obtained and sorted, and an ELO score sorting result serves as a content quality sorting result. According to the method, a large language model is utilized to break through the cognitive depth bottleneck, evaluation objectivity is improved by relying on a dynamic game mechanism, two-stage architecture is combined to balance calculation efficiency and sorting precision, and finally a content quality evaluation universal solution capable of being rapidly adapted to multiple fields is formed.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a multi-stage content quality assessment method and system based on a large language model. Background Art

[0002] Content data is widely available on internet platforms, such as online reviews, creative proposal submissions, and design works. While this content provides vitality and diversity to the platforms, it also generates a large amount of low-quality, repetitive, or false content, making efficient and accurate quality assessment and screening a pressing issue.

[0003] In the related art, the existing content data evaluation methods are mainly divided into three types: comprehensive scoring using computable indicators, model prediction, and manual evaluation. Among them, the use of computable indicators for total score scoring, for example: CN104850642A discloses a network content quality evaluation method and device, which obtains the content quality characteristics of network content and calculates its characteristic score. Model prediction of content quality, for example: CN111428032A discloses a content quality evaluation method and device, electronic device, and storage medium, which uses a pre-trained model to filter vulgar content in multimedia. Manual evaluation is to call on evaluators or experts to evaluate specific content, or select data such as "usefulness likes" on the Internet as the basis for evaluation.

[0004] However, current internet platforms face the need for efficient quality assessment of massive amounts of content, especially when processing data such as online comments and creative submissions, which requires a balance between accuracy, cross-domain adaptability, and real-time performance. None of the aforementioned methods can meet the requirements of both real-time and cross-domain adaptability while considering deep semantics. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the shortcomings of the existing technology, the present invention provides a multi-stage content quality assessment method and system based on a large language model, which solves the technical problems of balancing accuracy, cross-domain adaptability and real-time performance.

[0007] (2) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A multi-stage content quality assessment method based on a large language model, including:

[0010] Obtain the content dataset to be evaluated and the evaluation indicators corresponding to its data type;

[0011] Based on the evaluation indicators, the content data set is subjected to batch multi-dimensional coarse screening using the first language model to obtain evaluation scores of all content data and sort them, and the content data with the highest scores are filtered and retained;

[0012] An improved ELO dynamic scoring mechanism is introduced, and the second largest language model is used to dynamically compare and sort the filtered content data in pairs, obtain the final ELO score of all content data and sort it, and use the ELO score sorting result as the content quality sorting result.

[0013] Preferably, before using the first large language model to perform multi-dimensional coarse screening on the content data set, a prompt template of the first large language model is first constructed, and the prompt template includes the role of the model and the evaluation index.

[0014] Preferably, after constructing the prompt template, a few-sample learning strategy is used to fine-tune the first language model; wherein the few-sample learning strategy refers to inputting multiple content data with quality labels as a single sample into the first language model, so that it returns a comparative evaluation score.

[0015] Preferably, the improved ELO dynamic scoring mechanism is introduced, and the second largest language model is used to dynamically compare and sort the filtered content data in pairs, obtain the final ELO scores of all content data and sort them, and use the ELO score sorting results as the content quality sorting results, including:

[0016] Initialize the ELO score of each piece of content data after filtering;

[0017] Determine whether the iteration termination condition is met. If so, output the final ELO scores of all content data and sort them. The ELO score sorting result is used as the content quality sorting result. Otherwise, repeat the following iterative steps in sequence:

[0018] According to the principle of selecting the smallest difference, we obtain two matching content data pairs based on ELO score sampling;

[0019] Using each content data pair as input to the second language model, determining the quality of the two content data in the content data pair, and quantifying the difference in content quality between the two content data;

[0020] Based on the content quality difference, the ELO scores of the two content data in the corresponding content data pair are updated.

[0021] Preferably, the iteration termination condition includes any one of standard deviation convergence, stability detection, and maximum number of iterations, or a combination of any several of them.

[0022] Preferably, a content data pair {A, B} is defined, and the update formula of the ELO score is expressed as:

[0023]

[0024] K=K base ×w×α

[0025]

[0026] E B =1-E A

[0027] in, are the ELO scores of content data A and content data B after updating; RA and RB are the ELO scores of content data A and content data B before updating; S A 、S B To judge the result score, if the content quality of content data A is better, then S A =1, S B = 0, if the content quality of content data B is better, then S A =0, S B =1, tie, both values are S A =S B =0.5; K is the fractional change rate coefficient, K base is the benchmark K value; w is the weight, and the greater the difference in content quality between content data A and content data B, the greater the value of w; α is the activity decay value; E A 、E B The expected win rate indicates the theoretical win rate based on the current content.

[0028] A multi-stage content quality assessment system based on a large language model, including:

[0029] An acquisition module is used to obtain the content data set to be evaluated and the evaluation indicators corresponding to its data type;

[0030] a primary screening module configured to perform a batch multi-dimensional coarse screening of the content data set based on the evaluation indicators and using the first language model, obtain evaluation scores of all content data, sort them, and filter out and retain content data with top scores;

[0031] The sorting module is used to introduce an improved ELO dynamic scoring mechanism, use the second largest language model to dynamically compare and sort the filtered content data, obtain the final ELO score of all content data and sort it, and use the ELO score sorting result as the content quality sorting result.

[0032] A storage medium stores a computer program for multi-stage content quality assessment based on a large language model, wherein the computer program enables a computer to execute the multi-stage content quality assessment method as described above.

[0033] An electronic device, comprising:

[0034] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including instructions for executing the multi-stage content quality assessment method as described above.

[0035] (3) Beneficial effects

[0036] The present invention provides a multi-stage content quality assessment method and system based on a large language model. Compared with the existing technology, it has the following advantages:

[0037] In this method, evaluation metrics corresponding to the content dataset to be evaluated and its data type are obtained. Based on these metrics, a first large language model is used to perform a batch, multi-dimensional coarse screening of the content dataset, obtaining and ranking the evaluation scores of all content data, and filtering out the top-scoring content data. An improved ELO dynamic scoring mechanism is introduced, and a second large language model is used to dynamically compare and sort the filtered content data, obtaining and ranking the final ELO scores of all content data, and using the ELO score ranking results as the content quality ranking results. This method utilizes a large language model to overcome the bottleneck of cognitive depth, relies on a dynamic game mechanism to enhance evaluation objectivity, and combines a two-stage architecture to balance computational efficiency and ranking accuracy, ultimately forming a universal solution for content quality assessment that can be rapidly adapted to multiple fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A block diagram of a multi-stage content quality assessment method based on a large language model provided by an embodiment of the present invention;

[0040] Figure 2 A flowchart of a multi-stage content quality assessment method based on a large language model provided by an embodiment of the present invention;

[0041] Figure 3A schematic diagram of a prompt template provided by an embodiment of the present invention;

[0042] Figure 4 Schematic diagram of the average scores given by evaluators under different evaluation methods. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0044] The embodiments of the present application solve the technical problem of balancing accuracy, cross-domain adaptability and real-time performance by providing a multi-stage content quality assessment method and system based on a large language model.

[0045] The technical solution in the embodiments of the present application is to solve the above technical problems, and the technical advantages are as follows:

[0046] (1) The embodiment of the present invention uses the large language model role scenario simulation to enable the large language model evaluator to have the cognitive framework of a domain expert, breaking through the previous limitation of only being able to process surface features (such as keyword frequency and text length), and realizing the essential modeling of abstract concepts such as "innovation" and "social value".

[0047] (2) The embodiment of the present invention introduces multiple reviews for simultaneous preliminary screening when using a large language model for evaluation, and enhances the model's understanding of quality dimensions through comparative learning.

[0048] (3) The present invention utilizes the ELO dynamic game mechanism to convert absolute scores into relative competitiveness indices, addressing the evaluation distortion problem caused by the distribution bias of manually annotated data. Furthermore, through multiple rounds of content comparison, a network of relative quality relationships between content is gradually established, overcoming the "score inflation" problem (e.g., a large amount of content is concentrated in a certain score range) caused by inconsistent annotation standards in traditional static scoring.

[0049] (4) The embodiment of the present invention adopts a two-stage architecture of "multi-dimensional coarse screening-dynamic comparison and refined evaluation" to achieve intelligent allocation of computing resources. Through parallel batch scoring of large language models, low-quality entries are quickly eliminated. In the refined evaluation stage, the present invention intelligently matches and dynamically generates battle strategies: adjacent content with small ELO score differences are preferentially compared to ensure that each round of comparison effectively improves the ranking accuracy. Compared with full random comparison, the number of required battle rounds is reduced.

[0050] (4) The present invention uses modular design to achieve rapid cross-domain migration of the evaluation system. It does not require retraining the model; it can be adapted to new domains simply by prompting engineering and parameter configuration.

[0051] In addition, the following terms are introduced in the embodiments of the present invention:

[0052] First, Large Language Models (LLMs), also known as large language models, are AI models designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, and sentiment analysis.

[0053] Note that this embodiment of the present invention does not limit the types of the first and second language models, nor does it limit whether they are of the same type. They can be any combination of mainstream large language models such as DeepSeek-v3 and GPT4o.

[0054] Second, the Elo Rating System (note that Elo here is the surname of the inventor of the system) is an evaluation method for measuring the level of various types of chess activities. It is a recognized authoritative method for evaluating chess levels today.

[0055] It should be noted that the improved ELO dynamic scoring mechanism in the embodiments of the present invention is based on the classic ELO game scoring model, combined with a content comparison and evaluation method driven by a large language model. Through a matching strategy that prioritizes scores based on proximity, a score adjustment mechanism based on large model feedback, and multiple convergence judgment criteria, it achieves higher accuracy and stability in ranking high-quality content. It is suitable for complex evaluation scenarios where the quality of content is unstructured and the evaluation dimensions are diverse.

[0056] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0057] Example 1:

[0058] like Figure 1 As shown, an embodiment of the present invention provides a multi-stage content quality assessment method based on a large language model, including:

[0059] S1. Obtain the content dataset to be evaluated and the evaluation indicators corresponding to its data type;

[0060] S2. Based on the evaluation indicators, use the first language model to perform batch multi-dimensional coarse screening on the content data set, obtain evaluation scores for all content data, sort them, and filter out and retain content data with top scores;

[0061] S3. Introduce an improved ELO dynamic scoring mechanism, use the second largest language model to dynamically compare and sort the filtered content data, obtain the final ELO score of all content data and sort it, and use the ELO score sorting result as the content quality sorting result.

[0062] The method provided by the embodiment of the present invention utilizes a large language model to break through the bottleneck of cognitive depth, relies on a dynamic game mechanism to improve the objectivity of evaluation, and combines a two-stage architecture to balance computing efficiency and sorting accuracy, ultimately forming a universal solution for content quality assessment that can be quickly adapted to multiple fields.

[0063] like Figure 2 As shown, Figure 2 The flowchart of the multi-stage content quality assessment method based on the large language model is disclosed. Figure 2 The following steps are described in detail:

[0064] In step S1, the content data set to be evaluated and the evaluation index corresponding to its data type are obtained.

[0065] This step obtains the content dataset to be evaluated. The content data type here can be online reviews, creative proposal content, design works, etc. After selecting the content data type, this step needs to construct the corresponding evaluation indicators based on the type.

[0066] In other words, this step does not limit the content data type, but usually requires obtaining / constructing different evaluation indicators to evaluate the content quality of the corresponding type of content data. For example:

[0067] For online review content, the indicators for evaluating its quality include the amount of review information, review relevance, and review persuasiveness.

[0068] For the content of creative proposals, the indicators for evaluating their quality include the novelty of the proposal, the strategic feasibility of the proposal, the environmental value of the proposal, etc.

[0069] For design works, the indicators for evaluating their quality include visual expression, functional realization, user experience and innovation.

[0070] Furthermore, the embodiment of the present invention will take online review content as an example to describe the details of the solution in detail. If the online review content is further limited to hotel user reviews on online travel platforms, the evaluation indicators in this step are set as {"Indicator 1: Persuasiveness: Whether the review has persuasive elements that can attract the interest of other readers.", "Indicator 2: Review information volume: The richness of vocabulary and the probability of the occurrence of non-common words reflect the depth of information.", "Indicator 3: Topic relevance: Whether the keywords contained in the review overlap with the keywords of the topic, indicating the relevance of the review to the topic being discussed. Check whether the review involves specific topics such as service, cleanliness, facilities, location, room condition, cost-effectiveness and catering."}.

[0071] In step S2, based on the evaluation index, the content data set is subjected to batch multi-dimensional coarse screening using the first language model to obtain and sort the evaluation scores of all content data, and filter and retain several content data with high scores.

[0072] Based on the evaluation index obtained in the previous step, this step first constructs a prompt template for the first language model. The prompt template includes the role of the model and the evaluation index. Its framework diagram is as follows: Figure 3 shown.

[0073] After constructing the prompt template, this step can also use a few-sample learning strategy to fine-tune the first language model; wherein the few-sample learning strategy refers to inputting multiple content data with quality labels as a single sample into the first language model, allowing it to return evaluation scores of multiple content data, so that the evaluation scores are comparative.

[0074] After clarifying all the contents of the prompt words based on the prompt template, this step uses the fine-tuned first language model to perform batch multi-dimensional coarse screening on the content data set, obtain the evaluation scores of all the content data and sort them, and filter out the content data with the highest scores.

[0075] The evaluation score here can be set to any integer in the range of 1-10.

[0076] Taking hotel user reviews on an online travel platform as an example, a feasible prompt word for the first language model (the italicized part) is:

[0077] Persona: You are a consumer browsing reviews of a product / service

[0078] User: Please evaluate the quality of consumer reviews based on the following indicators: {"Indicator 1: Persuasiveness: Does the review contain persuasive elements that can attract other readers' interest?", "Indicator 2: Review Information Content: The richness of the vocabulary and the probability of the presence of non-common words reflect the depth of the information.", "Indicator 3: Topic Relevance: Whether the keywords contained in the review overlap with the keywords of the topic, indicating the relevance of the review to the topic discussed. Check whether the review covers specific topics such as service, cleanliness, facilities, accommodation, room condition, value for money, and food."

[0079] First, read through all the comments through the thought chain, then think about the score of each dimension of each comment, give each content a score of 1-10 based on the evaluation of each indicator, and format the reply to form a json file with a certain structure:

[0080]

[0081] Now process the following data:

[0082] {“Comment 1”: “The experience of staying at this hotel was great! Check-in was fast at the front desk, and the staff were friendly and professional. They also provided helpful recommendations for nearby attractions. The room was spotlessly clean, and the facilities were all new. However, the breakfast variety was a bit limited, and it would be better if it could be more diverse.”, “Comment 2”: “xxxx”, “Comment 3”: “xxxx”, ..., “Comment 8”: “xxxx”}

[0083] In step S3, an improved ELO dynamic scoring mechanism is introduced, and the second largest language model is used to dynamically compare and sort the filtered content data in pairs, obtain the final ELO scores of all content data and sort them, and use the ELO score sorting results as the content quality sorting results.

[0084] Based on the rough screening, in order to further quantify the subtle differences between high-quality content, this step introduces an improved ELO dynamic scoring mechanism, which achieves refined ranking of content quality through pairwise comparison driven by a large language model. The relevant steps include:

[0085] S31. Initialize the ELO score of each piece of filtered content data.

[0086] Specifically, the filtered content dataset is defined as C = {c1, c2, ..., c M}, M represents the number of content data, and the i-th content data c i The initial ELO score Ri, then enters the iteration of the ELO score.

[0087] S32. Determine whether an iteration termination condition is met. The iteration termination condition here includes any one or any combination of the following:

[0088] Standard deviation convergence (for example, the standard deviation of the overall ELO score changes by less than 2% for three consecutive rounds);

[0089] Stability test (Top 20 content has no ranking change for two consecutive rounds);

[0090] Maximum number of iterations (N max =log2(M)×10).

[0091] If the iteration termination condition is met, the final ELO scores of all content data are output and sorted, and the ELO score sorting result is used as the content quality sorting result. Otherwise, the following iterative steps are repeated in sequence:

[0092] S101. Matching sampling: According to the principle of selecting the smallest difference, pairwise matching content data is obtained based on ELO score sampling.

[0093] Specifically, based on the current ELO score, the uncompared content pairs with the smallest difference are selected from the comparison set (c A ,c B ), that is, satisfy This strategy improves the distinguishing efficiency of model judgment by focusing on content with similar scores.

[0094] S102, model determination: using each content data pair as input to the second language model, determining the quality of the two content data in the content data pair, and quantifying the difference in content quality between the two.

[0095] Specifically:

[0096] Similar to the processing of the first language model, here we also first set the role of the second language model. Taking hotel user reviews on an online travel platform as an example, a possible prompt word for the second language model (the italicized part) is:

[0097] “Role: You are the quality reviewer. User: Now combine these indicators to evaluate which of the following two is of higher quality (review A, review B, tie), and by how much higher (slight difference, moderate difference, extreme difference). The indicators are {″Indicator 1: Persuasiveness: whether the review has persuasive elements that can attract the interest of other readers. ″, ″Indicator 2: Review information volume: the richness of vocabulary and the probability of the occurrence of non-common words, reflecting the depth of information. ″, ″Indicator 3: Topic relevance: whether the keywords contained in the review overlap with the keywords of the topic, indicating the relevance of the review to the topic discussed. Check whether the review covers specific topics such as service, cleanliness, facilities, location, room condition, value for money, and dining. ″} Now compare the following: {“Hotel Review A”: xxxxxxxx, “Hotel Review B”: xxxxxxxx}”

[0098] After fine-tuning the second language model, it is used to compare content data pairs, determine the quality of the two pieces of content in the pair, and quantify the difference in quality between them. The second language model determines the winner and quantifies the degree of difference in content quality d∈{d1,d2,d3}, where d1, d2, and d3 are slightly different, moderately different, and significantly different, respectively. The difference weight w is adaptively updated to adjust the score more for significant differences and less for subtle differences. For example, different weights w = [1.0, 1.5, 2.0] correspond to different degrees of difference.

[0099] S103 , score update: based on the content quality difference, update the ELO scores of the two content data in the corresponding content data pair.

[0100] Specifically, for the content data pair {A, B}, the update formula of the ELO score is expressed as:

[0101]

[0102] K=K base ×w×α

[0103]

[0104] E B =1-E A

[0105] in, are the updated ELO scores of content data A and content data B respectively; R A 、R B are the ELO scores of content data A and content data B before updating; S A 、S B To judge the result score, if the content quality of content data A is better, then S A =1, SB = 0, if the content quality of content data B is better, then S A =0, S B =1, tie, both values are S A =S B =0.5; K is the fractional change rate coefficient, K base is the benchmark K value; α is the activity decay value; E A 、E B The expected win rate indicates the theoretical win rate based on the current content.

[0106] Thus, the embodiment of the present invention completes the entire process of the multi-stage content quality assessment method based on a large language model.

[0107] To demonstrate the superiority of the method provided by the present invention, the following comparisons were made using an example of measuring the quality of hotel user reviews on an online travel platform: directly using the number of votes for review usefulness (the distribution of the number of likes in the review data is shown in Table 1); using a machine learning method to predict the number of votes; and directly using a large language model for usefulness scoring.

[0108] Table 1. Distribution of likes in comment data

[0109] Likes range No likes 1-3 4-10 ＞10 Proportion 54％ 42.4％ 3.3％ 0.3％

[0110] 1. Directly use the number of votes for review usefulness

[0111] User-voted usefulness quantification mechanisms are prevalent in mainstream online platforms. Specifically, each user comment features a "usefulness" voting button, allowing other users to crowdsource their evaluation of the review's quality by clicking "up" or "down." The platform then presents the voting results as numerical values (e.g., "3 people found this review useful"). This method offers direct data acquisition and real-time feedback, but it has three limitations: First, the cold start problem results in a lack of voting data for newly published comments; second, voting results are easily influenced by the characteristics of the platform's user base, making cross-platform comparability difficult; and third, there is a risk of manipulation, such as malicious vote manipulation, which impacts data credibility.

[0112] 2. Use the method provided by the present invention

[0113] In the comparative experiment, the values of each variable parameter in the scheme are clearly defined as follows:

[0114] The first language model uses deepseek-v3, batch scoring (8 items / batch), and retains the top 200 (score ≥ 8) to enter the refined ranking stage.

[0115] The second largest language model uses deepseek-v3 and initializes the ELO score R i=1500,K base =32, w=[1.0, 1.5, 2.0], α=0.98^t (number of iterations, a total of 15 rounds of iterative comparison, each round generates 100 groups of review pairs, and finally outputs a stable sorted list. The iteration termination condition is set to converge to 4.8 for the standard deviation.

[0116] 3. Using machine learning to predict vote counts

[0117] To address the real-time limitations of vote count methods, a supervised learning model is proposed for predicting potential votes. While this method enables immediate evaluation, its accuracy is limited by the completeness of feature engineering. Here, a comparative approach to predicting vote counts using machine learning is presented. First, a prediction model is trained based on a historical dataset (including user voting records). The model uses BERT to represent comment text and outputs predicted like probabilities through a fully connected network and activation layers. The trained model then feeds the review set to be evaluated, sorting the top 20 high-quality comments based on their predicted probabilities.

[0118] 4. Directly use the large language model

[0119] Large language model evaluation method 1 involves using a large language model (here using DeepSeek-v3) to directly score review quality, similar to the initial screening step of the present invention, and the resulting scores are used to rank the review quality. Large language model evaluation method 2 involves inputting the review set obtained after the initial screening step, and using a large language model (here using DeepSeek-v3) to directly return the 20 highest-quality reviews.

[0120] In the implementation of the review quality measurement, an online questionnaire platform was used to integrate the top 20 reviews of each method (including ties, a total of 108 reviews), and 3 reviews were randomly selected from each method to form 15 questions, which the evaluators were asked to evaluate.

[0121] The results are as follows Figure 4 As shown, the method provided by the present invention scored first in the evaluation by the evaluators.

[0122] Taking the number of helpfulness likes as the benchmark, a linear mixed model was built for the review scores and the method used to screen out the reviews. The parameter estimation results based on the linear mixed effect model are shown in Table 2.

[0123] Table 2 Fixed effects of various methods on scores

[0124] method Rating Estimates t-value p-value Comment helpfulness votes 6.4049 30.281 ＜2e-16 The method provided by the present invention 0.3858 1.699 0.0923 Large Language Model Evaluation Method 1 0.2453 1.109 0.2697 Large Language Model Evaluation Method 2 0.2934 1.254 0.2132 Machine Learning Methods -0.3116 -1.307 0.1948

[0125] Fixed-effects analysis showed that, using the "number of review helpfulness votes" as the baseline, the differences in scores across methods exhibited the following characteristics: the mean score of high-quality reviews screened by the "method provided by the present invention" increased by 0.3858 compared to the baseline, achieving statistical significance (t=1.699, p>0.1). Two variants of the "Large Language Model Evaluation Method"—Large Language Model Evaluation Method 1 and LLM Evaluation Method 2—showed positive trends of 0.2453 (p=0.270) and 0.2934 (p=0.213), respectively, while the "Machine Learning Method" exhibited a negative shift of -0.3116 (p=0.195). None of the three methods showed statistically significant differences.

[0126] Example 2:

[0127] An embodiment of the present invention provides a multi-stage content quality assessment system based on a large language model, comprising:

[0128] An acquisition module is used to obtain the content data set to be evaluated and the evaluation indicators corresponding to its data type;

[0129] a primary screening module configured to perform a batch multi-dimensional coarse screening of the content data set based on the evaluation indicators and using the first language model, obtain evaluation scores of all content data, sort them, and filter out and retain content data with top scores;

[0130] The sorting module is used to introduce an improved ELO dynamic scoring mechanism, use the second largest language model to dynamically compare and sort the filtered content data, obtain the final ELO score of all content data and sort it, and use the ELO score sorting result as the content quality sorting result.

[0131] Example 3:

[0132] An embodiment of the present invention provides a storage medium storing a computer program for multi-stage content quality assessment based on a large language model, wherein the computer program enables a computer to execute the multi-stage content quality assessment method as described in any one of Embodiment 1.

[0133] Example 4:

[0134] An embodiment of the present invention provides an electronic device, including:

[0135] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including means for executing the multi-stage content quality assessment method as described in Example 1.

[0136] It can be understood that the multi-stage content quality assessment system, storage medium and electronic device based on a large language model provided in the embodiments of the present invention correspond to the multi-stage content quality assessment method based on a large language model provided in the embodiments of the present invention. The explanations, examples and beneficial effects of the relevant content can refer to the corresponding parts in the multi-stage content quality assessment method, and will not be repeated here.

[0137] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0138] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multi-stage content quality assessment method based on a large language model, characterized in that: include: Obtain the content dataset to be evaluated and the evaluation indicators corresponding to its data type; Based on the evaluation indicators, the content data set is subjected to batch multi-dimensional coarse screening using the first language model to obtain evaluation scores of all content data and sort them, and the content data with the highest scores are filtered and retained; An improved ELO dynamic scoring mechanism is introduced, and the second largest language model is used to dynamically compare and sort the filtered content data in pairs, obtain the final ELO score of all content data and sort it, and use the ELO score sorting result as the content quality sorting result.

2. The multi-stage content quality assessment method according to claim 1, wherein: Before using the first large language model to perform multi-dimensional coarse screening on the content data set, a prompt template of the first large language model is first constructed, and the prompt template includes the role of the model and the evaluation index.

3. The multi-stage content quality assessment method according to claim 2, wherein: After constructing the prompt template, a few-sample learning strategy is used to fine-tune the first language model; wherein the few-sample learning strategy is to input multiple pieces of content data with quality labels as a single sample into the first language model, so that it returns a comparative evaluation score.

4. The multi-stage content quality assessment method according to claim 1, wherein: The improved ELO dynamic scoring mechanism is introduced, and the second largest language model is used to dynamically compare and sort the filtered content data. The final ELO scores of all content data are obtained and sorted, and the ELO score sorting results are used as the content quality sorting results, including: Initialize the ELO score of each piece of content data after filtering; Determine whether the iteration termination condition is met. If so, output the final ELO scores of all content data and sort them. The ELO score sorting result is used as the content quality sorting result. Otherwise, repeat the following iterative steps in sequence: According to the principle of selecting the smallest difference, we obtain two matching content data pairs based on ELO score sampling; Using each content data pair as input to the second language model, determining the quality of the two content data in the content data pair, and quantifying the difference in content quality between the two content data; Based on the content quality difference, the ELO scores of the two content data in the corresponding content data pair are updated.

5. The multi-stage content quality assessment method according to claim 4, wherein: The iteration termination condition includes any one of standard deviation convergence, stability detection, and maximum number of iterations, or a combination of any several of them.

6. The multi-stage content quality assessment method according to claim 5, wherein: Define a content data pair {A, B}, then the update formula of the ELO score is expressed as: K=K base ×w×α AND B =1-E A in, are the ELO scores of content data A and content data B after updating; RA and RB are the ELO scores of content data A and content data B before updating; S A 、S B To judge the result score, if the content quality of content data A is better, then S A =1, S B = 0, if the content quality of content data B is better, then S A =0, S B =1, tie, both values are S A =S B =0.5; K is the fractional change rate coefficient, K base is the benchmark K value; w is the weight, and the greater the difference in content quality between content data A and content data B, the greater the value of w; α is the activity decay value; E A 、E B The expected win rate indicates the theoretical win rate based on the current content.

7. A multi-stage content quality assessment system based on a large language model, characterized in that: include: An acquisition module is used to obtain the content data set to be evaluated and the evaluation indicators corresponding to its data type; a primary screening module configured to perform a batch multi-dimensional coarse screening of the content data set based on the evaluation indicators and using the first language model, obtain evaluation scores of all content data, sort them, and filter out and retain content data with top scores; The sorting module is used to introduce an improved ELO dynamic scoring mechanism, use the second largest language model to dynamically compare and sort the filtered content data, obtain the final ELO score of all content data and sort it, and use the ELO score sorting result as the content quality sorting result.

8. A storage medium, characterized in that: The computer program for multi-stage content quality assessment based on a large language model is stored therein, wherein the computer program enables a computer to execute the multi-stage content quality assessment method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including instructions for executing the multi-stage content quality assessment method according to any one of claims 1 to 6.

Images