Multi-stage content quality evaluation method and system based on large language model

Abstract

The application provides a kind of multi-stage content quality evaluation method and system based on large language model, it is related to natural language processing technical field.In the application, the content data set to be evaluated and the evaluation index corresponding to its data type are obtained;Based on the evaluation index, the first large language model is used to carry out batch multi-dimensional rough screening on the content data set, the evaluation score of all content data is obtained and sorted, and several content data with high score are filtered and reserved;An improved ELO dynamic scoring mechanism is introduced, and the second large language model is used to dynamically compare and sort the filtered content data, to obtain the final ELO score of all content data and sort it, and the ELO score sorting result is taken as the content quality sorting result.This method uses large language model to break through the cognitive depth bottleneck, relies on dynamic game mechanism to improve the objectivity of evaluation, balances the calculation efficiency and sorting accuracy by combining two-stage architecture, and finally forms a general solution for content quality evaluation that can quickly adapt to multiple fields.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a multi-stage content quality evaluation method and system based on a large language model. BACKGROUND

[0002] Content data is widely present on Internet platforms, such as online reviews, creative scheme submissions, design works, etc. While these contents provide vitality and diversity for the platform, they also bring a large amount of low-quality, repetitive or fake contents, making efficient and accurate quality evaluation and screening a problem to be solved.

[0003] In related technologies, existing content data evaluation methods are mainly divided into three types: using computable indicators for comprehensive scoring, model prediction, and manual evaluation. Among them, using computable indicators for comprehensive scoring, for example, CN104850642A discloses a network content quality evaluation method and device, which obtains content quality features of network content and calculates feature scores. Model prediction of content quality, for example, CN111428032A discloses a content quality evaluation method and device, electronic equipment and storage medium, which uses a pre-trained model to filter obscene content in multimedia. Manual evaluation, i.e., calling evaluators or experts to evaluate specific content, or using data such as "usefulness like number" on the network as evaluation basis.

[0004] However, current Internet platforms face the demand for efficient quality evaluation of massive content, especially when dealing with online reviews, creative submissions, etc. It is necessary to consider precision, cross-domain adaptability and real-time performance. The above methods cannot meet the requirements of real-time performance and cross-domain adaptability while considering deep semantic information. SUMMARY

[0005] (I) Technical problems solved

[0006] To solve the technical problems of the prior art, the present application provides a multi-stage content quality evaluation method and system based on a large language model, which solves the technical problems of considering precision, cross-domain adaptability and real-time performance.

[0007] (II) Technical solutions

[0008] To achieve the above purposes, the present application is implemented by the following technical solutions:

[0009] A multi-stage content quality evaluation method based on a large language model, comprising:

[0010] obtaining a content data set to be evaluated and evaluation indicators corresponding to the data type of the content data set;

[0011] Based on the evaluation index, a first large language model is used to perform batch multi-dimensional rough screening on the content data set, to obtain evaluation scores of all content data and sort them, and to filter and retain several content data with high scores;

[0012] A modified ELO dynamic scoring mechanism is introduced, and a second large language model is used to perform dynamic pairwise comparison and sorting on the filtered content data, to obtain final ELO scores of all content data and sort them, and to take the ELO score sorting result as the content quality sorting result.

[0013] Preferably, before performing multi-dimensional rough screening on the content data set by using the first large language model, a prompt template of the first large language model is constructed, and the prompt template includes the role of the model and the evaluation index.

[0014] Preferably, after the prompt template is constructed, a few-shot learning strategy is used to fine-tune the first large language model; wherein the few-shot learning strategy refers to inputting multiple content data with quality labels as a single sample into the first large language model, so as to return comparative evaluation scores.

[0015] Preferably, the modified ELO dynamic scoring mechanism is introduced, and the second large language model is used to perform dynamic pairwise comparison and sorting on the filtered content data, to obtain final ELO scores of all content data and sort them, and to take the ELO score sorting result as the content quality sorting result, which includes:

[0016] Initialize the ELO score of each content data after filtering;

[0017] Determine whether the iteration termination condition is reached, if yes, output the final ELO scores of all content data and sort them, and take the ELO score sorting result as the content quality sorting result, otherwise repeat the following iteration steps:

[0018] According to the principle of minimum difference, sample and obtain a pair of content data based on the ELO score;

[0019] Take each pair of content data as input of the second large language model, determine the content quality of the two content data in the pair, and quantify the content quality difference between them;

[0020] Based on the content quality difference, update the ELO scores of the two content data in the corresponding pair.

[0021] Preferably, the iteration termination condition includes any one or a combination of standard deviation convergence, stability detection, and maximum iteration number.

[0022] Preferably, define a content data pair {A, B}, and the update formula of the ELO score is expressed as:

[0023]

[0024] K = K base × w × a

[0025]

[0026] E B = 1 - E A

[0027] wherein, respectively, the updated ELO scores of the content data A and the content data B; RA and RB respectively are the ELO scores of the content data A and the content data B before being updated; S A , S B is the judgment result score, if the content quality of the content data A is better, then S A = 1, S B = 0, if the content quality of the content data B is better, then S A = 0, S B = 1, and the value of both in a draw is S A = S B = 0.5; K is a score change speed coefficient, K base is a reference K value; w is a weight, and the greater the difference in content quality between the content data A and the content data B, the greater the value of w; a is an activity decay value; E A , E B is an expected win rate, indicating the theoretical win rate based on the current content.

[0028] A multi-stage content quality evaluation system based on a large language model, comprising:

[0029] An acquisition module for acquiring a content data set to be evaluated and an evaluation index corresponding to the data type thereof;

[0030] A preliminary screening module for performing batch multi-dimensional rough screening on the content data set based on the evaluation index by using a first large language model, acquiring evaluation scores of all content data and sorting them, and filtering and retaining a number of content data with high scores;

[0031] A sorting module for introducing an improved ELO dynamic scoring mechanism, performing dynamic pairwise comparison and sorting on the filtered content data by using a second large language model, acquiring final ELO scores of all content data and sorting them, and taking the ELO score sorting result as a content quality sorting result.

[0032] A storage medium storing a computer program for multi-stage content quality evaluation based on a large language model, wherein the computer program causes a computer to perform the multi-stage content quality evaluation method as described above.

[0033] An electronic device comprising:

[0034] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs comprising instructions for performing the multi-stage content quality evaluation method as described above.

[0035] (III) Beneficial Effects

[0036] The present application provides a multi-stage content quality evaluation method and system based on a large language model. Compared with the prior art, the following beneficial effects are achieved:

[0037] In the present application, the content data set to be evaluated and the evaluation index corresponding to its data type are obtained; based on the evaluation index, the first large language model is used to perform batch multi-dimensional rough screening on the content data set, the evaluation scores of all content data are obtained and sorted, and a number of content data with high scores are filtered and retained; an improved ELO dynamic scoring mechanism is introduced, the second large language model is used to perform dynamic pairwise comparison and sorting on the filtered content data, the final ELO scores of all content data are obtained and sorted, and the ELO score sorting result is taken as the content quality sorting result. This method uses a large language model to break through the cognitive depth bottleneck, relies on a dynamic game mechanism to improve the objectivity of evaluation, balances the calculation efficiency and sorting accuracy in combination with a two-stage architecture, and finally forms a general solution to content quality evaluation that can quickly adapt to multiple fields. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, brief descriptions will be given below to the drawings needed to be used in the embodiments or prior art descriptions. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0039] Figure 1 A block diagram of a multi-stage content quality evaluation method based on a large language model provided by an embodiment of the present application;

[0040] Figure 2 A flowchart of a multi-stage content quality evaluation method based on a large language model provided by an embodiment of the present application;

[0041] Figure 3A frame schematic diagram of a prompt template provided by the embodiment of the present application;

[0042] Figure 4 An average score diagram given by evaluators under different evaluation methods. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application are described clearly and completely. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0044] The embodiment of the present application solves the technical problems of considering precision, cross-domain adaptability and real-time performance by providing a multi-stage content quality evaluation method and system based on a large language model.

[0045] The technical solutions in the embodiment of the present application have the following technical advantages to solve the above technical problems:

[0046] (1) The embodiment of the present application makes the large language model evaluator have the cognitive framework of a domain expert by utilizing the large language model role situation simulation, breaks through the limitation of only processing surface features (such as keyword frequency and text length) in the past, and realizes the essential modeling of abstract concepts such as "innovation" and "social value".

[0047] (2) The embodiment of the present application introduces multiple comments for simultaneous preliminary screening when using the large language model for evaluation, and enhances the understanding of the quality dimension of the model through comparative learning.

[0048] (3) The embodiment of the present application converts the absolute score into a relative competitiveness index by using the ELO dynamic game mechanism, solves the evaluation distortion problem caused by the distribution deviation of artificial labeling data, and gradually establishes the relative quality relationship network between contents through multiple rounds of content comparison, and overcomes the "score inflation" problem (such as a large number of contents concentrated in a certain score interval) caused by the non-uniform labeling standard in the traditional static score.

[0049] (4) The embodiment of the present application adopts a two-stage architecture of "multi-dimensional rough screening-dynamic comparison fine evaluation", realizes intelligent allocation of computing resources, and quickly eliminates low-quality items through parallel batch scoring of the large language model. The present application intelligently matches and dynamically generates a comparison strategy in the fine evaluation stage: preferentially selects adjacent contents with small ELO score difference for comparison to ensure that each round of comparison effectively improves the sorting accuracy. Compared with full-quantity random comparison, the required comparison rounds are reduced.

[0050] (4) The embodiment of the application realizes rapid cross-field migration of the evaluation system through modular design. Without retraining the model, the new field can be adapted only by prompting engineering and parameter configuration.

[0051] In addition, the terms involved in the embodiment of the application are introduced:

[0052] First, the large language model (LLM), also known as the large language model, is an artificial intelligence model designed to understand and generate human language. They are trained on a large amount of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc.

[0053] Note that the embodiments of the present application do not limit the types of the first large language model and the second large language model, and do not limit whether the types of the first large language model and the second large language model are the same. They can be any combination of mainstream large language models such as DeepSeek-v3, GPT4o, etc.

[0054] Second, the Elo rating system (Elo Rating System, note that Elo here is the surname of the creator of the mechanism) is an evaluation method for measuring the level of various playing activities, and is currently the recognized authoritative method for evaluating playing level.

[0055] Note that the improved ELO dynamic scoring mechanism in the embodiments of the present application is based on the classic ELO playing scoring model, combined with the content comparison evaluation method driven by the large language model, through the matching strategy of score proximity priority, the score adjustment mechanism based on the feedback of the large model, and the multiple convergence judgment standard, higher precision and stability of high-quality content sorting are realized, which is suitable for complex evaluation scenarios with non-structured content and multiple evaluation dimensions.

[0056] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with the drawings in the specification and specific embodiments.

[0057] Embodiment 1:

[0058] As shown in Figure 1 The embodiment of the application provides a multi-stage content quality evaluation method based on a large language model, which comprises:

[0059] S1, obtaining the content data set to be evaluated and the evaluation index corresponding to the data type thereof;

[0060] S2, based on the evaluation index, using the first large language model to perform batch multi-dimensional rough screening on the content data set, obtaining the evaluation score of all content data and sorting, and filtering and retaining a number of content data with high scores;

[0061] S3, introduce an improved ELO dynamic scoring mechanism, use the second large language model to dynamically compare and sort the filtered content data, obtain the final ELO score of all content data and sort, and take the ELO score sorting result as the content quality sorting result.

[0062] The method provided by the embodiment of the application breaks through the cognitive depth bottleneck by using a large language model, improves the objectivity of evaluation by relying on a dynamic game mechanism, balances the calculation efficiency and sorting accuracy by combining a two-stage architecture, and finally forms a general solution to content quality evaluation that can quickly adapt to multiple fields.

[0063] As shown in Figure 2 , a flowchart of a multi-stage content quality evaluation method based on a large language model is disclosed. Next, each step of the above scheme will be introduced in detail: Figure 2 The flowchart of the multi-stage content quality evaluation method based on the large language model is disclosed. Next, each step of the above scheme will be introduced in detail: Figure 2

[0064] In step S1, the content data set to be evaluated and the evaluation indicators corresponding to the data type thereof are obtained.

[0065] This step obtains the content data set to be evaluated. The content data type here can be online comments, creative scheme content, design works, etc. After selecting the content data type, this step needs to construct the corresponding evaluation indicators based on the type.

[0066] In other words, this step does not limit the content data type, but usually needs to obtain / construct different evaluation indicators for evaluating the content quality of the corresponding type of content data. For example:

[0067] For online comment content, the indicators for evaluating its quality include comment information quantity, comment relevance, comment persuasiveness, etc.

[0068] For creative scheme content, the indicators for evaluating its quality include scheme novelty, scheme strategic feasibility, scheme environmental value, etc.

[0069] For design works, the indicators for evaluating its quality include visual expressiveness, function implementation, user experience, and innovation, etc.

[0070] ​Further, the embodiment of the present application will take online review content as an example to describe the details of the scheme in detail. If the online review content is further limited to online travel platform hotel user reviews, the evaluation indexes in this step are set to {“Index 1: Persuasiveness: Whether the review has elements of persuasiveness that can attract the interest of other readers. Index 2: Amount of review information: The richness of vocabulary and the probability of occurrence of non-common words reflect the depth of information. Index 3: Theme relevance: Whether the keywords in the review overlap with the keywords of the theme, indicating the relevance of the review to the theme discussed. Check if the review involves specific topics such as service, cleanliness, facilities, location, room condition, cost-effectiveness, and catering.”}.

[0071] In step S2, based on the evaluation indexes, the first large language model is used to perform batch multi-dimensional rough screening on the content data set, and the evaluation scores of all content data are obtained and sorted, and several content data with high scores are filtered and retained.

[0072] Based on the evaluation indexes obtained in the previous step, this step first constructs a prompt template for the first large language model, which includes the role of the model and the evaluation indexes, and the framework diagram is as shown in Figure 3 .

[0073] After the prompt template is constructed, the first large language model can also be fine-tuned using a few-shot learning strategy; wherein the few-shot learning strategy refers to inputting multiple content data with quality labels as a single sample into the first large language model, and returning the evaluation scores of multiple content data, so that the evaluation scores have comparability.

[0074] Based on the prompt template, all content of the prompt word is determined, and the fine-tuned first large language model is used to perform batch multi-dimensional rough screening on the content data set, and the evaluation scores of all content data are obtained and sorted, and several content data with high scores are filtered and retained.

[0075] The evaluation score here can be set to any integer in the interval of 1-10.

[0076] Still taking online travel platform hotel user reviews as an example, a feasible prompt word (i.e. italic part) of the first large language model is:

[0077] Role: You are a consumer browsing reviews of a product / service

[0078] User: Please evaluate the quality of consumer-posted reviews based on the following criteria: "Persuasiveness: Does the review have elements that can attract the interest of other readers? "Review Information: The richness of vocabulary and the probability of non-common words appearing, reflecting the depth of information. "Theme Relevance: Do the keywords in the review overlap with the keywords of the theme, indicating the relevance of the review to the topic discussed. Check if the review is related to specific topics such as service, cleanliness, facilities, location, room condition, value for money, and dining."

[0079] Through the thought chain, first read all the reviews, then think about the score of each dimension of each review, give each content a score of 1-10 according to the evaluation of each indicator, and format the reply to form a json file with a certain structure:

[0080]

[0081] Now process the following data:

[0082] {"Review 1": "The experience of staying in this hotel is super good! The front desk check-in speed is very fast, the staff is warm and professional, and also introduces the nearby interesting places. The room is clean and tidy, and the facilities are also new. However, the breakfast variety is a bit less, and the style can be more diverse.", "Review 2": "xxxx", "Review 3": "xxxx",..., "Review 8": "xxxx"}

[0083] In step S3, the improved ELO dynamic scoring mechanism is introduced, and the filtered content data is dynamically compared and sorted by the second large language model to obtain the final ELO score of all content data and sort the ELO score sorting results as the content quality sorting results.

[0084] On the basis of rough screening, to further quantify the subtle differences between high-quality contents, this step introduces an improved ELO dynamic scoring mechanism to realize the fine sorting of content quality of content data through pairwise comparison driven by large language models. Related steps include:

[0085] S31, initialize the ELO score of each piece of filtered content data.

[0086] Specifically, define the filtered content data set as C = {c1, c2,..., c M}, M represents the number of content data, and the initial ELO score of the i-th content data c i is Ri, and then enter the iteration of the ELO score.

[0087] S32, judge whether the iteration termination condition is reached, the iteration termination condition here includes any one or any combination of the following:

[0088] standard deviation convergence (for example, the total ELO score standard deviation change of the last 3 rounds <2%);

[0089] stability detection (Top20 content without rank change for 2 consecutive rounds);

[0090] maximum number of iterations (N max = log2(M) x 10).

[0091] If the iteration termination condition is reached, output the final ELO score of all content data and sort, and take the ELO score sorting result as the content quality sorting result, otherwise repeat the following iteration steps:

[0092] S101, matching sampling: based on the ELO score, sample and obtain a pair of content data for matching according to the principle of minimum difference.

[0093] Specifically, based on the current ELO score, select the content pair with the smallest difference (c A ,c B ) from the comparison set, that is, satisfy This strategy focuses on content with similar scores to improve the discrimination efficiency of the model.

[0094] S102, model judgment: take each content data pair as the input of the second large language model, judge the content quality of the two content data in the content data pair, and quantify the content quality difference between them.

[0095] Specifically:

[0096] Similar to the processing of the first large language model, here also set the role of the second large language model, still taking the hotel user comments of the online travel platform as an example, a feasible prompt word (that is, the italic part) of the second large language model is:

[0097] "Role: You are the quality reviewer. User: Now, using these metrics, evaluate which of the following two reviews is of higher quality (Review A, Review B, Tie), and by how much (minor difference, moderate difference, significant difference). The metrics are: {"Metric 1: Persuasiveness: Does the review contain persuasive elements that can attract the interest of other readers?", "Metric 2: Information Content: The richness of the vocabulary and the probability of the appearance of non-common words, reflecting the depth of information.", "Metric 3: Topic Relevance: Does the keyword contained in the review overlap with the keyword of the topic, indicating the relevance of the review to the topic under discussion? Check if the review covers specific topics such as service, cleanliness, facilities, location, room condition, value for money, and dining."} Now compare the following: {"Hotel Review A": xxxxxxxx, "Hotel Review B": xxxxxxxx}"

[0098] After the second language model is fine-tuned, it is used to compare content data pairs to determine the quality of the two data points and quantify the difference in content quality between them. The second language model needs to determine the winner and quantify the degree of content quality difference d∈{d1,d2,d3}, where d1, d2, and d3 represent subtle, moderate, and significant differences, respectively. The difference weight w implements adaptive updates of "significant differences are adjusted by large adjustments, and subtle differences are adjusted by small adjustments," for example, different degrees of difference correspond to different weights w=[1.0, 1.5, 2.0].

[0099] S103, Rating Update: Based on the differences in content quality, update the ELO scores of the two content data in the corresponding content data pair.

[0100] Specifically, for the content data pair {A, B}, the ELO score update formula is expressed as:

[0101]

[0102] K = K base ×w×α

[0103]

[0104] E B =1-E A

[0105] in, These are the updated ELO scores for content data A and content data B, respectively; R A R B These are the ELO scores for content data A and content data B before the update; S A S B To determine the score, if content data A has better content quality, then S... A =1,SB = 0, if the content quality of the content data B is better, then S A = 0, S B = 1, draw both values are S A = S B = 0.5; K is a fractional change speed coefficient, K base is a baseline K value; a is an activity decay value; E A , E B is the expected win rate, representing the theoretical win rate based on the current content.

[0106] So far, the embodiment of the application has completed the entire process of the multi-stage content quality evaluation method based on a large language model.

[0107] In order to demonstrate the superiority of the method provided by the application, the following comparisons are made on the example of online travel platform hotel user comment quality measurement: directly using the number of votes for comment usefulness (where the comment data point like number distribution is shown in Table 1); using a machine learning method to predict the number of votes; directly using a large language model to score usefulness.

[0108] Table 1 Comment data point like number distribution

[0109] Like count interval No like count 1-3 4-10 ＞10 Proportion 54％ 42.4％ 3.3％ 0.3％

[0110] 1. Directly using the number of votes for comment usefulness

[0111] In the user comment system of mainstream online platforms, there is generally a usefulness quantification mechanism based on user voting. Specifically, a "usefulness" voting button is provided below each user comment, and other users can evaluate the comment quality through the like / dislike button, and the platform finally presents the voting statistics in numerical form (such as "3 people think the comment is useful"). This method has the characteristics of direct data acquisition and strong real-time feedback, but its limitations are reflected in three aspects: first, the cold start problem leads to a lack of voting data for newly published comments; second, the voting results are easily influenced by the characteristics of the platform user group, and the cross-platform comparability is poor; third, there is a risk of malicious vote manipulation, affecting the data credibility.

[0112] 2. Using the method provided by the application

[0113] In the comparative experiment, the values of each variable parameter in the scheme are determined as follows:

[0114] The first large language model uses deepseek-v3, batch scoring (8 per batch), and the top 200 (score >= 8) are retained to enter the fine ranking stage.

[0115] The second large language model uses deepseek-v3, and the initial ELO score R i=1500, K base =32, w=[1.0, 1.5, 2.0], α=0.98^t (number of iterations, a total of 15 iterations are performed, 100 sets of comment pairs are generated in each round, and a stable sorted list is finally output. The iteration termination condition is set to convergence of standard deviation to 4.8.

[0116] 3. Use machine learning to predict vote counts

[0117] To address the real-time limitations of traditional voting methods, a supervised learning model is proposed to predict potential vote counts. While this method enables instant evaluation, its prediction accuracy is constrained by the completeness of feature engineering. A comparative approach using machine learning to predict vote counts is presented here. First, a prediction model is trained based on a historical dataset (including user voting records). BERT is used to represent comment text, and a fully connected network and activation layers output predicted like probabilities. Then, the set of comments to be evaluated is input into the fully trained model, and the top 20 high-quality comments are selected based on their predicted probabilities.

[0118] 4. Directly use large language models

[0119] Large language model evaluation method 1 refers to the large language model (deepseek-v3 used here) directly scoring the quality of comments, similar to the preliminary screening step of the method in this invention, and the obtained score is used as the ranking of comment quality. Large language model evaluation method 2 refers to the large language model (deepseek-v3 used here) directly returning the 20 high-quality comments after inputting the comment set obtained from the preliminary screening step.

[0120] In an implementation of the review quality measurement, an online questionnaire platform was used to integrate the top 20 reviews for each method (including ties, totaling 108 reviews). Three reviews for each method were randomly selected to form 15 questions, which were then evaluated by evaluators.

[0121] The results are as follows Figure 4 As shown, the method provided by this invention scores first in the evaluator's assessment.

[0122] Based on the number of usefulness likes, a linear mixture model was constructed for the comment score and the method used to filter the comments. The parameter estimation results based on the linear mixture effect model are shown in Table 2.

[0123] Table 2 shows the fixed effects of each method on the scoring.

[0124] Method Score estimate value t value p value Comment usefulness vote 6.4049 30.281 ＜2e-16 The method provided by the application 0.3858 1.699 0.0923 Large language model evaluation method 1 0.2453 1.109 0.2697 Large language model evaluation method 2 0.2934 1.254 0.2132 Machine learning method -0.3116 -1.307 0.1948

[0125] The fixed effect analysis shows that, based on the "review usefulness vote number" as the benchmark group, the score difference of each method presents the following characteristics: the average score of the screened high-quality reviews of "the method provided by the invention" is improved by 0.3858 compared with the benchmark group, which reaches statistical significance (t=1.699, p>0.1). The two variants of "large language model evaluation method" 1 and "large language model evaluation method" 2 respectively present positive trends of 0.2453 (p=0.270) and 0.2934 (p=0.213), while "machine learning method" shows a negative offset of-0.3116 (p=0.195), and none of the three presents statistical significant difference.

[0126] Embodiment 2:

[0127] The embodiment of the present application provides a multi-stage content quality evaluation system based on a large language model, which comprises:

[0128] The acquisition module is configured to acquire a content data set to be evaluated and an evaluation index corresponding to a data type of the content data set.

[0129] The preliminary screening module is configured to perform batch multi-dimensional rough screening on the content data set based on the evaluation index by using a first large language model, acquire evaluation scores of all content data and sort the evaluation scores, and filter and reserve a plurality of content data with high scores.

[0130] The sorting module is configured to introduce an improved ELO dynamic scoring mechanism, perform dynamic pairwise comparison and sorting on the filtered content data by using a second large language model, acquire final ELO scores of all content data and sort the ELO scores, and take the ELO score sorting result as a content quality sorting result.

[0131] Embodiment 3:

[0132] The embodiment of the present application provides a storage medium which stores a computer program for multi-stage content quality evaluation based on a large language model, wherein the computer program enables a computer to execute the multi-stage content quality evaluation method according to any one of the embodiments 1.

[0133] Embodiment 4:

[0134] The embodiment of the present application provides an electronic device, which comprises:

[0135] one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs comprise a program for executing the multi-stage content quality evaluation method according to the embodiment 1.

[0136] It can be understood that the multi-stage content quality evaluation system based on a large language model, the storage medium and the electronic device provided by the embodiments of the present application correspond to the multi-stage content quality evaluation method based on a large language model provided by the embodiments of the present application, and the related content explanations, examples and beneficial effects of the multi-stage content quality evaluation method can be referred to the corresponding parts of the multi-stage content quality evaluation method, which will not be repeated here.

[0137] It should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual such relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0138] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-stage content quality evaluation method based on a large language model, characterized in that, The method comprises the following steps: obtain the evaluation index corresponding to the content data set to be evaluated and its data type; based on the evaluation index, use a first large language model to perform batch multi-dimensional rough screening on the content data set, obtain the evaluation scores of all content data and sort them, and filter and retain several content data with high scores; introduce an improved ELO dynamic scoring mechanism, use a second large language model to perform dynamic pairwise comparison and sorting on the filtered content data, obtain the final ELO scores of all content data and sort them, and take the ELO score sorting result as the content quality sorting result; The introduction of the improved ELO dynamic scoring mechanism, the use of the second large language model to perform dynamic pairwise comparison and sorting on the filtered content data, the obtaining of the final ELO scores of all content data and the sorting, and the taking of the ELO score sorting result as the content quality sorting result, comprise: initializing the ELO score of each piece of filtered content data; determine whether the iteration termination condition is reached, if yes, output the final ELO scores of all content data and sort them, and take the ELO score sorting result as the content quality sorting result, otherwise repeat the following iteration steps: According to the principle of minimum difference, based on the ELO score, sample and obtain a pair of content data; take each pair of content data as the input of the second large language model, judge the content quality of the two content data in the pair of content data, and quantify the content quality difference between the two content data; based on the content quality difference, update the ELO scores of the two content data in the corresponding content data pair.

2. The multi-stage content quality assessment method of claim 1, wherein, Before using the first large language model to perform multi-dimensional rough screening on the content data set, a prompt template of the first large language model is constructed, and the prompt template comprises the role of the model and the evaluation index.

3. The multi-stage content quality evaluation method of claim 2, wherein, After the prompt template is constructed, the first large language model is fine-tuned by using a few-shot learning strategy; wherein the few-shot learning strategy refers to inputting multiple content data with quality labels as a single sample into the first large language model, so as to return an evaluation score with contrast.

4. The multi-stage content quality assessment method of claim 1, wherein, The iteration termination condition includes any one or a combination of standard deviation convergence, stability detection, and maximum iteration number.

5. The multi-stage content quality assessment method of claim 4, wherein, Define a content data pair {A, B}, and the update formula of the ELO score is: K = K base x w x a E B = 1 - E A Wherein, ELO score of content data A and content data B after update respectively; R A , R B ELO score of content data A and content data B before update respectively; S A , S B is the judgment result score, if the content quality of content data A is better, then S A =1, S B =0, if the content quality of content data B is better, then S A =0, S B =1, and the value of both in a draw is S A =S B =0.5; K is the score change speed coefficient, K base is the reference K value; w is the weight, and the greater the content quality difference between content data A and content data B, the greater the value of w; α is the activity decay value; E A , E B is the expected win rate, which represents the theoretical win rate based on the current content.

6. A multi-stage content quality evaluation system based on a large language model, characterized in that, The method for performing the multi-stage content quality evaluation method of claim 1 comprises: an acquisition module for acquiring an evaluation index corresponding to a content data set to be evaluated and its data type; a preliminary screening module for performing batch multi-dimensional rough screening on the content data set based on the evaluation index by using a first large language model, obtaining the evaluation scores of all content data and sorting them, and filtering and retaining several content data with high scores; a sorting module for introducing an improved ELO dynamic scoring mechanism, using a second large language model to perform dynamic pairwise comparison and sorting on the filtered content data, obtaining the final ELO scores of all content data and sorting them, and taking the ELO score sorting result as the content quality sorting result.

7. A storage medium, characterized by A computer program for multi-stage content quality assessment based on large language models is stored in the storage, wherein the computer program causes a computer to execute the multi-stage content quality assessment method according to any one of claims 1-5.

8. An electronic device, comprising: Comprise: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs comprise programs for executing the multi-stage content quality assessment method according to any one of claims 1-5.

Images