Large language model selection method based on multi-dimensional evaluation and dynamic weight adjustment

Abstract

The invention discloses an automatic large language model selection method based on multi-dimensional evaluation and dynamic weight adjustment, and aims to solve the problems of low text generation quality, inaccurate dialogue questions and answers, low online assistant task completion rate and the like in an existing large language model by introducing a multi-dimensional evaluation mechanism. According to the method, a plurality of evaluation dimensions such as ROUGE index evaluation, text quality evaluation, semantic similarity evaluation, safety compliance evaluation, front suitability evaluation, question and answer evaluation, dialogue evaluation, online customer service evaluation and online assistant evaluation are comprehensively considered, the performance of the large language model is comprehensively evaluated, and through analysis of operation data and user feedback, the evaluation accuracy of the large language model is improved. And dynamically adjusting the weight of each evaluation dimension to adapt to continuously changing application requirements and user expectations. And finally, according to the comprehensive score, an optimal large language model is automatically selected, and high quality and user satisfaction of Web services are ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of large models, and in particular relates to a large language model selection method based on multi-dimensional evaluation and dynamic weight adjustment. Background Art

[0002] Currently, large language models perform well in various natural language processing tasks, such as text generation, question-answering systems, and dialogue simulation. However, with the complexity of large language models and the diversification of application scenarios, it is becoming increasingly difficult to select the large language model that best suits a specific task. Existing technologies mainly rely on a single evaluation indicator or manual experience, and are unable to comprehensively and objectively evaluate model performance. For example, although the ROUGE indicator can evaluate the similarity between the generated text and the reference text, it cannot evaluate multiple aspects of performance such as text quality, semantic similarity, security and compliance of the generated content. Although logic can check whether the model's answers are relevant to the user's questions and whether it can provide reasonable answers based on the dialogue context, it cannot comprehensively evaluate the performance of large language models in actual dialogue scenarios, and cannot ensure its effectiveness and reliability in different application scenarios. In addition, it is necessary to comprehensively evaluate the performance of large language models in online assistant applications to ensure their effectiveness and user satisfaction in actual applications.

[0003] In addition, the existing technology lacks a mechanism to dynamically adjust the evaluation weights and cannot continuously optimize the evaluation criteria based on feedback from actual applications. Therefore, how to establish an automated large language model selection method with multi-dimensional evaluation and dynamic weight adjustment has become an important challenge in the current technical field. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present application provides a large language model selection method based on multi-dimensional evaluation and dynamic weight adjustment.

[0005] In a first aspect, the present application proposes a large language model selection method based on multi-dimensional evaluation and dynamic weight adjustment, comprising the following steps:

[0006] Select any number of pre-trained large language models as the large language models to be evaluated;

[0007] Performing text generation function evaluation, dialogue question-answering function evaluation, and Web application function evaluation on the large language model to be evaluated, and extracting operation process data in each evaluation dimension from each evaluation process as actual performance data of the large language model to be evaluated, and extracting user rating data from each evaluation result as user feedback data of the large language model to be evaluated;

[0008] Calculate the weight of each evaluation dimension of the large language model to be evaluated according to the actual performance data and the user feedback data, and dynamically adjust the weight of each evaluation dimension of the large language model to be evaluated according to the calculation result;

[0009] The comprehensive score of the large language model to be evaluated is calculated according to the dynamically adjusted weight, and the large language model with the highest comprehensive score is selected from the large language models to be evaluated as the optimal large language model.

[0010] In some embodiments, the text generation function evaluation, the dialogue question and answer function evaluation and the Web application function evaluation are performed on the large language model to be evaluated, and the text generation function evaluation includes:

[0011] Generate a set of text data through the large language model to be evaluated, and perform ROUGE indicator evaluation, text quality evaluation, semantic similarity evaluation, security compliance evaluation, and positive suitability evaluation on the generated text;

[0012] The ROUGE indicator evaluation: quantifies the accuracy of the generated text by calculating the number of N-gram overlaps between the generated text and the reference text;

[0013] The text quality assessment: the generated text is evaluated by a pre-trained language model to obtain a corresponding text quality score to quantify language fluency and syntactic correctness;

[0014] The semantic similarity evaluation: uses the pre-trained semantic similarity model to calculate the semantic similarity between the generated text and the reference text. The formula for quantifying the semantic proximity between the generated text and the reference text is as follows:

[0015]

[0016] in, and are the vector representations of generated text and reference text respectively;

[0017] The security compliance assessment: classifies the generated text through a pre-trained classification model to determine whether it contains sensitive words or content that violates the policy. If the generated text does not contain sensitive words or content that violates the policy, the higher the score, otherwise the lower the score;

[0018] The positive suitability assessment is as follows: the generated text is input into the sentiment analysis model to obtain sentiment polarity and sentiment intensity scores. If the text sentiment is positive and the intensity meets the preset standard, the higher the score, and if the text sentiment is negative or the intensity does not meet the preset standard, the lower the score.

[0019] In some embodiments, the large language model to be evaluated is evaluated for text generation function, dialogue question and answer function, and Web application function, wherein the dialogue question and answer function evaluation includes question and answer evaluation and dialogue evaluation:

[0020] The question-answering evaluation is:

[0021] Prepare a set of assessment questions and corresponding model answers;

[0022] Inputting the evaluation question into the large language model to be evaluated to obtain a model-generated answer;

[0023] The BLEU indicator is used to evaluate the similarity between the generated answer and the standard answer. The higher the similarity, the higher the corresponding score, and vice versa. The user's question and answer satisfaction score is collected through user feedback;

[0024] The dialogue evaluation is:

[0025] Design a set of dialogues that simulate actual application scenarios;

[0026] The designed dialogue is input into the large language model to be evaluated for multiple rounds of dialogue. The fluency of the dialogue is scored using the pre-trained language model. The consistency and logical rationality of the dialogue are analyzed and scored using the dialogue management model. The user's dialogue satisfaction score is collected through user feedback.

[0027] In some embodiments, the large language model to be evaluated is evaluated for text generation function, dialogue question and answer function, and Web application function, and the Web application function evaluation includes online customer service evaluation and online assistant evaluation:

[0028] The online customer service evaluation is:

[0029] Design a variety of user consultation scenarios, input the user consultation scenarios into the large language model to be evaluated, simulate the conversation between real users and customer service, record the time it takes for the large language model to generate a reply, complete the response time evaluation, check the accuracy of the model's answer, complete the accuracy evaluation, and collect the user's customer service satisfaction score through user feedback;

[0030] The online assistant is evaluated as:

[0031] Set a variety of online assistant tasks, input the online assistant tasks into the large language model to be evaluated, simulate the interaction process between the user and the online assistant, calculate the task completion rate through the statistical model, and collect the user's assistant satisfaction score through user feedback.

[0032] In some embodiments, extracting the running process data in each evaluation dimension from each evaluation process as the actual performance data of the large language model to be evaluated, and extracting the user rating data from each evaluation result as the user feedback data of the large language model to be evaluated, includes:

[0033] Extract the running process data of ROUGE indicator evaluation, text quality evaluation, semantic similarity evaluation, security compliance evaluation, positive suitability evaluation, question-answering evaluation, dialogue evaluation, online customer service evaluation, and online assistant evaluation as the actual performance data of the large language model to be evaluated;

[0034] The question-and-answer satisfaction score, conversation satisfaction score, customer service satisfaction score, and assistant satisfaction score are extracted as user feedback data for the large language model to be evaluated.

[0035] In some embodiments, the weight calculation of each evaluation dimension of the large language model to be evaluated according to the actual performance data and the user feedback data, and the dynamic adjustment of the weight of each evaluation dimension of the large language model to be evaluated according to the calculation result, include:

[0036] Performing data cleaning and preprocessing on the actual performance data and the user feedback data, and performing statistical analysis on the processed actual performance data and the user feedback data, respectively, to obtain evaluation dimension statistical results and user feedback statistical results;

[0037] Calculate the adjustment factor of each evaluation dimension based on the evaluation dimension statistics and user feedback statistics;

[0038] Define the loss function and combine it with adaptive moment estimation to calculate the gradient, and dynamically update the weights based on the gradient calculation results;

[0039] The updated weights are normalized to verify the dynamic weight adjustment results.

[0040] In some embodiments, calculating the comprehensive score of the large language model to be evaluated according to the dynamically adjusted weight, and selecting the large language model with the highest comprehensive score from the large language models to be evaluated as the optimal large language model, includes:

[0041] Collect the score records of the large language model to be evaluated under various evaluation dimensions;

[0042] Get the dynamically adjusted weights and calculate the comprehensive score of each large language model to be evaluated according to the comprehensive scoring formula:

[0043]

[0044] in, Indicates The comprehensive score of the large language model to be evaluated, Indicates The weight of the evaluation dimension, Indicates The large language model to be evaluated is Scoring records for each assessment dimension;

[0045] The calculated comprehensive scores of all large language models to be evaluated are sorted, and the large language model with the highest comprehensive score is selected as the optimal large language model.

[0046] In the second aspect, the present application proposes a large language model selection system based on multi-dimensional evaluation and dynamic weight adjustment, including a model extraction module, a multi-dimensional evaluation module, a weight adjustment module and a model scoring module;

[0047] The model extraction module is used to select any number of pre-trained large language models as large language models to be evaluated;

[0048] The multi-dimensional evaluation module is used to evaluate the text generation function, dialogue question-answering function and Web application function of the large language model to be evaluated, and extract the operation process data in each evaluation dimension from each evaluation process as the actual performance data of the large language model to be evaluated, and extract the user rating data from each evaluation result as the user feedback data of the large language model to be evaluated;

[0049] The weight adjustment module is used to calculate the weight of each evaluation dimension of the large language model to be evaluated according to the actual performance data and the user feedback data, and dynamically adjust the weight of each evaluation dimension of the large language model to be evaluated according to the calculation result;

[0050] The model scoring module is used to calculate the comprehensive score of the large language model to be evaluated according to the dynamically adjusted weight, and select the large language model with the highest comprehensive score from the large language models to be evaluated as the optimal large language model.

[0051] In a third aspect, the present application proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0052] In a fourth aspect, the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0053] Beneficial effects of the present invention:

[0054] The present invention aims to solve the problems of low text generation quality, inaccurate dialogue questions and answers, and low online assistant task completion rate in existing large language models by introducing a multi-dimensional evaluation mechanism. Specifically, the present invention will comprehensively consider multiple evaluation dimensions such as ROUGE indicator evaluation, text quality evaluation, semantic similarity evaluation, security compliance evaluation, positive suitability evaluation, question and answer evaluation, dialogue evaluation, online customer service evaluation, and online assistant evaluation, and comprehensively evaluate the performance of the large language model. By analyzing the operation data and user feedback, the weights of each evaluation dimension are dynamically adjusted to adapt to the ever-changing application requirements and user expectations. Finally, according to the comprehensive score, the optimal large language model is automatically selected to ensure the high quality and user satisfaction of the Web service. This method not only enhances the adaptability and user-friendliness of the system, but also provides new directions and possibilities for the further development of natural language processing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is the overall flow chart of the present invention.

[0056] Figure 2 It is a system principle block diagram of the present invention. DETAILED DESCRIPTION

[0057] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein; on the contrary, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0058] In a first aspect, the present application proposes a large language model selection method based on multi-dimensional evaluation and dynamic weight adjustment, comprising the following steps:

[0059] S100: Select any number of pre-trained large language models as large language models to be evaluated;

[0060] S200: Performing text generation function evaluation, dialogue question answering function evaluation, and Web application function evaluation on the large language model to be evaluated,

[0061] In some embodiments, the text generation function evaluation, the dialogue question and answer function evaluation and the Web application function evaluation are performed on the large language model to be evaluated, and the text generation function evaluation includes:

[0062] Generate a set of text data through the large language model to be evaluated, and perform ROUGE indicator evaluation, text quality evaluation, semantic similarity evaluation, security compliance evaluation, and positive suitability evaluation on the generated text;

[0063] The ROUGE indicator evaluation: quantifies the accuracy of the generated text by calculating the number of N-gram overlaps between the generated text and the reference text;

[0064] ROUGE is a set of metrics used to automatically evaluate machine translation and automatic summarization. Specifically, it includes ROUGE-N (N-gram matching), ROUGE-L (longest common subsequence matching), etc. By calculating the number of N-gram overlaps between the generated text and the reference text, the accuracy of the generated text can be quantified.

[0065] The text quality assessment: the generated text is evaluated by a pre-trained language model to obtain a corresponding text quality score to quantify language fluency and syntactic correctness;

[0066] For example, the generated text is evaluated by GPT-3 to obtain a quality score. This score can be obtained by the perplexity of the model or by training a dedicated quality assessment model. This method can capture quality information such as language fluency and syntactic correctness.

[0067] The semantic similarity evaluation: uses the pre-trained semantic similarity model to calculate the semantic similarity between the generated text and the reference text. The formula for quantifying the semantic proximity between the generated text and the reference text is as follows:

[0068]

[0069] in, and The vector representations of the generated text and the reference text are respectively used, and the semantic similarity between the generated text and the reference text is calculated using a pre-trained semantic similarity model (such as BERT, Sentence-BERT). The text is embedded into a high-dimensional vector space and the cosine similarity between the two vectors is calculated.

[0070] The security compliance assessment: classifies the generated text through a pre-trained classification model to determine whether it contains sensitive words or content that violates the policy. If the generated text does not contain sensitive words or content that violates the policy, the higher the score, otherwise the lower the score;

[0071] The positive suitability assessment is as follows: the generated text is input into the sentiment analysis model to obtain sentiment polarity and sentiment intensity scores. If the text sentiment is positive and the intensity meets the preset standard, the higher the score, and if the text sentiment is negative or the intensity does not meet the preset standard, the lower the score.

[0072] Among them, the sentiment analysis model can be a pre-trained sentiment classifier (such as VADER, TextBlob), which can identify the sentiment polarity (positive, neutral, negative) and sentiment intensity in the text. By calculating the sentiment score of the generated text, it can be evaluated whether it is suitable for a specific application scenario.

[0073] In order to better understand the specific implementation steps of text generation function evaluation, take the evaluation of a generated text as an example:

[0074] 1. ROUGE evaluation: Assume that the generated text is "The cat is sleeping in the sun" and the reference text is "The cat is resting in the sun". First, calculate the N-gram matching between the two. For example, calculate the number of overlaps of ROUGE-1 (word matching) and calculate the ROUGE-1 score according to the formula.

[0075] 2. Text quality assessment: Use a large language model to calculate the perplexity of the generated text. The lower the perplexity, the higher the quality of the text. Alternatively, use a specially trained quality assessment model to score the text and get a quality score.

[0076] 3. Semantic similarity assessment: Input the generated text and the reference text into the large language model to obtain their vector representations. Calculate the cosine similarity between the two vectors to obtain the semantic similarity score.

[0077] 4. Security compliance assessment: The generated text is input into the sensitive word detection model to determine whether it contains sensitive information. If it does, its score is lowered. Then, the text is input into the policy compliance classification model to determine whether it is compliant.

[0078] 5. Positive suitability assessment: The generated text is input into the sentiment analysis model to obtain sentiment polarity and sentiment intensity scores. If the text sentiment is positive and the intensity is moderate, the score is high; if the sentiment is negative or the intensity is too high, the score is low.

[0079] In some embodiments, the large language model to be evaluated is evaluated for text generation function, dialogue question and answer function, and Web application function, wherein the dialogue question and answer function evaluation includes question and answer evaluation and dialogue evaluation:

[0080] The question-answering evaluation is:

[0081] Prepare a set of assessment questions and corresponding model answers;

[0082] Inputting the evaluation question into the large language model to be evaluated to obtain a model-generated answer;

[0083] The BLEU indicator is used to evaluate the similarity between the generated answer and the standard answer. The higher the similarity, the higher the corresponding score, and vice versa. The user's question and answer satisfaction score is collected through user feedback;

[0084] The dialogue evaluation is:

[0085] Design a set of dialogues that simulate actual application scenarios;

[0086] The designed dialogue is input into the large language model to be evaluated for multiple rounds of dialogue. The fluency of the dialogue is scored using the pre-trained language model. The consistency and logical rationality of the dialogue are analyzed and scored using the dialogue management model. The user's dialogue satisfaction score is collected through user feedback.

[0087] The following is a detailed description using the dialogue simulation evaluation as an example:

[0088] 1. Conversation scenario preparation: Assume that you want to evaluate the online customer service function of a large language model. First, design a set of common customer service conversation scenarios, such as users inquiring about product information, checking order status, solving after-sales problems, etc.

[0089] 2. Dialogue: These dialogue scenarios are input into the large language model to be evaluated one by one, and multiple rounds of dialogue are conducted. For example, the user asks: "Can I check the status of my order?" The answer generated by the model may be: "Of course, please provide your order number."

[0090] 3. Evaluation indicators:

[0091] – Fluency: Evaluate whether the dialogue generated by the model is natural and fluent. For example, check whether the model’s answers are grammatically correct and whether the sentences are coherent.

[0092] – Logic: Evaluate the logical rationality of the conversation. For example, check whether the model’s answer is relevant to the user’s question and whether it can provide a reasonable answer based on the conversation context.

[0093] – User satisfaction: Collect user feedback on the conversation and evaluate the user’s satisfaction with the model’s answers. For example, through a user rating system, let the user rate each round of conversation.

[0094] 4. Specific methods:

[0095] – Fluency Assessment: Use a pre-trained language model to score the fluency of the conversation.

[0096] –Logical evaluation: Use the dialogue management model to analyze the consistency and logical rationality of the dialogue to ensure that the model’s answer is consistent with the dialogue context.

[0097] User satisfaction evaluation: Collect user ratings through the user feedback system and calculate the average satisfaction rating.

[0098] In some embodiments, the large language model to be evaluated is evaluated for text generation function, dialogue question and answer function, and Web application function, and the Web application function evaluation includes online customer service evaluation and online assistant evaluation:

[0099] The online customer service evaluation is:

[0100] Design a variety of user consultation scenarios, input the user consultation scenarios into the large language model to be evaluated, simulate the conversation between real users and customer service, record the time it takes for the large language model to generate a reply, complete the response time evaluation, check the accuracy of the model's answer, complete the accuracy evaluation, and collect the user's customer service satisfaction score through user feedback;

[0101] The online assistant is evaluated as:

[0102] Set a variety of online assistant tasks, input the online assistant tasks into the large language model to be evaluated, simulate the interaction process between the user and the online assistant, calculate the task completion rate through the statistical model, and collect the user's assistant satisfaction score through user feedback.

[0103] Among them, the online assistant evaluation is taken as an example for detailed description:

[0104] 1. Task setting: Suppose we want to evaluate the online assistant function of a large language model. First, set some common online assistant tasks, such as "help me check tomorrow's weather", "remind me of the meeting at 3 pm", "find nearby restaurants", etc.

[0105] 2. Task execution: These tasks are input into the large language model to be evaluated one by one to simulate the interaction between the user and the online assistant. For example, if the user inputs "Help me check tomorrow's weather.", the response generated by the model may be "Tomorrow's weather forecast shows cloudy, with a temperature between 15 and 20 degrees."

[0106] 3. Evaluation indicators:

[0107] – Task completion rate: evaluates whether the model has successfully completed the task set. For example, check whether the model correctly queries tomorrow's weather information.

[0108] – User experience: Collect user ratings of their experience interacting with the online assistant through a user feedback system. For example, users can rate the model’s answers (e.g., 1 to 5 stars).

[0109] 4. Specific methods:

[0110] –Task completion rate evaluation: Statistical model success rate in multiple task executions. For example, if 10 tasks are set and the model successfully completes 8, the task completion rate is 80%.

[0111] – User experience evaluation: Collect experience ratings from multiple users and calculate the average score. For example, if 10 users give ratings of 4, 5, 3, 4, 5, 4, 5, 3, 4, 5, the average score is 4.2.

[0112] In some embodiments, extracting the running process data in each evaluation dimension from each evaluation process as the actual performance data of the large language model to be evaluated, and extracting the user rating data from each evaluation result as the user feedback data of the large language model to be evaluated, includes:

[0113] Extract the running process data of ROUGE indicator evaluation, text quality evaluation, semantic similarity evaluation, security compliance evaluation, positive suitability evaluation, question-answering evaluation, dialogue evaluation, online customer service evaluation, and online assistant evaluation as the actual performance data of the large language model to be evaluated;

[0114] The question-and-answer satisfaction score, conversation satisfaction score, customer service satisfaction score, and assistant satisfaction score are extracted as user feedback data for the large language model to be evaluated.

[0115] S300: Calculating weights of various evaluation dimensions of the large language model to be evaluated according to the actual performance data and the user feedback data, and dynamically adjusting weights of various evaluation dimensions of the large language model to be evaluated according to the calculation results;

[0116] In some embodiments, the weight calculation of each evaluation dimension of the large language model to be evaluated according to the actual performance data and the user feedback data, and the dynamic adjustment of the weight of each evaluation dimension of the large language model to be evaluated according to the calculation result, include:

[0117] Performing data cleaning and preprocessing on the actual performance data and the user feedback data, and performing statistical analysis on the processed actual performance data and the user feedback data, respectively, to obtain evaluation dimension statistical results and user feedback statistical results;

[0118] The actual performance data and the user feedback data are cleaned, invalid data is removed, and necessary preprocessing is performed, such as time format conversion, data type conversion, etc. The actual performance data and the user feedback data after preprocessing are statistically analyzed, and statistical indicators such as the average value and standard deviation of each evaluation dimension are calculated. For example, suppose a model performs as follows in multiple evaluation dimensions: the average score of text quality evaluation is 4.5, the average score of online customer service evaluation is 4.2, and the average score of user customer service satisfaction is 3.8.

[0119] Calculate the adjustment factor of each evaluation dimension based on the evaluation dimension statistics and user feedback statistics;

[0120] The adjustment factor can be calculated based on the performance of the evaluation dimension and the user satisfaction score. For example, a baseline satisfaction score , if the user satisfaction score of a certain evaluation dimension Higher than , then increase the weight of this dimension; otherwise, reduce the weight of this dimension.

[0121] Define the loss function and combine it with adaptive moment estimation to calculate the gradient, and dynamically update the weights based on the gradient calculation results;

[0122] Define a loss function L to measure the difference between the evaluation results and the actual user feedback or benchmark satisfaction. The loss function can be defined as the mean square error between the user satisfaction score and the benchmark satisfaction score;

[0123]

[0124] Use the AdamW optimizer to calculate gradients and update weights: In each iteration, the AdamW optimizer calculates gradients and updates weights using the following formula:

[0125]

[0126] in, Indicates that at time step Time Weight The first-order momentum (i.e., the loss function About weight The weighted average of the partial derivatives of Represents the first-order momentum decay rate, which is usually set to a value close to 1 (such as 0.9) to control the decay rate of past gradients. Indicates that at time step The first-order momentum at time , Represents the loss function Weight The instant gradient of Represents the weight of the instantaneous gradient, ensuring the contribution of the current gradient, Indicates that at time step Time Weight The second-order momentum (i.e., the loss function About weight The weighted average of the squares of the partial derivatives of Represents the second-order momentum decay rate, which is usually set to a value close to 1 (such as 0.999) to control the decay rate of the past squared gradient. Indicates that at time step The second-order momentum at represents the square of the instantaneous gradient, Represents the weight of the instantaneous squared gradient.

[0127] : The weight of the instantaneous squared gradient

[0128] because and Usually close to 1, so and In the initial stage, it may be biased towards 0. To correct this bias, and Make the following corrections:

[0129]

[0130]

[0131] Finally, the weight update formula is:

[0132]

[0133] in, Express Bias-corrected first-order momentum. Indicates the current time step. express of Power. is a small constant used to prevent the denominator from being zero. Express Second-order momentum after bias correction. Represents the current uncorrected second-order momentum. express of Power. Represents the weight decay rate, used for Regularization to prevent overfitting. Indicates that at time step The weight of time. Represents the learning rate, which controls the step size of weight update. Indicates that at time step This formula is continuously run in the training process of the neural network, and the weight vector is updated in each iteration. , until a certain stopping condition is reached, such as reaching a predetermined number of iterations, the loss function value is lower than a certain threshold, or the performance on the validation set no longer improves.

[0134] The updated weights are normalized to verify the dynamic weight adjustment results.

[0135] Normalize the updated weights to ensure that the sum of all weights is 1. The normalization formula is as follows:

[0136]

[0137] in, is the normalized weight, is the updated weight, is the total number of evaluation dimensions, Indicates The weight of each evaluation dimension.

[0138] S400: Calculate the comprehensive scores of the large language models to be evaluated according to the dynamically adjusted weights, and select the large language model with the highest comprehensive score from the large language models to be evaluated as the optimal large language model.

[0139] In some embodiments, calculating the comprehensive score of the large language model to be evaluated according to the dynamically adjusted weight, and selecting the large language model with the highest comprehensive score from the large language models to be evaluated as the optimal large language model, includes:

[0140] Collect the score records of the large language model to be evaluated under various evaluation dimensions;

[0141] Collect the scores of each evaluation dimension. Suppose we have N large language models, each model has Each evaluation dimension has a corresponding score record. The score of the large language model on the Kth evaluation dimension is

[0142] Get the dynamically adjusted weights and calculate the comprehensive score of each large language model to be evaluated according to the comprehensive scoring formula:

[0143]

[0144] in, Indicates The comprehensive score of the candidate models, is the normalized weight, Indicates The score record of a large language model on the Kth evaluation dimension;

[0145] The calculated comprehensive scores of all large language models to be evaluated are sorted, and the large language model with the highest comprehensive score is selected as the optimal large language model.

[0146] Furthermore, by combining the scores of each dimension and the dynamically adjusted weights, a comprehensive score is calculated and the optimal model is automatically selected. This embodiment takes five dimensions, namely, ROUGE indicator evaluation, text quality evaluation, semantic similarity evaluation, security compliance evaluation, and positive suitability evaluation, as examples for explanation. The specific steps are as follows:

[0147] Selection of nonlinear weighting method for each dimension:

[0148] The most appropriate non-linear weighting method will be selected for each evaluation dimension:

[0149] 1. ROUGE indicator evaluation: Use the ReLU function. ReLU can highlight high scores and suppress low scores, and is suitable for emphasizing high-quality matching. Among them:

[0150]

[0151]

[0152] 2. Text quality assessment: Use the Sigmoid function. Sigmoid can compress the score to between 0 and 1, which is suitable for smoothing the difference in text quality. Among them:

[0153]

[0154]

[0155] 3. Semantic similarity evaluation: Use the Softmax function. Softmax can highlight the most similar results and is suitable for comparing the relative similarity between multiple models.

[0156]

[0157] 4. Security compliance evaluation: Use the ReLU function. Similar to ROUGE, we want to emphasize highly secure and compliant content.

[0158]

[0159] 5. Positive suitability evaluation: Use Sigmoid function. Similar to text quality, we want to smoothly evaluate the positivity of the content.

[0160] Comprehensive score calculation:

[0161] Add the nonlinear weighted scores of each dimension to get the final comprehensive score:

[0162]

[0163] Model selection: Sort the comprehensive scores of all candidate models and select the model with the highest comprehensive score as the optimal model.

[0164] In order to more intuitively illustrate this improved automatic model selection process, we use a specific example to illustrate:

[0165] Assume that there are three candidate large language models (models A, B, and C), and the original scores on the six evaluation dimensions are shown in Table 1:

[0166] Table 1

[0167]

[0168] Now, we calculate the comprehensive score of each model according to the new scheme:

[0169] For Model A:

[0170]

[0171]

[0172] Similar calculations were performed on models B and C, and the comprehensive scores of the three models were compared, and the model with the highest score was selected as the optimal model.

[0173] In the second aspect, the present application proposes a large language model selection system based on multi-dimensional evaluation and dynamic weight adjustment, including a model extraction module, a multi-dimensional evaluation module, a weight adjustment module and a model scoring module;

[0174] The model extraction module is used to select any number of pre-trained large language models as large language models to be evaluated;

[0175] The multi-dimensional evaluation module is used to evaluate the text generation function, dialogue question-answering function and Web application function of the large language model to be evaluated, and extract the operation process data in each evaluation dimension from each evaluation process as the actual performance data of the large language model to be evaluated, and extract the user rating data from each evaluation result as the user feedback data of the large language model to be evaluated;

[0176] The weight adjustment module is used to calculate the weight of each evaluation dimension of the large language model to be evaluated according to the actual performance data and the user feedback data, and dynamically adjust the weight of each evaluation dimension of the large language model to be evaluated according to the calculation result;

[0177] The model scoring module is used to calculate the comprehensive score of the large language model to be evaluated according to the dynamically adjusted weight, and select the large language model with the highest comprehensive score from the large language models to be evaluated as the optimal large language model.

[0178] In a third aspect, the present application proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0179] In a fourth aspect, the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0180] The above are only preferred implementations of the present invention. It should be pointed out that a number of modifications and improved technical solutions made by those skilled in the art without departing from the technical solution should also be deemed to fall within the scope of protection required by the claims.

Claims

1. A large language model selection method based on multi-dimensional evaluation and dynamic weight adjustment, characterized by: The following steps are involved: Select any number of pre-trained large language models as the large language models to be evaluated; Performing text generation function evaluation, dialogue question-answering function evaluation, and Web application function evaluation on the large language model to be evaluated, and extracting operation process data in each evaluation dimension from each evaluation process as actual performance data of the large language model to be evaluated, and extracting user rating data from each evaluation result as user feedback data of the large language model to be evaluated; Calculate the weight of each evaluation dimension of the large language model to be evaluated according to the actual performance data and the user feedback data, and dynamically adjust the weight of each evaluation dimension of the large language model to be evaluated according to the calculation result; The comprehensive score of the large language model to be evaluated is calculated according to the dynamically adjusted weight, and the large language model with the highest comprehensive score is selected from the large language models to be evaluated as the optimal large language model.

2. The method according to claim 1, characterized in that: The text generation function evaluation, dialogue question and answer function evaluation and Web application function evaluation are performed on the large language model to be evaluated, and the text generation function evaluation includes: Generate a set of text data through the large language model to be evaluated, and perform ROUGE indicator evaluation, text quality evaluation, semantic similarity evaluation, security compliance evaluation, and positive suitability evaluation on the generated text; The ROUGE indicator evaluation: quantifies the accuracy of the generated text by calculating the number of N-gram overlaps between the generated text and the reference text; The text quality assessment: the generated text is evaluated by a pre-trained language model to obtain a corresponding text quality score to quantify language fluency and syntactic correctness; The semantic similarity evaluation: uses the pre-trained semantic similarity model to calculate the semantic similarity between the generated text and the reference text. The formula for quantifying the semantic proximity between the generated text and the reference text is as follows: in, and are the vector representations of generated text and reference text respectively; The security compliance assessment: classifies the generated text through a pre-trained classification model to determine whether it contains sensitive words or content that violates the policy. If the generated text does not contain sensitive words or content that violates the policy, the higher the score, otherwise the lower the score; The positive suitability assessment is as follows: the generated text is input into the sentiment analysis model to obtain sentiment polarity and sentiment intensity scores. If the text sentiment is positive and the intensity meets the preset standard, the higher the score, and if the text sentiment is negative or the intensity does not meet the preset standard, the lower the score.

3. The method according to claim 2, characterized in that: The text generation function evaluation, dialogue question and answer function evaluation and Web application function evaluation are performed on the large language model to be evaluated, and the dialogue question and answer function evaluation includes question and answer evaluation and dialogue evaluation: The question-answering evaluation is: Prepare a set of assessment questions and corresponding model answers; Inputting the evaluation question into the large language model to be evaluated to obtain a model-generated answer; The BLEU indicator is used to evaluate the similarity between the generated answer and the standard answer. The higher the similarity, the higher the corresponding score, and vice versa. The user's question and answer satisfaction score is collected through user feedback; The dialogue evaluation is: Design a set of dialogues that simulate actual application scenarios; The designed dialogue is input into the large language model to be evaluated for multiple rounds of dialogue. The fluency of the dialogue is scored using the pre-trained language model. The consistency and logical rationality of the dialogue are analyzed and scored using the dialogue management model. The user's dialogue satisfaction score is collected through user feedback.

4. The method according to claim 3, characterized in that: The text generation function evaluation, dialogue question and answer function evaluation and Web application function evaluation of the large language model to be evaluated, wherein the Web application function evaluation includes online customer service evaluation and online assistant evaluation: The online customer service evaluation is: Design a variety of user consultation scenarios, input the user consultation scenarios into the large language model to be evaluated, simulate the conversation between real users and customer service, record the time it takes for the large language model to generate a reply, complete the response time evaluation, check the accuracy of the model's answer, complete the accuracy evaluation, and collect the user's customer service satisfaction score through user feedback; The online assistant is evaluated as: Set a variety of online assistant tasks, input the online assistant tasks into the large language model to be evaluated, simulate the interaction process between the user and the online assistant, calculate the task completion rate through the statistical model, and collect the user's assistant satisfaction score through user feedback.

5. The method according to claim 4, characterized in that: The process data in each evaluation dimension is extracted from each evaluation process as the actual performance data of the large language model to be evaluated, and the user rating data is extracted from each evaluation result as the user feedback data of the large language model to be evaluated, including: Extract the running process data of ROUGE indicator evaluation, text quality evaluation, semantic similarity evaluation, security compliance evaluation, positive suitability evaluation, question-answering evaluation, dialogue evaluation, online customer service evaluation, and online assistant evaluation as the actual performance data of the large language model to be evaluated; The question-and-answer satisfaction score, conversation satisfaction score, customer service satisfaction score, and assistant satisfaction score are extracted as user feedback data for the large language model to be evaluated.

6. The method according to claim 5, characterized in that: The weight calculation of each evaluation dimension of the large language model to be evaluated is performed according to the actual performance data and the user feedback data, and the weight of each evaluation dimension of the large language model to be evaluated is dynamically adjusted according to the calculation result, including: Performing data cleaning and preprocessing on the actual performance data and the user feedback data, and performing statistical analysis on the processed actual performance data and the user feedback data, respectively, to obtain evaluation dimension statistical results and user feedback statistical results; Calculate the adjustment factor of each evaluation dimension based on the evaluation dimension statistics and user feedback statistics; Define the loss function and combine it with adaptive moment estimation to calculate the gradient, and dynamically update the weights based on the gradient calculation results; The updated weights are normalized to verify the dynamic weight adjustment results.

7. The method according to claim 6, characterized in that: The step of calculating the comprehensive score of the large language model to be evaluated according to the dynamically adjusted weight, and selecting the large language model with the highest comprehensive score from the large language models to be evaluated as the optimal large language model, comprises: Collect the score records of the large language model to be evaluated under various evaluation dimensions; Get the dynamically adjusted weights and calculate the comprehensive score of each large language model to be evaluated according to the comprehensive scoring formula: in, Indicates The comprehensive score of the large language model to be evaluated, Indicates The weight of the evaluation dimension, Indicates The large language model to be evaluated is Scoring records for each assessment dimension; The calculated comprehensive scores of all large language models to be evaluated are sorted, and the large language model with the highest comprehensive score is selected as the optimal large language model.

8. A large language model selection system based on multi-dimensional evaluation and dynamic weight adjustment, characterized by: It includes model extraction module, multi-dimensional evaluation module, weight adjustment module and model scoring module; The model extraction module is used to select any number of pre-trained large language models as large language models to be evaluated; The multi-dimensional evaluation module is used to evaluate the text generation function, dialogue question-answering function and Web application function of the large language model to be evaluated, and extract the operation process data in each evaluation dimension from each evaluation process as the actual performance data of the large language model to be evaluated, and extract the user rating data from each evaluation result as the user feedback data of the large language model to be evaluated; The weight adjustment module is used to calculate the weight of each evaluation dimension of the large language model to be evaluated according to the actual performance data and the user feedback data, and dynamically adjust the weight of each evaluation dimension of the large language model to be evaluated according to the calculation result; The model scoring module is used to calculate the comprehensive score of the large language model to be evaluated according to the dynamically adjusted weight, and select the large language model with the highest comprehensive score from the large language models to be evaluated as the optimal large language model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images