Text evaluation method, electronic equipment and storage medium
By constructing multiple large models and customizing evaluation strategies, the problems of high cost and poor flexibility in existing text evaluation technologies have been solved, achieving efficient and flexible text evaluation that can adapt to the evaluation needs of different business scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-04
- Publication Date
- 2026-04-07
AI Technical Summary
Existing text evaluation methods require the collection of a large amount of manually labeled data, which is costly, time-consuming, and lacks flexibility and applicability to different scenarios, making it difficult to adapt to the text evaluation needs of different business scenarios.
By pre-building multiple large models with text evaluation capabilities, and adopting single-model mode, multi-model voting mode, or multi-model proxy mode according to business scenario requirements, a customized evaluation strategy is generated to perform text evaluation.
It reduces model training costs and time, improves the flexibility and scenario applicability of text evaluation strategies, and enhances the accuracy and diversity of evaluation results.
Smart Images
Figure CN121809458A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and more particularly to a text evaluation method, electronic device, and storage medium. Background Technology
[0002] With the continuous development of artificial intelligence technology, many application scenarios have emerged that utilize large models to generate text-based data (such as instant messaging and human-computer dialogue). The text-based data generated in these scenarios can be used to assist user decision-making or conduct in-depth data analysis. Therefore, the quality and accuracy of the generated text-based data are crucial, hence the use of text evaluation techniques to assess this data.
[0003] Currently, the main method for evaluating text-based data is through automatic model evaluation. This method requires collecting a large amount of manually labeled text data for model training to obtain a model capable of automatically evaluating text-based data. This model can then be used to evaluate newly generated text-based data. However, this text evaluation method is costly, time-consuming, and the resulting model has poor flexibility and applicability to various scenarios. Summary of the Invention
[0004] To address, or at least partially address, the aforementioned technical problems, this disclosure provides a text evaluation method, an electronic device, and a storage medium.
[0005] In a first aspect, embodiments of this disclosure provide a text evaluation method, the method comprising: The text to be evaluated, the target evaluation mode, and the target strategy parameters are determined; wherein, the target evaluation mode is one of multiple candidate evaluation modes, and the candidate evaluation modes include single-model mode, multi-model voting mode, or multi-model proxy mode; the target strategy parameters include the model identifier of at least one base model; Based on the target strategy parameters and the target evaluation mode, a target evaluation strategy is generated; The target evaluation strategy is used to perform text evaluation on the text to be evaluated, and target evaluation results are generated.
[0006] Secondly, embodiments of this disclosure also provide a text evaluation device, the device comprising: The information determination module is used to determine the text to be evaluated, the target evaluation mode, and the target strategy parameters; wherein, the target evaluation mode is one of multiple candidate evaluation modes, and the candidate evaluation modes include a single-model mode, a multi-model voting mode, or a multi-model proxy mode; the target strategy parameters include the model identifier of at least one base model; The evaluation strategy generation module is used to generate a target evaluation strategy based on the target strategy parameters and the target evaluation mode; The text evaluation module is used to perform text evaluation on the text to be evaluated using the target evaluation strategy and generate target evaluation results.
[0007] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising: Processor and memory; The processor executes the text evaluation method described in any embodiment of this disclosure by calling programs or instructions stored in the memory.
[0008] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a program or instructions that cause a computer to execute the text evaluation method described in any embodiment of this disclosure.
[0009] Fifthly, embodiments of this disclosure also provide a computer program product for executing the text evaluation method described in any embodiment of this disclosure.
[0010] The text evaluation method, electronic device, and storage medium provided in this disclosure can select a suitable target evaluation mode from multiple candidate evaluation modes with text evaluation functions, and determine target policy parameters containing model identifiers of at least one base model, so that the target policy parameters can be applied to the target evaluation mode to generate a target evaluation strategy. Then, the target evaluation strategy is used to automatically evaluate the text to be evaluated to obtain the target evaluation result. Compared with the related technologies that have only one text evaluation model dedicated to a specific business scenario, the method of determining the target evaluation strategy through the combination of at least one base model and target policy parameters can not only reduce the cost and time consumption of model training, but also improve the diversity and flexibility of the target evaluation strategy, making the target evaluation strategy more adaptable to the text to be evaluated, thereby improving the scenario applicability of the target evaluation strategy and the accuracy of the text evaluation results. Attached Figure Description
[0011] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1A flowchart illustrating a text evaluation method provided in an embodiment of this disclosure; Figure 2 A schematic diagram of the process for generating a basic model in a text evaluation method provided in this embodiment of the disclosure; Figure 3 This is a schematic diagram of the structure of a text evaluation device provided in an embodiment of the present disclosure; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0014] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be described in further detail below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.
[0015] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.
[0016] Many related technologies utilize large models to generate text-based data, such as real estate agents' instant messaging conversations with users about properties, and dialogue data generated by various intelligent agents. This text-based data can be used to assist user decision-making or for data analysis in subsequent product development. Therefore, the quality and accuracy of the text-based data generated by large models are crucial. Related technologies utilize text evaluation techniques to assess the quality and accuracy of such text-based data. A common text evaluation method involves collecting a large amount of manually labeled text-based data for a specific business scenario, and then using this labeled text-based data to train a pre-trained large model with text evaluation capabilities to obtain a text evaluation model corresponding to the aforementioned business scenario. This text evaluation model can then evaluate newly generated text-based data in the aforementioned business scenario. However, this text evaluation solution requires collecting a large amount of manually labeled text-based data, which is time-consuming, labor-intensive, costly, and slow. Furthermore, the text evaluation model generated in this way is mainly applicable to the aforementioned business scenario and has poor applicability to other business scenarios, resulting in poor flexibility in text evaluation.
[0017] Based on the above, this disclosure proposes a technical solution for automatically evaluating text using generative models. This involves pre-constructing multiple large models (referred to as base models) with text evaluation capabilities. Then, according to business scenario requirements, the evaluation strategies of these base models are combined using a single-model mode, a multi-model voting mode, or a multi-model proxy mode to quickly obtain a target evaluation strategy adapted to the business scenario. Finally, the newly generated text data is evaluated using the target evaluation strategy. This eliminates the need for separate model training for each business scenario, reducing resource consumption, manpower consumption, and time consumption in generating text evaluation strategies. It also improves the flexibility and scenario applicability of the text evaluation strategy, thereby enhancing the accuracy of the text evaluation results. Furthermore, the scalability of the target evaluation strategy can be quickly achieved by adding or removing base models.
[0018] The text evaluation method provided in this disclosure is mainly applicable to scenarios involving automatic evaluation of text-based data. The text-based data can be text content generated using a large model or text content not generated by a large model. This text evaluation method can be executed by a text evaluation device, which can be implemented in software and / or hardware. The device can be integrated into an electronic device with certain data processing capabilities, such as a laptop, desktop computer, server, or server cluster.
[0019] Figure 1 This is a flowchart of a text evaluation method provided in an embodiment of this disclosure. See also... Figure 1 The text evaluation method specifically includes: S110. Determine the text to be evaluated, the target evaluation mode, and the target strategy parameters.
[0020] The text to be evaluated is text data that needs to be evaluated from one or more evaluation dimensions. These dimensions can be quality dimensions, accuracy dimensions, or dimensions of alignment with the task objective (such as whether it answers the question, whether it fits the business scenario, etc.). Specific evaluation dimensions can be determined by model prompts, such as by at least one of the evaluation criteria, evaluation examples, and evaluation task descriptions in the model prompts. The text to be evaluated can be the text content output by a large model in a specific business scenario and its contextual text content (referred to as a set of text content), or it can be multiple sets of text content. The target evaluation mode is one of several candidate evaluation modes.
[0021] The candidate evaluation mode serves as the framework for the text evaluation strategy, defining the overall logic and structure of the text evaluation. For example, it can specify whether one or more basic models participate in the text evaluation, and whether the collaboration between these basic models is a simple parallel approach or a complex interactive approach. In this embodiment, the candidate evaluation modes include a single-model mode, a multi-model voting mode, or a multi-model proxy mode. The single-model mode utilizes only one basic model for text evaluation. The multi-model voting mode uses multiple different basic models to evaluate the same text relatively independently, and then integrates the multiple evaluation results through a "voting" mechanism (such as majority voting or weighted fusion) to obtain a final text evaluation result. The multi-model proxy mode uses multiple different basic models as "proxy models," evaluating the same text through collaboration between these basic models. For example, one implementation of the multi-model proxy mode could be a multi-model discussion mode, where the basic models can evaluate the text in a manner similar to a group discussion. The aforementioned basic models are pre-built large models with text evaluation capabilities. Multiple base models can be large models with different model structures, or they can be large models with the same model structure but different model configurations (such as different roles).
[0022] The target strategy parameters are used to specify the specific configuration and execution details in the target strategy pattern. In this embodiment, the target strategy parameters include model identifiers for at least one base model. These model identifiers are identifying information about the base model, such as its model name, model number, etc. Thus, the target strategy parameters can at least specify which base models are used in the text evaluation within the target strategy pattern. Furthermore, the target strategy parameters may also include proprietary parameters corresponding to the target strategy pattern, such as parameters specifying the collaboration method between multiple base models.
[0023] Specifically, the electronic device can respond to relevant instructions to obtain the text to be evaluated, as well as the target evaluation mode and target strategy parameters applied to the text to be evaluated.
[0024] In some embodiments, when the electronic device corresponds to a server, it can determine the text to be evaluated, the target evaluation mode, and the target policy parameters in response to a received request. Specifically, S110 includes: receiving a text evaluation request, parsing the text evaluation request, and obtaining the text to be evaluated, the target evaluation mode, and the target policy parameters. The electronic device can receive text evaluation requests initiated by a client or sent by upstream or downstream services. Then, the electronic device parses the text evaluation request to obtain the text to be evaluated, the target evaluation mode, and the target policy parameters.
[0025] In other embodiments, when the electronic device corresponds to a client, the electronic device can present a relevant interactive interface for text evaluation to the user and execute S110 through interactive operations on that interface. Thus, S110 includes the following steps A to E.
[0026] Step A: Display the text evaluation interface, and show the text input component, mode selection component, and parameter configuration component in the text evaluation interface.
[0027] The text evaluation interface is the human-computer interaction interface that carries the text evaluation function. The text input component is an interactive component used to input the text to be evaluated; it can be an input box, a file upload control, etc. The mode selection component is an interactive component used to select candidate strategy modes; it can be a dropdown list, a radio button, etc. The parameter configuration component is an interactive component used to configure strategy parameters; it can be set to an input box, dropdown list, radio button, or checkbox, etc., depending on the characteristics of the parameter items.
[0028] Specifically, the electronic device can respond to the user's interactive operation of activating the text evaluation function by displaying a text evaluation interface, in which text input components, mode selection components, and parameter configuration components are displayed.
[0029] Step B: In response to the trigger operation of the text input component, the received text is identified as the text to be evaluated.
[0030] Specifically, users can input text data they want to evaluate by interacting with the text input component. For example, a user can paste at least one set of text content into an input box, or a user can upload a file containing at least one set of text content using a file upload control. The electronic device can then respond to these interactions by receiving the text content or file and identifying the corresponding at least one set of text content as the text to be evaluated.
[0031] Step C: In response to the trigger operation of the mode selection component, display the mode description information of each candidate evaluation mode and the candidate evaluation mode.
[0032] The pattern description information briefly introduces the evaluation principles and / or characteristics (such as advantages and disadvantages) of the candidate evaluation patterns. For example, when the pattern description information includes the evaluation characteristics of the candidate evaluation patterns, the pattern description information for a single-model pattern may emphasize evaluation speed and have relatively poor evaluation quality; the pattern description information for a multi-model voting pattern may balance evaluation speed and evaluation quality; and the pattern description information for a multi-model proxy pattern may emphasize evaluation quality and have relatively slow evaluation speed.
[0033] Specifically, the electronic device can respond to a user's triggering action on the mode selection component by displaying each candidate evaluation mode and its brief description information using controls such as pop-ups or floating windows. Alternatively, the electronic device can display each candidate evaluation mode in the mode selection component, and then, in response to a user's light interactive action on the candidate evaluation mode (such as mouse hover), present its corresponding brief description information in the surrounding area of the triggered candidate evaluation mode using controls such as pop-ups or floating bars. Alternatively, the electronic device can directly display multiple candidate evaluation modes and their corresponding brief description information in the mode selection component.
[0034] Step D: In response to the selection operation of the candidate evaluation mode, the selected candidate evaluation mode is determined as the target evaluation mode, and the parameter configuration items corresponding to the target evaluation mode are displayed in the parameter configuration component.
[0035] Specifically, users can select a candidate evaluation mode as needed, and the electronic device will identify it as the target evaluation mode. Considering that different candidate evaluation modes may have different strategy parameters, the electronic device can display an interactive component adapted to the target evaluation mode for setting these strategy parameters—namely, a parameter configuration item—within the parameter configuration component. The implementation of this parameter configuration item is adapted to the way the strategy parameters are set. For example, if the electronic device provides strategy parameters for candidate content (such as model identifier, collaboration method, etc.), the parameter configuration item can be implemented as a dropdown, radio button, or checkbox; if the electronic device does not provide strategy parameters for candidate content, the parameter configuration item can be implemented as an input box.
[0036] Step E: In response to the triggering operation of the parameter configuration item, the triggering result of the triggered parameter configuration item is determined as the target strategy parameter.
[0037] Specifically, users can select or input corresponding policy parameters by triggering the parameter configuration item. The electronic device then uses the content obtained from the triggering operation (referred to as the triggering result) as the target policy parameter corresponding to the triggered parameter configuration item.
[0038] For example, when the user selects a multi-mode voting mode or a multi-model proxy mode as the target evaluation mode, the electronic device can at least display the parameter configuration items for the number of models and model identifiers in the parameter configuration component. The user can set the parameter configuration item for the number of models to 3, then the parameter configuration items for 3 model identifiers can be displayed. The user can then trigger operations on the parameter configuration items for each model identifier to determine the 3 specific model identifiers. In this way, the target strategy parameters can be customized.
[0039] It should be noted that if the user does not trigger the parameter configuration item, the target strategy parameters corresponding to the target evaluation mode can use the default strategy parameters.
[0040] By setting steps A through E above, users can more quickly and clearly customize strategy modes and strategy parameters through human-computer interaction, thereby further improving the flexibility of the subsequent target evaluation strategy and the degree of fit between the target evaluation strategy and user needs, and thus further improving the accuracy of the subsequent evaluation results.
[0041] In some embodiments, if the target evaluation mode is a multi-model voting mode, the target strategy parameters also include model weight values and / or voting rules. Model weight values are used for weighted calculation when integrating the evaluation results output by each base model, and the corresponding voting rule is weighted voting. Voting rules are used to determine how to generate the final evaluation result from the evaluation results output by multiple base models, specifying the rules for integrating the evaluation results. Given that the text evaluation in this embodiment assesses the quality of text data, scores it, or selects the best from multiple texts, it can be categorized as a classification task. Therefore, the voting rules could be, for example, majority voting, weighted voting, or unanimous voting.
[0042] Therefore, in addition to the configuration items for the number of models and model identifiers described above, the parameter configuration items in step D above may also include parameter items for voting rules and / or model weight values.
[0043] In some embodiments, if the target evaluation mode is a multi-model proxy mode, the target strategy parameters also include a collaboration frequency threshold and / or collaboration method. The collaboration method refers to the way the base models collaborate with each other, which may include a group discussion where each base model acts as a member, a debate where each base model acts as a debater, or a review method where one base model is evaluated first while other base models act as reviewers for iterative review. The collaboration frequency threshold can be the maximum number of times the base models collaborate with each other, such as the maximum number of discussion rounds, the maximum number of debates, or the maximum number of reviews.
[0044] Therefore, the parameter configuration items in step D above may include, in addition to the configuration items for the number of models and model identifiers described above, parameter items for the collaboration frequency threshold and / or collaboration method.
[0045] By setting the above strategy parameters, the flexibility and diversity of the target evaluation strategy can be further improved, thereby further enhancing the reliability of the final evaluation results.
[0046] It should be noted that, in addition to the strategy parameters described in the aforementioned embodiments, there are other more complex strategy parameters in the multi-model voting mode and the multi-model proxy mode, which have higher understanding or configuration costs. These strategy parameters can be configured with default parameter values to reduce the application difficulty of text evaluation.
[0047] S120. Generate a target evaluation strategy based on the target strategy parameters and the target evaluation mode.
[0048] Specifically, the electronic device applies the aforementioned determined target strategy parameters to the target evaluation mode to obtain a target evaluation strategy that can be specifically executed.
[0049] For example, if the target evaluation mode is a single-model mode and the target strategy parameters include at least a model identifier, then the target evaluation strategy is to use the basic model corresponding to the model identifier to evaluate the text to be evaluated.
[0050] For example, if the target evaluation mode is a multi-model voting mode, and the target strategy parameters include multiple model identifiers corresponding to the number of models, voting rules for weighted voting, and model weight values, then the target evaluation strategy is to first use the basic model corresponding to each model identifier to evaluate the text to be evaluated, and then use the model weight values to weight and integrate the obtained evaluation results.
[0051] For example, if the target evaluation mode is a multi-model proxy mode, and the target strategy parameters include multiple model identifiers corresponding to the number of models, collaboration methods, and collaboration number thresholds, then the target evaluation strategy is to evaluate the text according to the basic model corresponding to each model identifier and the collaboration method, and end the text evaluation after the number of collaborations in the collaborative evaluation reaches the collaboration number threshold.
[0052] It should be noted that the specific details of text evaluation, such as the business background, evaluation steps, and evaluation criteria, are all determined through the model prompts of the base model. Thus, the target evaluation strategy not only includes the specific execution details of text evaluation by a single base model, but also the collaborative execution details between multiple base models, forming a complete and detailed text evaluation solution.
[0053] S130. Use the target evaluation strategy to perform text evaluation on the text to be evaluated and generate target evaluation results.
[0054] The target evaluation result is the final output of the text to be evaluated. It may include evaluation conclusions (such as good / bad, specific scores, comparisons with selected high-quality texts, etc.), as well as the reasons for generating the evaluation conclusions and / or the confidence level of the evaluation conclusions. The confidence level is a value in the range of 0-1, used to characterize the credibility of the evaluation conclusions. The higher the confidence level, the more credible the evaluation conclusions.
[0055] Specifically, the electronic device takes the text to be evaluated as input data, runs the target evaluation strategy to evaluate the text, and finally outputs the target evaluation result.
[0056] In some embodiments, if the target evaluation result includes a confidence level, then the output constraints in the target evaluation strategy include the method for calculating the confidence level. For example, for a single-model mode, the target evaluation strategy can specify the method for calculating the confidence level; for a multi-model voting mode, the target evaluation strategy can specify that the confidence level is calculated based on the voting ratio / model weight value; and for a multi-model proxy mode, the target evaluation strategy can specify that the confidence level is calculated based on the actual number of collaborations performed.
[0057] The text evaluation method provided in the above embodiments of this disclosure can select a target evaluation mode suitable for the text to be evaluated from multiple candidate evaluation modes with text evaluation functions, and determine the target strategy parameters containing at least one base model, so that the target strategy parameters can be applied to the target evaluation mode to generate a target evaluation strategy; then, the target evaluation strategy is used to automatically evaluate the text to be evaluated to obtain the target evaluation result. Compared with the related technologies that only have a text evaluation model dedicated to a specific business scenario, the method of determining the target evaluation strategy through the combination of at least one base model and the target strategy parameters can not only reduce the cost and time consumption of model training, but also improve the diversity and flexibility of the target evaluation strategy, so that the target evaluation strategy can be more adapted to the text to be evaluated, thereby improving the scenario applicability of the target evaluation strategy and the accuracy of the text evaluation results.
[0058] In some embodiments, after S130, the method further includes: if parsing the target evaluation result according to the preset data format fails, inputting the target evaluation result into the third generative model and converting the data format of the target evaluation result into the preset data format.
[0059] The preset data format is the output format of the target evaluation results, determined in the target evaluation strategy. The third generative model is a pre-obtained generative model with the ability to convert or rewrite data formats; for example, it could be a large-scale language model.
[0060] Specifically, the target evaluation strategy enforces constraints on the output format of the target evaluation results. However, because the base model may deviate from these constraints to varying degrees, the output format of the target evaluation results may not conform to the pre-defined data format. Therefore, after obtaining the target evaluation results, the electronic device first parses them using the pre-defined data format. If parsing is successful, it indicates that the output format of the target evaluation results is correct, and the evaluation conclusions, the reasons for generating the conclusions, and the confidence level of the conclusions can be extracted from them. If parsing fails, it indicates that the output format of the target evaluation results is incorrect. In this case, the electronic device inputs the target evaluation results into a third generative model to convert or rewrite its data format to the pre-defined data format. Then, the parsing process is repeated until successful. This ensures that the output format of the target evaluation results remains consistent with the required data format, thereby further improving the accuracy of the target evaluation results.
[0061] In some embodiments, after S130, the method further includes: if parsing the target evaluation result according to the preset data format fails, then when the model temperature corresponding to the target evaluation strategy does not reach the preset temperature, or the number of retries does not reach the preset number, the model temperature is increased according to the preset temperature difference, and S120 is re-triggered based on the increased model temperature.
[0062] Model temperature is a key parameter in large language models when generating text. It controls the randomness and determinism of the output. The lower the model temperature, the higher the determinism and the lower the randomness of the output; conversely, the higher the model temperature, the lower the determinism and the higher the randomness of the output. The preset temperature is the highest model temperature that can be received in the text evaluation scenario. To ensure the accuracy of the evaluation results, the preset temperature can be set to a small value, such as a value in the range of 0 to 0.7. The preset temperature difference is the preset range of model temperature adjustment.
[0063] Specifically, if the target evaluation result fails to be parsed, it means that the target evaluation result has not been output as required. In this case, the electronic device can initiate a retry mechanism by increasing the model temperature. In practice, the electronic device first determines whether the current model temperature has reached the preset temperature, or whether the number of retries has reached the preset maximum number of retries (i.e., the preset number). If either judgment result is yes, the retries stop. If both judgment results are no, then the retries can continue. At this time, the electronic device can start from the initial model temperature of 0, increase the model temperature by the preset temperature difference each time, and then re-execute S120 to re-output the target evaluation result until the target evaluation result is parsed successfully.
[0064] In some embodiments, after determining that the parsing of the target evaluation result has failed, the electronic device can first perform data format conversion processing as described above. If the parsing of the target evaluation result still fails after the conversion processing, the retry mechanism described above can be triggered. That is, after S130, the method includes: if parsing the target evaluation result according to the preset data format fails, the target evaluation result is input into the third generative model, and the data format of the target evaluation result is converted to the preset data format; if parsing the converted target evaluation result according to the preset data format fails, if the model temperature corresponding to the target evaluation strategy has not reached the preset temperature, or the number of retries has not reached the preset number, the model temperature is increased according to the preset temperature difference, and S120 is re-triggered based on the increased model temperature. This can further ensure the accuracy of the target evaluation result.
[0065] In some embodiments, after S130, the method further includes: if it is detected that the execution time corresponding to the text to be evaluated exceeds a preset time, then the evaluation processing of the text to be evaluated is terminated.
[0066] The preset duration is the maximum execution time for a text evaluation task, set in advance. Here, the text evaluation task corresponds to the text to be evaluated as defined in S110. Therefore, a text evaluation task can be for evaluating a set of text content or for evaluating multiple sets of text content.
[0067] Specifically, to prevent zombie processes from failing for extended periods due to system blockage, the electronic device assigns a globally unique task identifier (such as a task ID) to each text evaluation task and then monitors the execution time of the text evaluation process carrying that task ID. If the execution time exceeds a preset duration, the text evaluation task is considered blocked, and the electronic device can directly terminate the evaluation process of the text to be evaluated contained in that task. For example, the electronic device can kill the process corresponding to the task ID, freeing up computing resources to support other evaluation tasks. This improves the efficiency of text evaluation.
[0068] In some embodiments, to further improve the efficiency of text evaluation, electronic devices may also employ parallelized computational mechanisms to process the text evaluation process.
[0069] In one example, if the text to be evaluated contains multiple sets of text content, then regardless of the target evaluation strategy used, the electronic device can set the evaluation process of each set of text content to be executed in parallel to improve the evaluation speed.
[0070] In another example, for any set of text content, for a multi-model voting mechanism, the electronic device can set the text evaluation process of each basic model to be executed in parallel to further improve the evaluation speed.
[0071] In another example, for any set of text content, in a multi-model proxy mechanism, the electronic device can monitor whether there are evaluation processes for the underlying models in the collaboration method where they do not interact with each other's data. For example, in a collaboration method where each underlying model evaluates itself first and then summarizes and discusses / debates, the electronic device can execute the evaluation processes of the underlying models in parallel computing to further improve the evaluation speed.
[0072] Figure 2 This is a flowchart illustrating the generation of a basic pattern in a text evaluation method provided in this embodiment. In practical applications, although all are text evaluation business scenarios, different types of text data exhibit further subdivisions within specific business scenarios. This can lead to a well-trained large model not being well-suited to each subdivision. Retraining the large model for each subdivision would result in the aforementioned high cost and low efficiency issues. Therefore, this embodiment can fine-tune the large model by adjusting model prompts based on the already trained model, enabling it to better adapt to subdivisions. In this embodiment, model prompts can be divided into six modules: background description, evaluation criteria, evaluation method, mandatory requirements, evaluation examples, and evaluation task description. The background description primarily explains the business context, providing the large model with scenario-related knowledge corresponding to the subdivisions. The evaluation criteria are mainly used to clarify the quantitative or qualitative indicators for judging the quality of the evaluation results. The evaluation method, also known as the evaluation steps, clarifies the specific execution flow of the text evaluation. Mandatory requirements can be understood as mandatory constraints on the large model, such as constraints on output content and format. The evaluation examples are specific examples of text evaluation. The evaluation task description is a description of the specific text evaluation task to be performed.
[0073] The various modules of the aforementioned model suggestion words are affected differently by the specific business scenarios and the data formats and fields of the text data, and the process of writing model suggestion words is also time-consuming and labor-intensive. Therefore, this disclosure provides a solution to automatically generate model suggestion words using a small amount of labeled text data under specific business scenarios through a large model, so as to efficiently generate high-quality model suggestion words that are adapted to specific business scenarios and corresponding text data, thereby further reducing the difficulty of using large models for text evaluation and further improving the efficiency of text evaluation.
[0074] See Figure 2 The steps for generating the basic pattern in this text evaluation method specifically include: S210. Obtain the first number of text evaluation samples in the business scenario to which the text to be evaluated belongs.
[0075] The first quantity is a pre-set minimum number of text evaluation samples, such as a small value like 5 or 10. Text evaluation samples are training data used for model training in text evaluation scenarios. They include evaluation text examples (a set of text content) and reference evaluation results for the evaluation text examples (human-annotated evaluation results, such as scores or good / bad labels).
[0076] Specifically, in order to automatically generate high-quality and highly adaptable model prompts, a first number of text evaluation samples can be collected for the business scenario to which the text to be evaluated belongs.
[0077] S220. Input each text evaluation sample into the first generative model, obtain the model output results, and generate model prompt words adapted to the business scenario based on the model output results.
[0078] The first generative model is another pre-trained generative model that has the ability to automatically generate model prompts or the content of some modules within them.
[0079] Specifically, the electronic device takes each text evaluation sample as input data and feeds it into the first generative model. After model processing, the first generative model learns the text evaluation-related content corresponding to each text evaluation sample, obtaining the model output result. This model output result may include the model hint words from the aforementioned six modules, or it may only include the model hint words from some modules. Then, the electronic device can use the model output result to generate model hint words adapted to the business scenario of the text to be evaluated. For example, if the model output result includes the model hint words from all modules, the electronic device can convert or rewrite them according to the hint word format requirements to generate the required model hint words; or, if the model output result includes the model hint words from some modules, the electronic device can combine the model output result with the pre-written content corresponding to other modules to generate the required model hint words.
[0080] S230, Based on the model prompt words and the second number of second generative models, generate the base model.
[0081] The second quantity is another pre-defined quantity, whose value is an integer greater than or equal to 1. The second generative model is yet another pre-trained generative model with general text evaluation capabilities. There are differences in model structure among the various second generative models.
[0082] Specifically, electronic devices can apply model prompts to different second generative models, thereby obtaining different base models. Therefore, the number of base models is consistent with the second number of second generative models.
[0083] In some embodiments, S220 includes: if the model output includes a background description, evaluation criteria, and evaluation examples adapted to the business scenario, then a third number of model prompt words are generated based on the background description, evaluation criteria, evaluation examples, a third number of evaluation role information, and evaluation task description.
[0084] The third quantity is another pre-defined quantity, whose value is an integer greater than 1. The test role information is the information in the model prompts describing the role the model will play.
[0085] Specifically, among the six prompt word modules described above, the background description, evaluation criteria, and evaluation examples are significantly influenced by the business context and text-based data, and have a substantial impact on the evaluation results. Furthermore, it is relatively difficult to write high-quality corresponding content for these modules (e.g., it is difficult for users to determine a relatively objective evaluation criterion). The content of the remaining modules, however, is relatively easy to write and determine. Therefore, in this embodiment, the first generative model can automatically generate the background description, evaluation criteria, and evaluation examples corresponding to each text evaluation sample. Then, by combining the preset content corresponding to the remaining modules (at least one of the evaluation method, mandatory requirements, and evaluation task description), model prompt words can be generated.
[0086] Building upon the above, to further enhance the diversity of the base model, the electronic device can add relevant information describing the model's role, i.e., evaluation role information, to the aforementioned prompts, and set a third set of evaluation role information. In this way, each set of evaluation role information, combined with the content of the aforementioned prompts, yields a new model prompt. Thus, a third set of model prompts can be obtained, where the evaluation roles differ, but other content remains the same.
[0087] Based on the above embodiments, S230 includes: generating a base model based on a third number of model prompt words and a second number of second generative models.
[0088] The number of basic models is determined based on the second and third numbers.
[0089] Specifically, the electronic device can combine each model cue word with each second generative model to obtain a new base model. Therefore, in this embodiment, a maximum of (second number × third number) base models can be obtained. Some of these base models have the same model structure but different model cue words, some have the same model cue words but different model structures, and some have both different model cue words and model structures. This can greatly improve the diversity and scalability of the base models, thereby further enhancing the flexibility and diversity of the target evaluation strategy composed of at least some of the base models.
[0090] Figure 3This is a schematic diagram of the structure of a text evaluation device provided in an embodiment of this disclosure. Figure 3 As shown, the text evaluation device 300 includes: The information determination module 310 is used to determine the text to be evaluated, the target evaluation mode, and the target strategy parameters; wherein, the target evaluation mode is one of multiple candidate evaluation modes, and the candidate evaluation modes include single model mode, multi-model voting mode, or multi-model proxy mode; the target strategy parameters include the model identifier of at least one base model; The evaluation strategy generation module 320 is used to generate a target evaluation strategy based on the target strategy parameters and the target evaluation mode. The text evaluation module 330 is used to perform text evaluation on the text to be evaluated using the target evaluation strategy and generate target evaluation results.
[0091] The text evaluation apparatus provided in this disclosure can select a target evaluation mode suitable for the text to be evaluated from multiple candidate evaluation modes with text evaluation functions, and determine target policy parameters including model identifiers of at least one base model, so that the target policy parameters can be applied to the target evaluation mode to generate a target evaluation strategy; then, the target evaluation strategy is used to automatically evaluate the text to be evaluated to obtain the target evaluation result. Compared with the related technologies that have only one text evaluation model dedicated to a specific business scenario, the method of determining the target evaluation strategy through the combination of at least one base model and target policy parameters can not only reduce the cost and time consumption of model training, but also improve the diversity and flexibility of the target evaluation strategy, so that the target evaluation strategy can be more adapted to the text to be evaluated, thereby improving the scenario applicability of the target evaluation strategy and the accuracy of the text evaluation results.
[0092] In some embodiments, the information determination module 310 is specifically used for: Display the text evaluation interface, and display the text input component, mode selection component, and parameter configuration component in the text evaluation interface; In response to a trigger operation on the text input component, the received text is identified as the text to be evaluated; In response to a trigger operation on the mode selection component, display each candidate evaluation mode and a brief description of the candidate evaluation modes; In response to the selection of a candidate evaluation mode, the selected candidate evaluation mode is determined as the target evaluation mode, and the parameter configuration items corresponding to the target evaluation mode are displayed in the parameter configuration component. In response to a trigger operation on a parameter configuration item, the trigger result of the triggered parameter configuration item is determined as the target strategy parameter.
[0093] In some embodiments, if the target evaluation mode is a multi-model voting mode, the target strategy parameters may also include model weight values and / or voting rules. If the target evaluation mode is a multi-model agent mode, the target strategy parameters also include the threshold for the number of collaborations and / or the collaboration method.
[0094] In some embodiments, the text evaluation device 300 further includes: The text evaluation sample acquisition module is used to acquire the first number of text evaluation samples in the business scenario to which the text to be evaluated belongs before determining the text to be evaluated, the target evaluation mode, and the target strategy parameters. The model prompt word generation module is used to input each text evaluation sample into the first generative model, obtain the model output results, and generate model prompt words adapted to the business scenario based on the model output results; The basic model generation module is used to generate a basic model based on model prompt words and a second number of second generative models; wherein the number of basic models is determined based on the second number.
[0095] Furthermore, the model prompt word generation module is specifically used for: If the model output includes a background description, evaluation criteria, and evaluation examples that are adapted to the business scenario, then a third number of model prompt words will be generated based on the background description, evaluation criteria, evaluation examples, third number of evaluation role information, and evaluation task description. Based on the model prompt words and the second number of second generative models, a base model is generated, including: A base model is generated based on a third number of model prompts and a second number of second generative models; wherein the number of base models is determined based on the second and third numbers.
[0096] In some embodiments, the text evaluation device 300 further includes a data format conversion module, used for: After using the target evaluation strategy to evaluate the text and generate the target evaluation results, if the target evaluation results fail to be parsed according to the preset data format, the target evaluation results are input into the third generative model to convert the data format of the target evaluation results into the preset data format.
[0097] In some embodiments, the text evaluation device 300 further includes a retry triggering module, configured to: After using the target evaluation strategy to perform text evaluation on the text to be evaluated and generating the target evaluation result, if the target evaluation result fails to be parsed according to the preset data format, then if the model temperature corresponding to the target evaluation strategy does not reach the preset temperature or the number of retries does not reach the preset number, the model temperature is increased according to the preset temperature difference, and the step of using the target evaluation strategy to perform text evaluation on the text to be evaluated and generating the target evaluation result is retried based on the increased model temperature.
[0098] In some embodiments, the text evaluation device 300 further includes an evaluation termination module, configured to: After generating the target evaluation strategy based on the target strategy parameters and target evaluation mode, if the execution time of the text to be evaluated is detected to exceed the preset time, the evaluation processing of the text to be evaluated will be terminated.
[0099] The text evaluation apparatus provided in this disclosure can execute the text evaluation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the method execution.
[0100] It is worth noting that in the embodiments of the above-mentioned text evaluation device, the various modules are divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional module are only for easy differentiation and are not used to limit the scope of protection of this disclosure.
[0101] This disclosure also provides an electronic device including one or more processors and a memory; wherein the memory is used to store one or more programs or instructions. When the one or more programs or instructions are executed by the one or more processors, the one or more processors implement the text evaluation method provided in any embodiment of this disclosure.
[0102] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Figure 4 As shown, the electronic device 400 includes a processor 401, a memory 402, an input device 403, and an output device 404, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). The number of processors 401 and memory 402 can be one or more. Figure 4 The example below uses a processor 401 and a memory 402.
[0103] The processor 401 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.
[0104] Memory 402 may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. In some embodiments, memory 402 may further include memory remotely located relative to processor 401, which can be connected to electronic device 400 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. One or more computer programs or instructions may be stored in memory 402, which processor 401 can execute to implement the text evaluation methods described in any embodiment of this disclosure and / or other desired functions. Memory 402 may also store various content such as model hints, base models, policy parameters, etc.
[0105] Input device 403 may include, for example, a keyboard, a mouse, etc. Output device 404 may output various information to the outside, including target evaluation results, model prompts, etc. Output device 404 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0106] Understandably, for the sake of simplification, Figure 4 Only some of the components of the electronic device 400 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 400 may include any other suitable components depending on the specific application.
[0107] In addition to the methods and apparatus described above, the text evaluation methods in any embodiment of this disclosure can also be implemented as computer software programs. For example, embodiments of this disclosure also include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from memory. When the computer program is run by a processor, it causes the processor to execute the text evaluation methods provided in any embodiment of this disclosure.
[0108] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this disclosure. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0109] Furthermore, embodiments of this disclosure also provide a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, causes the processor to perform the text evaluation method provided in embodiments of this disclosure.
[0110] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
[0111] It should be noted that the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of this disclosure. As shown in this specification and claims, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" are not specifically singular and may include plural. Relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation and do not necessarily require or imply any such actual relationship or order between these entities or operations. The term "and / or" includes any and all combinations of one or more of the associated listed items. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element.
[0112] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A text evaluation method, characterized in that, include: The text to be evaluated, the target evaluation mode, and the target strategy parameters are determined; wherein, the target evaluation mode is one of multiple candidate evaluation modes, and the candidate evaluation modes include single-model mode, multi-model voting mode, or multi-model proxy mode; the target strategy parameters include the model identifier of at least one base model; Based on the target strategy parameters and the target evaluation mode, a target evaluation strategy is generated; The target evaluation strategy is used to perform text evaluation on the text to be evaluated, and target evaluation results are generated.
2. The method according to claim 1, characterized in that, The process of determining the text to be evaluated, the target evaluation mode, and the target strategy parameters includes: Display a text evaluation interface, and display a text input component, a mode selection component, and a parameter configuration component in the text evaluation interface; In response to a trigger operation on the text input component, the received text is identified as the text to be evaluated; In response to a trigger operation on the mode selection component, each candidate evaluation mode and its brief description are displayed. In response to the selection operation of the candidate evaluation mode, the selected candidate evaluation mode is determined as the target evaluation mode, and the parameter configuration items corresponding to the target evaluation mode are displayed in the parameter configuration component. In response to the triggering operation of the parameter configuration item, the triggering result of the triggered parameter configuration item is determined as the target strategy parameter.
3. The method according to claim 1 or 2, characterized in that, If the target evaluation mode is the multi-model voting mode, then the target strategy parameters also include model weight values and / or voting rules; If the target evaluation mode is the multi-model agent mode, then the target strategy parameters also include a collaboration number threshold and / or collaboration method.
4. The method according to claim 1, characterized in that, Before determining the text to be evaluated, the target evaluation mode, and the target strategy parameters, the method further includes: Obtain the first number of text evaluation samples in the business scenario to which the text to be evaluated belongs; Each of the text evaluation samples is input into the first generative model to obtain the model output results, and model prompt words adapted to the business scenario are generated based on the model output results. The base model is generated based on the model prompt words and the second number of second generative models; wherein the number of base models is determined based on the second number.
5. The method according to claim 4, characterized in that, The process of generating model suggestion words adapted to the business scenario based on the model output includes: If the model output includes a background description, evaluation criteria, and evaluation examples adapted to the business scenario, then based on the background description, the evaluation criteria, the evaluation examples, the third number of evaluation role information, and the evaluation task description, the third number of model prompt words are generated. The second generative model, based on the model prompt words and the second number of words, generates the base model, including: The base model is generated based on the third number of model prompts and the second number of the second generative models; wherein the number of base models is determined based on the second number and the third number.
6. The method according to claim 1, characterized in that, After performing text evaluation on the text to be evaluated using the target evaluation strategy and generating target evaluation results, the method further includes: If parsing the target evaluation result according to the preset data format fails, the target evaluation result is input into the third generative model to convert the data format of the target evaluation result into the preset data format.
7. The method according to claim 1 or 6, characterized in that, After performing text evaluation on the text to be evaluated using the target evaluation strategy and generating target evaluation results, the method further includes: If parsing the target evaluation result according to the preset data format fails, then if the model temperature corresponding to the target evaluation strategy does not reach the preset temperature, or the number of retries does not reach the preset number, the model temperature is increased according to the preset temperature difference, and the step of using the target evaluation strategy to perform text evaluation on the text to be evaluated and generating the target evaluation result is retried based on the increased model temperature.
8. The method according to claim 1, characterized in that, After generating the target evaluation strategy based on the target strategy parameters and the target evaluation mode, the method further includes: If the execution time of the text to be evaluated is detected to exceed the preset time, the evaluation process of the text to be evaluated will be terminated.
9. An electronic device, characterized in that, The electronic device includes: Processor and memory; The processor executes the text evaluation method as described in any one of claims 1 to 8 by calling the program or instructions stored in the memory.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that cause a computer to perform the text evaluation method as described in any one of claims 1 to 8.