Evaluation method and equipment for large model generation text, readable storage medium and program product

Abstract

The invention discloses an assessment method and device for a large model generated text, a readable storage medium and a program product, and relates to the technical field of artificial intelligence, and the assessment method for the large model generated text comprises the steps: obtaining a user-defined assessment template, and analyzing a dependency relationship and an execution priority of each preset assessment dimension in the user-defined assessment template, obtaining a directed acyclic graph used for representing the topological sequence of each preset evaluation dimension; according to a topological sequence of each preset evaluation dimension in the directed acyclic graph, evaluating the to-be-evaluated large model generation text on each preset evaluation dimension to obtain a score corresponding to each preset evaluation dimension; and determining a text evaluation result by counting the score corresponding to each preset evaluation dimension. According to the technical scheme, multi-dimensional evaluation can be carried out on the large model generation text, the multi-aspect capability of the model can be measured more comprehensively and accurately, and the scene adaptation capability of the text evaluation scheme is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a method, device, computer readable storage medium and computer program product for evaluating text generated by a large model. BACKGROUND

[0002] At present, the processing capability of a model for language is increasingly strong, and the quality evaluation of text output by the model is increasingly important. In a traditional evaluation test process for text generated by a large model, a single task index (for example, text generation accuracy) is usually relied on, and thus the evaluation result obtained cannot comprehensively and accurately measure the generalization capability, multi-task processing capability and ethical fairness of the model in a real scene. Moreover, the domain and scene adaptation capability of an artificially customized evaluation rule is poor, and it is difficult to adapt to the needs of various industries.

[0003] The above information disclosed in the background section of this document merely to provide background information for understanding the concepts of the present application, and therefore, it can contain information that does not constitute prior art. SUMMARY

[0004] The main purpose of the present application is to provide a method, device, computer readable storage medium and computer program product for evaluating text generated by a large model, which aims to build a custom evaluation template to evaluate text generated by a large model in multiple dimensions, more comprehensively and accurately measure the multi-aspect capability of the model, and improve the scene adaptation capability of the text evaluation scheme.

[0005] To achieve the above purpose, the present application provides a method for evaluating text generated by a large model, which comprises: obtaining a user-defined evaluation template, and obtaining a directed acyclic graph for representing the topological order of each preset evaluation dimension by analyzing the dependency relationship and execution priority of each preset evaluation dimension in the user-defined evaluation template; evaluating the large model text to be evaluated on each preset evaluation dimension according to the topological order of each preset evaluation dimension in the directed acyclic graph, and obtaining the score corresponding to each preset evaluation dimension, respectively; determining a text evaluation result by counting the scores corresponding to each preset evaluation dimension, respectively.

[0006] In an embodiment, the step of obtaining a directed acyclic graph for representing the topological order of each preset evaluation dimension by analyzing the dependency relationship and execution priority of each preset evaluation dimension in the user-defined evaluation template comprises: The dependency relationship and execution priority of each preset evaluation dimension in the user-defined evaluation template are parsed through a preset markup language analysis library to obtain at least one evaluation dimension corresponding to the user-defined evaluation template, a weight corresponding to the preset evaluation dimension, a natural language rule, and a topological order between each preset evaluation dimension. A directed acyclic graph corresponding to the user-defined evaluation template is generated according to each preset evaluation dimension corresponding to the user-defined evaluation template, the weight corresponding to each preset evaluation dimension, the natural language rule, and the topological order between each preset evaluation dimension.

[0007] In an embodiment, after the step of obtaining a directed acyclic graph for representing the topological order of each preset evaluation dimension by parsing the dependency relationship and execution priority of each preset evaluation dimension in the user-defined evaluation template, the method further comprises: loading model input data in parallel; establishing a connection with a local / cloud service through an application programming interface, and sending a request for multi-threaded parallel inference of the model input data to the local / cloud model service model; receiving large model generated text returned by the application programming interface.

[0008] In an embodiment, the step of evaluating the large model generated text to be evaluated according to the topological order of each preset evaluation dimension in the directed acyclic graph to obtain a score corresponding to each preset evaluation dimension comprises: converting the natural language rule in the user-defined evaluation template into a corresponding abstract syntax tree according to a preset rule engine; evaluating the large model generated text based on the abstract syntax tree corresponding to each preset evaluation dimension and the topological order represented by the directed acyclic graph to obtain a score corresponding to each preset evaluation dimension, wherein the preset evaluation dimensions include at least one of text quality, logical consistency, and sensitivity.

[0009] In an embodiment, the step of evaluating the large model generated text based on the abstract syntax tree corresponding to each preset evaluation dimension and the topological order represented by the directed acyclic graph to obtain a score corresponding to each preset evaluation dimension comprises: determining the topological order of each node with the node corresponding to a preset evaluation dimension without dependency in the directed acyclic graph as a starting node; packing the large model generated text and the rules and resource information corresponding to each preset evaluation dimension in the user-defined evaluation template into corresponding context objects, respectively; Based on the topological order of each node, each context object is evaluated in turn to obtain a score corresponding to each preset evaluation dimension.

[0010] In an embodiment, the text evaluation result at least includes a text evaluation total score and a score anomaly point. The step of determining the text evaluation result by counting the scores corresponding to each preset evaluation dimension includes: According to the scores corresponding to each preset evaluation dimension and the weights, a weighted calculation is performed to obtain a text evaluation total score. According to the preset Isolation Forest algorithm, the scores corresponding to each preset evaluation dimension are detected to obtain a score anomaly point, wherein the absolute value of the difference between the score corresponding to the preset evaluation dimension of the score anomaly point and the average score is greater than a preset difference threshold.

[0011] In an embodiment, after the step of detecting the scores corresponding to each preset evaluation dimension according to the preset Isolation Forest algorithm to obtain a score anomaly point, the method further includes: The scores corresponding to each preset evaluation dimension in the text evaluation result and the score anomaly point are pushed to a data monitoring platform for visual display on the data monitoring platform.

[0012] In addition, the present application also provides an evaluation device for large model generated text, which includes: A template analysis module is configured to obtain a user-defined evaluation template, and by analyzing the dependency relationship and execution priority of each preset evaluation dimension in the user-defined evaluation template, a directed acyclic graph representing the topological order of each preset evaluation dimension is obtained. A text scoring module is configured to evaluate a large model generated text to be evaluated according to the topological order of each preset evaluation dimension in the directed acyclic graph, and obtain a score corresponding to each preset evaluation dimension. A score summary module is configured to determine a text evaluation result by counting the scores corresponding to each preset evaluation dimension.

[0013] In addition, the present application also provides an evaluation device for large model generated text, which at least includes a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the evaluation method for large model generated text of the evaluation device for large model generated text as described above.

[0014] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the evaluation method for generating text from a large model as described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the evaluation method for generating text from a large model as described above.

[0016] This application provides an evaluation method for text generated by a large model. First, a user-defined evaluation template is obtained. Then, by parsing the dependencies and execution priorities of each preset evaluation dimension in the user-defined evaluation template, a directed acyclic graph (DAG) is obtained to represent the topological order of each preset evaluation dimension. Next, based on the topological order of each preset evaluation dimension in the DAG, the text generated by the large model to be evaluated is evaluated on each preset evaluation dimension, obtaining scores corresponding to each preset evaluation dimension. Finally, the text evaluation result is determined by statistically analyzing the scores corresponding to each preset evaluation dimension. In this application's technical solution, a pre-defined user-defined evaluation template is used to represent the rules for text evaluation. Furthermore, the DAG is used to determine the corresponding topological order, clearly representing the execution order and relative dependencies between each preset evaluation dimension. This allows for a multi-dimensional and comprehensive evaluation of the text generated by the large model, obtaining scores on multiple evaluation dimensions. Finally, the evaluation result is obtained by combining the scores from multiple evaluation dimensions. Compared to traditional text evaluation strategies based on a single metric, this application supports cross-task metric linkage analysis, which can more comprehensively and accurately measure the model's multi-faceted capabilities. Moreover, the predefined user-defined evaluation templates can be customized for different scenarios, thereby improving the scenario adaptability of the text evaluation scheme. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the evaluation method for generating text from a large model in an embodiment of this application. Figure 2This is a schematic diagram illustrating the process of calling the model service for inference and obtaining the large model-generated text before performing text evaluation in an embodiment of this application. Figure 3 This is a schematic diagram illustrating the entire process of evaluating text generated from a large model in an embodiment of this application; Figure 4 This is a schematic diagram of the structural composition of the evaluation device for generating text from a large model in an embodiment of this application; Figure 5 This is a schematic diagram of the equipment structure of the air conditioner's hardware operating environment involved in the evaluation method for generating text from a large model in this embodiment of the application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] Traditional benchmarking of models often relies excessively on single-task metrics (such as text generation accuracy) and fixed datasets, making it difficult to comprehensively measure the model's generalization ability, multi-task processing capabilities, and ethical fairness in real-world scenarios. Furthermore, manually customized evaluation rules are inefficient and cannot dynamically adapt to the needs of different industries.

[0025] The technical solution of this application combines a dynamic template-driven evaluation framework with a multi-dimensional rule engine. The system can define multiple preset evaluation dimensions (such as text quality, logical consistency, etc.), weight allocation, and inference steps through pre-configured user-defined evaluation templates, supporting cross-task indicator linkage analysis. It can also introduce an automated rule engine to transform evaluation rules described in natural language (such as "bias detection statements") into executable code logic, combined with the Isolation Forest algorithm to detect data distribution anomalies. Furthermore, the system can also include a pre-built fairness evaluation module, which can quantify the model's bias on sensitive evaluation dimensions such as gender and culture.

[0026] To overcome the aforementioned technical deficiencies and achieve the aforementioned friendship effect, this application provides an evaluation method for large-model generated text, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the evaluation method for generating text from a large model according to this application. The evaluation method for generating text from a large model includes: Step S10: Obtain the user-defined evaluation template. By parsing the dependencies and execution priorities of each preset evaluation dimension in the user-defined evaluation template, a directed acyclic graph is obtained to represent the topological order of each preset evaluation dimension. Among them, the user-defined evaluation template is an evaluation template that users can pre-configure or set through the system. The user-defined evaluation template can be in the form of a YAML (markup language) file, used to define multiple preset evaluation dimensions (such as text quality, logical consistency, etc.), weight allocation, and the order between reasoning steps, and can also be used for comprehensive analysis across evaluation dimensions.

[0027] A Directed Acyclic Graph (DAG) is a graph containing multiple nodes with topological connections between them. It is used to define the computational dependencies and execution priorities of each preset evaluation dimension, and can also represent the topological order of each preset evaluation dimension.

[0028] Step S20: Based on the topological order of each preset evaluation dimension in the directed acyclic graph, evaluate the large model to be evaluated on each preset evaluation dimension to obtain the scores corresponding to each preset evaluation dimension. Once the DAG graph is determined, the order and dependencies of each node (corresponding to evaluation dimensions) can be clearly defined. This allows for multi-dimensional evaluation of the text generated by the large model, yielding a score for each evaluation dimension. Dependency refers to the necessary relationship between different preset evaluation dimensions; for example, a score for one preset evaluation dimension must be based on a score for another. For instance, the evaluation of logical consistency requires a score for text quality. If the first evaluation dimension depends on the output of the second evaluation dimension, then the node of the first evaluation dimension is located at the next level below the node of the second evaluation dimension.

[0029] It should be noted that the large model-generated text is text output by the model after pre-processing inference. This model can be a model from various application scenarios, such as a large language model. In response to a user's input prompt, the large language model can output the corresponding large model-generated text (i.e., the response text to the prompt). To evaluate the performance of this large language model and the instructions in its output text, multi-dimensional evaluation can be performed, and corresponding scores can be obtained.

[0030] Step S30: Determine the text evaluation result by statistically analyzing the scores corresponding to each preset evaluation dimension.

[0031] Finally, the scores of each preset evaluation dimension are statistically analyzed. The weights and evaluation criteria of each preset evaluation dimension can also be combined to comprehensively evaluate the text generated by the large model and determine whether it meets the preset requirements, thus providing a data foundation for the subsequent training and optimization of the model.

[0032] The text evaluation results can also be in the form of an evaluation report, which comprehensively displays the scores corresponding to each of the multiple evaluation dimensions, as well as the overall score and / or the overall evaluation conclusion (qualified or unqualified), and marks the abnormal or unqualified scores for manual review.

[0033] Furthermore, in a feasible embodiment, the step of obtaining a directed acyclic graph representing the topological order of each preset evaluation dimension by parsing the dependencies and execution priorities of each preset evaluation dimension in the user-defined evaluation template may include: Step S11: The dependencies and execution priorities of each preset evaluation dimension in the user-defined evaluation template are parsed using a preset markup language parsing library to obtain at least one evaluation dimension corresponding to the user-defined evaluation template, the weight of the evaluation dimension, the natural language rules, and the topological order between each preset evaluation dimension. Step S12: Generate a directed acyclic graph corresponding to the user-defined evaluation template based on the preset evaluation dimensions, the weights of the evaluation dimensions, the natural language rules, and the topological order among the preset evaluation dimensions.

[0034] During the template parsing phase, the evaluation template in YAML / JSON (a lightweight data exchange format) format can be read first. Then, PyYAML (a third-party library for processing YAML data) in Python (a computer programming language) is used to parse the preset evaluation dimensions, the weights corresponding to the evaluation dimensions, and the relevant rules in natural language form contained in the user-defined evaluation template, and generate a DAG task dependency graph.

[0035] It should be noted that user-defined evaluation templates include at least the following three core configuration dimensions: these define the evaluation dimensions, such as text_quality and fairness, and the corresponding weights for each dimension; rules describe the natural language rules (e.g., "if the output contains politically sensitive words, then fairness will be deducted by 20%); and dependencies declare the calculation order between evaluation dimensions. For example, logic_consistency depends on the text_quality output, so the text quality score must be calculated first, and then the logic consistency score must be calculated.

[0036] The Directed Acyclic Graph (DAG) serves as the central scheduling blueprint and execution engine for all subsequent evaluation tasks across all evaluation dimensions. The entire evaluation process strictly follows the topological order defined by this DAG, which includes multiple nodes, each corresponding to an evaluation dimension and storing its weights. The connections between nodes reflect the corresponding topological order. A DAG typically has at least one starting node. The preset evaluation dimension corresponding to the starting node does not depend on the scores of any other evaluation dimensions and can directly calculate its score.

[0037] In one feasible embodiment, such as Figure 2 As shown, after the step of obtaining a directed acyclic graph representing the topological order of each preset evaluation dimension by parsing the dependencies and execution priorities of each preset evaluation dimension in the user-defined evaluation template, the method may further include: Step A10: Load model input data in parallel; Step A20: Establish a connection with the local / cloud service through the application programming interface and send a request to the local / cloud model service model to perform multi-threaded parallel inference on the model input data; Step A30: Receive the large model generated text returned by the application programming interface.

[0038] This application also provides an automated evaluation strategy, which automatically connects to model services (such as OpenAI or locally deployed LLM) via API (Application Programming Interface) to perform multi-threaded inference on the model input data.

[0039] Specifically, Datatrove (an open-source data processing tool) can be used to load the test dataset (equivalent to the model input data) in parallel. For large language models, the model input data consists of user-inputted suggestions. The test dataset can be configured as needed, using publicly available standard benchmark datasets or industry-specific test sets. Furthermore, when connecting to the local or cloud model via API, multi-dimensional inference requests are sent to the model input data, and a thread pool is used to control concurrency and avoid service overload. After the model inference service is complete, the returned model input text can be received via the API interface for subsequent text evaluation.

[0040] In one feasible embodiment, the step of generating text for the large model to be evaluated based on the topological order of each preset evaluation dimension in the directed acyclic graph, and obtaining the scores corresponding to each preset evaluation dimension, may include: Step S21: Convert the natural language rules in the user-defined evaluation template into the corresponding abstract syntax tree according to the preset rule engine; Step S22: Based on the topological order of the pre-defined evaluation dimensions represented by the abstract syntax tree and the directed acyclic graph corresponding to each pre-defined evaluation dimension, the large model generates text and obtains the scores corresponding to each pre-defined evaluation dimension. The evaluation dimensions include at least one of text quality, logical consistency and sensitivity.

[0041] In order for the execution engine to recognize and understand the natural language rules in the user-defined evaluation template in order to perform the evaluation task, this embodiment of the application needs to first convert the natural language rules into AST (Abstract Syntax Tree) through the RuleEngine library. For example, "deduct points if the output contains sensitive words" is a natural language rule, which is mapped to an abstract syntax related to regular expression matching and weight calculation.

[0042] For example, the natural language rule "if a sensitive word appears ≥ 3 times, the ethics score is reset to zero" can be broken down into three steps: regular expression matching of the sensitive word list, counting statistics, and conditional score reset. Specifically, rule-engine.Context (the custom rule engine) dynamically binds variables through resolver (e.g., output_text binds the model output, and banned_words loads the sensitive word library). When apply() is executed, it iterates through the rule set. If matches() detects that the condition is met (e.g., the number of times a sensitive word appears exceeds the limit), it triggers the corresponding action (e.g., score reset), completing the scoring process for each evaluation dimension. Here, the ethics score is the score corresponding to the evaluation dimension "sensitivity".

[0043] Furthermore, in a feasible embodiment, the step of evaluating the text generated by the large model based on the topological order of the abstract syntax tree and the directed acyclic graph representation corresponding to each preset evaluation dimension, and obtaining the scores corresponding to each preset evaluation dimension, includes: Step S221: Using the nodes corresponding to the undependent preset evaluation dimensions in the directed acyclic graph as the starting nodes, determine the topological order of each node, where each node corresponds to an evaluation dimension. Step S222: Package the generated text of the large model and the rules and resource information corresponding to each preset evaluation dimension in the user-defined evaluation template into corresponding context objects; Step S223: Based on the topological order of each node, evaluate each context object in turn to obtain the scores corresponding to each preset evaluation dimension.

[0044] During the evaluation phase of the large model-generated text, each record in the large model-generated text is traversed through a defined Directed Acyclic Graph (DAG). For example, a record can be a set of prompt-response pairs. For each record, a complete DAG traversal evaluation is initiated. The execution engine starts from the starting node of the DAG (i.e., the preset evaluation dimension without any dependencies). For each node, the large model-generated text (output_text) of the current record, along with the rules and resources (such as the sensitive word library path) defined in the user-defined evaluation template for that evaluation dimension, are packaged into a context object (rule-engine.Context). The rule engine is then invoked to evaluate all rules for that evaluation dimension, ultimately obtaining the corresponding score.

[0045] It should be noted that after determining the scores for the evaluation dimensions of the large model-generated text, these scores are integrated into a specific, structured dataset for the current data record, containing quantified scores for each dimension. This dataset could be a Python dictionary, which forms the data foundation for step S30. Specifically, a Python dictionary can be generated for each piece of text generated by the large model, and then passed sequentially to Pandas (a popular software library for data manipulation and analysis in the Python programming language) for subsequent statistical and analytical processes.

[0046] In one feasible embodiment, the text evaluation result includes at least a total text evaluation score and outliers; the step of determining the text evaluation result by statistically analyzing the scores corresponding to each preset evaluation dimension includes: Step S31: Calculate the total text evaluation score by weighting the scores and weights corresponding to each preset evaluation dimension. Step S32: Detect the scores corresponding to each preset evaluation dimension according to the preset isolated forest algorithm to obtain scoring outliers. Among them, the absolute value of the difference between the score of the preset evaluation dimension corresponding to the scoring outlier and the average score is greater than the preset difference threshold.

[0047] In this embodiment of the application, Pandass can be used to calculate the scores of each preset evaluation dimension, and the total text evaluation score can be calculated according to the weights corresponding to each preset evaluation dimension contained in the user-defined evaluation template.

[0048] For example, pd.DataFrame (a core data structure for processing structured data) can be used to integrate the scoring data from various dimensions, and final_score can be weighted and summed according to template weights. The IsolationForest algorithm detects outliers in the fairness (final score) score, and these outliers can be marked in the text evaluation report for easy manual review.

[0049] It should be noted that outliers are data records whose scores on a certain preset evaluation dimension (fairness) are significantly different from those of the vast majority of samples in the dataset; i.e., outliers. In this embodiment, the isolated forest algorithm can be used to achieve this. The scores of outliers are usually much higher (or much lower) than the average standard (i.e., the average score of a certain evaluation dimension). If the scores of outliers are quantified, they are calculated by adding a preset difference threshold to the scores above the average standard, or subtracting the preset difference threshold from the scores below the average standard.

[0050] In the steps above, the list of text evaluation results containing all the large model-generated text is directly passed to Pandas, and efficiently converted into a regular data frame using the pd.DataFrame.from_records() method (a function used to convert structured data into a DataFrame, a two-dimensional labeled data structure).

[0051] In one feasible embodiment, after the step of detecting outliers in the scores corresponding to each preset evaluation dimension according to the preset isolated forest algorithm, the method may further include: Step S40: Push the scores and scoring anomalies corresponding to each preset evaluation dimension in the text evaluation results to the data monitoring platform for visualization.

[0052] To enhance the visualization of text evaluation results, this embodiment uses Prometheus to achieve real-time monitoring and visualization of the evaluation results of text generated by large models. As described in the previous embodiment, after Pandass calculates the scores for each preset evaluation dimension and analyzes the text evaluation results, it is pushed in real-time to the Prometheus Grafana dashboard (an interactive interface for visualizing and monitoring data, supporting real-time display of system operating status through components such as charts and text panels), showing the current evaluation results of the model's output text to the staff.

[0053] If the real-time evaluation result of the current output text of the pattern (such as the total text evaluation score) is significantly lower than the preset standard, an alarm message will be issued to prompt staff to handle the situation promptly.

[0054] For ease of understanding, Figure 3 This paper demonstrates a complete process for evaluating text generated by a large model in an embodiment of this application. Specifically, it includes: first, obtaining a defined evaluation template; then, performing template parsing and DAG construction on the defined evaluation template; then, performing multi-threaded model evaluation to obtain text generated by the large model; then, using a rule engine to execute the evaluation of the text generated by the large model to obtain scores for each preset evaluation dimension; then, performing aggregate analysis on the output results; and then, performing real-time monitoring and alarms when anomalies or significant anomalies appear in the scores.

[0055] It should be noted that this example is only for the purpose of assisting in understanding this application and does not constitute a limitation on the evaluation method of the large model generating text in this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0056] This application also provides an evaluation device for large model-generated text, such as... Figure 4 As shown, the evaluation device for the large model-generated text includes: Template parsing module 10 is used to obtain a user-defined evaluation template. By parsing the dependencies and execution priorities of each preset evaluation dimension in the user-defined evaluation template, a directed acyclic graph is obtained to represent the topological order of each preset evaluation dimension. The text scoring module 20 is used to evaluate the text generated by the large model to be evaluated on each of the preset evaluation dimensions according to the topological order of each preset evaluation dimension in the directed acyclic graph, and to obtain the scores corresponding to each preset evaluation dimension respectively. The scoring summary module 30 is used to determine the text evaluation result by statistically analyzing the scores corresponding to each of the preset evaluation dimensions.

[0057] In one embodiment, the template parsing module 10 is further configured to: The dependencies and execution priorities of each preset evaluation dimension in the user-defined evaluation template are parsed using a preset markup language parsing library to obtain at least one evaluation dimension corresponding to the user-defined evaluation template, the weights corresponding to the preset evaluation dimensions, natural language rules, and the topological order among the preset evaluation dimensions. Based on the preset evaluation dimensions corresponding to the user-defined evaluation template, the weights corresponding to each preset evaluation dimension, the natural language rules, and the topological order among the preset evaluation dimensions, the directed acyclic graph corresponding to the user-defined evaluation template is generated.

[0058] In one embodiment, the evaluation apparatus for large model-generated text further includes a model inference module, which is used for: Parallel loading of model input data; A connection is established with the local / cloud service through the application programming interface, and a request is sent to the local / cloud model service model to perform multi-threaded parallel inference on the model input data; Receive the large model generated text returned by the application programming interface.

[0059] In one embodiment, the text scoring module 20 is further configured to: The natural language rules in the user-defined evaluation template are converted into corresponding abstract syntax trees according to the preset rule engine; Based on the abstract syntax tree corresponding to each of the preset evaluation dimensions and the topological order of each preset evaluation dimension represented by the directed acyclic graph, the text generated by the large model is evaluated to obtain the score corresponding to each preset evaluation dimension. The preset evaluation dimensions include at least one of text quality, logical consistency and sensitivity.

[0060] In one embodiment, the text scoring module 20 is further configured to: Starting with the nodes corresponding to the undependent preset evaluation dimensions in the directed acyclic graph, the topological order of each node is determined, wherein each node corresponds to an evaluation dimension. The generated text of the large model and the rules and resource information corresponding to each preset evaluation dimension in the user-defined evaluation template are packaged into corresponding context objects; Based on the topological order of each node, each context object is evaluated sequentially to obtain a score corresponding to each preset evaluation dimension.

[0061] In one embodiment, the text evaluation result includes at least a total text evaluation score and outliers. The rating summary module 30 is also used for: The total text evaluation score is obtained by weighting the scores and weights of each preset evaluation dimension. The scores corresponding to each preset evaluation dimension are detected by the preset isolated forest algorithm to obtain scoring outliers. The absolute value of the difference between the score of the preset evaluation dimension corresponding to the scoring outlier and the average score is greater than the preset difference threshold.

[0062] In one embodiment, the scoring summary module 30 is further configured to: The scores and outliers corresponding to each preset evaluation dimension in the text evaluation results are pushed to the data monitoring platform for visualization.

[0063] The large model-generated text evaluation device provided in this application adopts the large model-generated text evaluation method in the above embodiments, which can perform multi-dimensional evaluation of large model-generated text, more comprehensively and accurately measure the multi-faceted capabilities of the model, and improve the scenario adaptability of the text evaluation scheme. Compared with the prior art, the beneficial effects of the large model-generated text evaluation device provided in this application are the same as the beneficial effects of the large model-generated text evaluation method provided in the above embodiments, and other technical features in the large model-generated text evaluation device are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0064] This application also provides an evaluation device for large model-generated text, the large model-generated text evaluation device comprising at least: at least one processor; and a memory communicatively connected to at least one processor; wherein the memory stores instructions executable by at least one processor, the instructions being executed by at least one processor to enable at least one processor to perform the large model-generated text evaluation method described above.

[0065] The following is for reference. Figure 5 It shows a schematic diagram of the structure of an evaluation device suitable for implementing large model text generation in the embodiments of this application. Figure 5 The large model-generated text evaluation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0066] like Figure 5As shown, the evaluation device for generating text from large models may include a processing unit 101 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 102 or a program loaded from storage device 103 into random access memory (RAM) 104. RAM 104 also stores various programs and data required for the operation of the evaluation device for generating text from large models. The processing unit 101, ROM 102, and RAM 104 are interconnected via bus 105. Input / output (I / O) interface 106 is also connected to the bus. Typically, the following systems can be connected to I / O interface 106: input devices 107 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 108 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 103 including, for example, magnetic tapes, hard disks, etc.; and communication devices 109. Communication device 109 allows the evaluation device for generating text from large models to communicate wirelessly or wiredly with other devices to exchange data. While the figures show evaluation devices for generating text from large models with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0067] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 103, or installed from ROM 102. When the computer program is executed by processing device 101, it performs the functions defined in the methods of the embodiments of this application.

[0068] The evaluation device for large model-generated text provided in this application adopts the evaluation method for large model-generated text in the above embodiments, enabling multi-dimensional evaluation of large model-generated text, more comprehensively and accurately measuring the model's multi-faceted capabilities, and improving the scenario adaptability of the text evaluation scheme. Compared with the prior art, the beneficial effects of the evaluation device for large model-generated text provided in this application are the same as those of the evaluation method for large model-generated text provided in the above embodiments, and other technical features in this evaluation device for large model-generated text are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0069] It should be understood that various parts of the embodiments of this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0070] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the above claims.

[0071] This application also provides a computer-readable storage medium storing a computer program that can run on a processor. The computer program is used to execute the evaluation method for generating text from a large model in the above embodiments.

[0072] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0073] The aforementioned computer-readable storage medium may be included in the evaluation device for large model-generated text; or it may exist independently and not be assembled into the evaluation device for large model-generated text.

[0074] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by an evaluation device for large model-generated text, the evaluation device for large model-generated text performs the following: It acquires a user-defined evaluation template; by parsing the dependencies and execution priorities of each preset evaluation dimension in the user-defined evaluation template, it obtains a directed acyclic graph (DAG) representing the topological order of each preset evaluation dimension; based on the topological order of each preset evaluation dimension in the DAG, it evaluates the large model-generated text to be evaluated on each preset evaluation dimension, obtaining scores corresponding to each preset evaluation dimension; and by statistically analyzing the scores corresponding to each preset evaluation dimension, it determines the text evaluation result.

[0075] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0076] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0077] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0078] The computer-readable storage medium provided in this application embodiment stores computer-readable program instructions for executing the above-described evaluation method for large model-generated text. This enables multi-dimensional evaluation of large model-generated text, providing a more comprehensive and accurate measure of the model's various capabilities and improving the scenario adaptability of the text evaluation scheme. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application embodiment are the same as those of the large model-generated text evaluation method provided in the above embodiments, and will not be repeated here.

[0079] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the evaluation method for generating text from a large model as described above.

[0080] The computer program product provided in this application can perform multi-dimensional evaluation of text generated by large models, more comprehensively and accurately measure the model's various capabilities, and improve the scenario adaptability of the text evaluation scheme. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the evaluation method for text generated by large models provided in the above embodiments, and will not be repeated here.

[0081] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. An evaluation method for text generated by a large model, characterized in that, The evaluation methods for the large model-generated text include: Obtain a user-defined evaluation template, and by parsing the dependencies and execution priorities of each preset evaluation dimension in the user-defined evaluation template, obtain a directed acyclic graph that represents the topological order of each preset evaluation dimension; Based on the topological order of each preset evaluation dimension in the directed acyclic graph, the large model to be evaluated is generated and evaluated on each preset evaluation dimension to obtain the scores corresponding to each preset evaluation dimension. The text evaluation result is determined by statistically analyzing the scores corresponding to each of the preset evaluation dimensions.

2. The evaluation method for large model-generated text as described in claim 1, characterized in that, The step of obtaining a directed acyclic graph representing the topological order of each preset evaluation dimension by parsing the dependencies and execution priorities of each preset evaluation dimension in the user-defined evaluation template includes: The dependencies and execution priorities of each preset evaluation dimension in the user-defined evaluation template are parsed using a preset markup language parsing library to obtain at least one evaluation dimension corresponding to the user-defined evaluation template, the weights corresponding to the preset evaluation dimensions, natural language rules, and the topological order among the preset evaluation dimensions. Based on the preset evaluation dimensions corresponding to the user-defined evaluation template, the weights corresponding to each preset evaluation dimension, the natural language rules, and the topological order among the preset evaluation dimensions, the directed acyclic graph corresponding to the user-defined evaluation template is generated.

3. The evaluation method for large model-generated text as described in claim 1, characterized in that, After the step of obtaining a directed acyclic graph representing the topological order of each preset evaluation dimension by parsing the dependencies and execution priorities of each preset evaluation dimension in the user-defined evaluation template, the method further includes: Parallel loading of model input data; A connection is established with the local / cloud service through the application programming interface, and a request is sent to the local / cloud model service model to perform multi-threaded parallel inference on the model input data; Receive the large model generated text returned by the application programming interface.

4. The evaluation method for large model-generated text as described in claim 1, characterized in that, The step of generating text for the large model to be evaluated based on the topological order of each preset evaluation dimension in the directed acyclic graph, and obtaining the scores corresponding to each preset evaluation dimension, includes: The natural language rules in the user-defined evaluation template are converted into corresponding abstract syntax trees according to the preset rule engine; Based on the abstract syntax tree corresponding to each of the preset evaluation dimensions and the topological order of each preset evaluation dimension represented by the directed acyclic graph, the text generated by the large model is evaluated to obtain the score corresponding to each preset evaluation dimension. The preset evaluation dimensions include at least one of text quality, logical consistency and sensitivity.

5. The evaluation method for large model-generated text as described in claim 4, characterized in that, The step of evaluating the text generated by the large model based on the abstract syntax tree corresponding to each of the preset evaluation dimensions and the topological order of each preset evaluation dimension represented by the directed acyclic graph, and obtaining the score corresponding to each preset evaluation dimension, includes: Starting with the nodes corresponding to the undependent preset evaluation dimensions in the directed acyclic graph, the topological order of each node is determined, wherein each node corresponds to an evaluation dimension. The generated text of the large model and the rules and resource information corresponding to each preset evaluation dimension in the user-defined evaluation template are packaged into corresponding context objects; Based on the topological order of each node, each context object is evaluated sequentially to obtain a score corresponding to each preset evaluation dimension.

6. The evaluation method for large model-generated text as described in claim 1, characterized in that, The text evaluation results shall include at least the total text evaluation score and outliers. The step of determining the text evaluation result by statistically analyzing the scores corresponding to each of the preset evaluation dimensions includes: The total text evaluation score is obtained by weighting the scores and weights corresponding to each of the preset evaluation dimensions. The scores corresponding to each preset evaluation dimension are detected according to the preset isolated forest algorithm to obtain scoring outliers. The absolute value of the difference between the score of the preset evaluation dimension corresponding to the scoring outlier and the average score is greater than the preset difference threshold.

7. The evaluation method for large model-generated text as described in claim 6, characterized in that, After the step of detecting outliers in the scores corresponding to each preset evaluation dimension according to the preset isolated forest algorithm, the method further includes: The scores and outliers corresponding to each preset evaluation dimension in the text evaluation results are pushed to the data monitoring platform for visualization.

8. An evaluation device for large model-generated text, characterized in that, The evaluation device for large model-generated text includes at least: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the evaluation method for large model-generated text as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program implementing an evaluation method for generating text from a large model, the program being executed by a processor to implement the steps of the evaluation method for generating text from a large model as claimed in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the evaluation method for generating text from a large model as described in any one of claims 1 to 7.

Images