Benchmark test driven prompt word tuning method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a benchmark test-driven prompt word tuning method, device, equipment and medium, and the method comprises the following steps: analyzing input data to generate a benchmark test set; generating a cue word parameter configuration set based on the benchmark test set; calling a language model interface to process the input text to obtain a model output set; comparing the model output set with expected output to generate a comprehensive performance index set; updating a cue word parameter configuration set according to the comprehensive performance index set; and judging whether a termination condition is met or not, if not, continuing to execute model calling and parameter updating, and if yes, outputting a target cue word configuration and optimization report. According to the method, a structured evaluation and parameter iteration mechanism is introduced, cue word optimization is converted into a systematic process from artificial experience driving, repeated trial and error are reduced, cue word tuning efficiency is improved, and stability and mobility of the model in different tasks are achieved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a benchmark-driven method, apparatus, device, and medium for optimizing prompt words. Background Technology

[0002] In recent years, large language models, such as GPT, Qwen, and Llama, have been widely used in various scenarios, particularly in the healthcare and fintech sectors. In healthcare, large language models are applied to tasks such as medical record text structuring, clinical guideline parsing, and patient dialogue summarization. In fintech, they are used for risk compliance auditing, customer intent identification, and business data inductive analysis. However, existing technologies have several shortcomings in prompt design and application, hindering their further promotion and stable implementation in these two fields.

[0003] In the healthcare field, current prompt design relies heavily on human experience, with different researchers or developers often using their own wording to drive model-generated results. This highly experience-dependent approach lacks a unified structured representation and reusable specifications, leading to inconsistent output quality for the same diagnostic task across different teams, resulting in insufficient diagnostic consistency and interpretability. Furthermore, manually adjusting prompts is time-consuming and costly, requiring extensive debugging for each new task, hindering rapid adaptation to new medical applications. More critically, current evaluation methods lack unified standards, relying heavily on manual comparisons or single metrics to judge model generation quality. This makes it difficult to accurately evaluate generated content across multiple dimensions in medical scenarios, considering both the accuracy of clinical terminology and the clarity and readability of the wording.

[0004] In the fintech business, prompt design also faces the problem of over-reliance on experience. Different project teams tend to use different expression styles or sample numbers when building prompts, resulting in significant performance differences when the model handles the same compliance review or text classification tasks. This difference not only affects the stability of the task but also increases the risks of regulatory review and customer interaction. In addition, a large amount of repetitive manual tuning work leads to low development efficiency, and the tuning results are difficult to migrate and reuse across different business lines. More importantly, existing evaluation mechanisms often lack a multi-dimensional indicator system that meets the multi-task requirements of finance, and cannot simultaneously consider multiple aspects such as accuracy, recall, semantic rationality, and execution cost, making it difficult to effectively identify and optimize bottlenecks. Summary of the Invention

[0005] The main objective of this invention is to provide a benchmark-driven prompt optimization method, apparatus, device, and storage medium, aiming to solve the technical problems of existing prompt design and optimization relying too much on human experience, lacking a unified structured representation and multi-dimensional automated evaluation mechanism, resulting in difficulties in cross-task migration, large performance fluctuations, and poor reusability.

[0006] To achieve the above objectives, the present invention provides a benchmark-driven prompt word optimization method, comprising: Parse the input data, extract the input text and the corresponding expected output, and generate a benchmark test set; Based on the benchmark test set, a set of prompt word parameter configurations is generated according to multiple parameter dimensions; Based on the prompt word parameter configuration set and the benchmark test set, the language model interface is called to process the input text and obtain the model output set. The model output set is compared with the expected output in the benchmark test set to generate a comprehensive performance index set. The prompt word parameter configuration set is adjusted and updated based on the comprehensive performance index set. After the set of prompt word parameter configurations is updated, determine whether the termination condition is met; If the termination condition is not met, the generated prompt word parameter configuration set is returned, and the model call, output comparison, and parameter update operations are executed in sequence; if the termination condition is met, the target prompt word configuration and optimization report are output.

[0007] Furthermore, to achieve the above objectives, the present invention provides a benchmark-driven prompt word tuning device, comprising: The data parsing module is used to parse input data, extract input text and corresponding expected output, and generate a benchmark test set. The parameter configuration generation module is used to generate a set of prompt word parameter configurations based on the benchmark test set and according to multiple parameter dimensions. The model invocation module is used to call the language model interface to process the input text based on the prompt word parameter configuration set and the benchmark test set, and obtain the model output set. The performance comparison module is used to compare the model output set with the expected output in the benchmark test set to generate a comprehensive performance index set. The parameter update module is used to adjust and update the prompt word parameter configuration set according to the comprehensive performance index set; The termination judgment module is used to determine whether the termination condition is met after the prompt word parameter configuration set is updated; The results output module is used to return the set of generated prompt word parameter configurations if the termination condition is not met, and to perform model invocation, output comparison and parameter update operations in sequence; if the termination condition is met, it outputs the target prompt word configuration and optimization report.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a benchmark-driven prompt word tuning program stored in the memory and executable on the processor, wherein when the benchmark-driven prompt word tuning program is executed by the processor, it implements the steps of the benchmark-driven prompt word tuning method as described above.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a benchmark-driven prompt word tuning program, wherein when the benchmark-driven prompt word tuning program is executed by a processor, it implements the steps of the benchmark-driven prompt word tuning method described above.

[0010] Beneficial Effects: This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a benchmark-driven prompt word optimization method, apparatus, device, and medium, comprising: parsing input data, extracting input text and expected output, and generating a benchmark set; generating a prompt word parameter configuration set based on the benchmark set according to multiple parameter dimensions; combining the prompt word parameter configuration set and the benchmark set to call a language model interface to process the input text, obtaining a model output set; comparing the model output set with the expected output of the benchmark set to generate a comprehensive performance index set; adjusting and updating the prompt word parameter configuration set according to the comprehensive performance index set; after updating the prompt word parameter configuration set, determining whether a termination condition is met; if not, returning to generate the prompt word parameter configuration set and sequentially executing model calls, output comparisons, and parameter updates; if the termination condition is met, outputting the target prompt word configuration and optimization report. This invention, by combining the benchmark set and the prompt word parameter configuration set and introducing an automated comparison and update mechanism during the iteration process, achieves systematic optimization of prompt word parameters, thereby reducing manual trial and error and reliance on experience, and improving the stability and transferability of the model in multi-task scenarios. Attached Figure Description

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a benchmark-driven prompt word optimization method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the benchmark-driven prompt word optimization method of the present invention; Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the benchmark-driven prompt word optimization device of the present invention. Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] The benchmark-driven prompt word optimization method provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can parse the input data from the client, extract the input text and expected output, and generate a benchmark test set. Based on the benchmark test set, it generates a set of prompt word parameter configurations according to multiple parameter dimensions. Combining the prompt word parameter configuration set and the benchmark test set, it calls the language model interface to process the input text, obtaining a model output set. It compares the model output set with the expected output of the benchmark test set to generate a comprehensive performance index set. It adjusts and updates the prompt word parameter configuration set according to the comprehensive performance index set. After updating the prompt word parameter configuration set, it checks whether the termination condition is met. If not, it returns to generate the prompt word parameter configuration set and sequentially executes the model call, output comparison, and parameter update operations. If the termination condition is met, it outputs the target prompt word configuration and optimization report. This invention achieves systematic optimization of prompt word parameters by combining the benchmark test set and the prompt word parameter configuration set and introducing an automated comparison and update mechanism during the iteration process. This reduces manual trial and error and reliance on experience, improving the stability and transferability of the model in multi-task scenarios. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0014] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the benchmark-driven prompt word tuning method provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0015] like Figure 2 As shown, the benchmark-driven prompt word optimization method proposed in this invention includes the following steps: S10: Parse the input data, extract the input text and the corresponding expected output, and generate a benchmark test set; In this embodiment, parsing the input data requires processing the source, format, and storage method of the input file layer by layer. The input data can originate from structured data files, such as CSV tables, JSON documents, or records exported from a database. These files typically contain several fields, including at least input text fields and expected output fields. The key to the parsing process is correctly identifying the file encoding, delimiters, and field mapping relationships to avoid data loss or mismatches due to inconsistent formats. During parsing, the file reading module scans the file header information, extracts field names and type information, and establishes a mapping index between the input text fields and the expected output fields, ensuring that subsequent processing can access the correct data content according to a fixed logic.

[0016] When extracting input text, the original text string needs to be read completely, and redundant symbols or abnormal characters need to be removed to ensure consistency of the text input. For example, data files from different sources may contain extra tabs, escape characters, or special encoded characters, which need to be cleaned up during the preprocessing stage. Methods for extracting input text include directly accessing the content of text fields and storing them as string arrays, or constructing a set of input text sequences. The purpose of this is to unify the input interface so that subsequent calls can retrieve the text content to be processed in the same format.

[0017] When extracting the expected output, it is necessary to ensure that it logically corresponds one-to-one with the input text. The expected output can be a classification label, a target text paragraph, or a numerical prediction result, depending on the task type. During implementation, the expected output field is read as a corresponding set of sequences and stored at the index position corresponding to the input text sequence. To prevent inconsistencies, it is necessary to verify whether the number of input texts and expected outputs are equal, and to detect missing or duplicate items through a field validation mechanism.

[0018] The process of generating a benchmark set is based on the matching results between the input text and the expected output. At this point, a mapping structure containing the input text and the expected output is constructed, which can be a key-value dictionary or a two-dimensional table structure. The benchmark set not only stores the direct correspondence between input and output, but can also include information such as task type, data source, and field annotations. The generated benchmark set is typically stored in memory or a database with a standardized data structure, ensuring that subsequent model calls can retrieve it in batches in a standardized format.

[0019] This embodiment eliminates the inefficient manual data processing by parsing the input data, extracting the input text and expected output, and then generating a benchmark test set. This structured and standardized the matching relationship between data and tasks. This provides a unified data interface for subsequent model calls and performance evaluations, avoiding errors caused by format differences and missing data, and improving the stability and automation of the optimization process.

[0020] S20, Based on the benchmark test set, generate a set of prompt word parameter configurations according to multiple parameter dimensions; In this embodiment, a set of prompt word parameter configurations is generated based on a benchmark test set, according to multiple parameter dimensions. First, it is necessary to clarify the scope and definition of these parameter dimensions. Parameter dimensions refer to the controllable variables that can be adjusted when generating prompt words; these variables determine the performance of the prompt words during task execution. Typical parameter dimensions include the number of examples, example selection strategy, language style, and trigger phrases, etc., with each parameter dimension corresponding to an operable configuration method.

[0021] The number of examples controls how many examples are included in the prompt words. These examples are typically derived from input text and expected output pairings in a benchmark set. By setting different numbers of examples, the difference in the model's output under conditions of few and many examples can be tested. For example, one example represents a one-shot, and multiple examples represent a few-shot; the range of values ​​can be predefined in the benchmark set.

[0022] Example selection strategy refers to the rules used when selecting examples from the benchmark set. It can be random sampling, uniformly distributed sampling based on task labels, or highly relevant sampling based on semantic similarity. Different strategies affect the representativeness and bias of the cue word coverage, thus impacting the model's generalization ability.

[0023] Language style refers to the stylistic control of cue words in text expression, such as formal written style, colloquial style, and concise style. Its definition comes from the expression methods of natural language. By rewriting the cue word text template in different styles, the tone and expression preferences of the model in the output can be affected.

[0024] Trigger phrases are specific keywords or sentence structures used to guide model generation. They are usually located at the beginning or end of prompt words to activate the model's generation pattern. For example, phrases like "Please answer the following question" or "The final answer is" can help improve the model's response accuracy and consistency in task invocation.

[0025] When associating these parameter dimensions with the benchmark set, an exploration range needs to be set for each dimension. For example, the number of examples can be 1 to 5, the language style can be divided into formal and concise categories, and three candidate trigger phrases can be set. Multiple prompt word configurations are generated through combinations of parameter Cartesian products, and each configuration is recorded as an independent entry in the prompt word parameter configuration set. These entries are bound to the benchmark set to ensure that each configuration can be applied to the input text of the benchmark set, forming an instantiated set that can be used for model invocation.

[0026] This embodiment generates a set of prompt word parameter configurations by setting parameter dimensions based on a benchmark test set. This avoids the inefficient process of repeated manual trials and allows for systematic coverage of different parameter combinations. This ensures that prompt word optimization is comprehensive in breadth and targeted in depth, providing a standardized input source for subsequent automated evaluation and parameter updates.

[0027] S30, based on the prompt word parameter configuration set and the benchmark test set, call the language model interface to process the input text and obtain the model output set; In this embodiment, the input text is processed by calling the language model interface according to the prompt word parameter configuration set and the benchmark test set to obtain the model output set. The core of this step is to combine the configured prompt words with the input text in the benchmark test set and pass them to the target language model in batches through the interface call mechanism, and then collect and organize the returned results.

[0028] Each parameter configuration in the prompt word configuration set is paired with each input text in the benchmark set to generate a prompt word instance. A prompt word instance is a concretized invocation request, typically composed of the input text, the expected output information, and elements from the parameter configuration such as the number of examples, selection strategy, language style, and trigger phrase. For example, a configuration containing three user examples, a formal style statement, and a specified trigger phrase will, together with the input text, form a structured input for the model to parse.

[0029] The language model interface serves as a channel to connect to external models, typically via an API. It receives parameterized requests and returns text results. Implementing the interface requires setting runtime parameters, which determine the model's generation behavior. For example, the temperature parameter affects the randomness of the output; a larger value results in greater diversity, while a smaller value approaches determinism. The maximum generation length parameter limits the number of words or tokens in the output, preventing it from being too long or too short. The kernel sampling parameter controls the selection range of candidate words by setting a cumulative probability threshold. By combining these runtime parameters, the model's performance in generation tasks can be adjusted.

[0030] Concurrency performance and stability must be guaranteed during the call process. To achieve this, asynchronous batch calls can be used to submit multiple prompt word instances to the model interface in parallel, reducing overall waiting time. Simultaneously, a timeout control mechanism needs to be designed. If a request does not return a result within a set time, it should be proactively terminated and error retry logic should be triggered to ensure that the overall process is not blocked due to delays in individual calls.

[0031] The responses returned by the model often contain a mixture of structured and unstructured information, thus requiring result parsing. The parsing process extracts the raw responses into standardized output units according to the output format requirements set in the prompt word parameter configuration set, and stores these units as the model output set. This set contains the model-generated results of all input texts in the benchmark set under different parameter configurations, serving as direct input for subsequent comparison and evaluation.

[0032] This embodiment utilizes an interface call mechanism based on the prompt word parameter configuration set and the benchmark test set to systematically collect the model's output under different conditions, avoiding the inefficiency and inconsistencies caused by manual trial and error. This mechanism ensures that the output set covers the diversity of parameter configurations and generates standardized results that can be used for objective evaluation through an automated calling and parsing process.

[0033] S40, compare the model output set with the expected output in the benchmark test set to generate a comprehensive performance index set; In this embodiment, the model output set is compared with the expected output in the benchmark test set to generate a comprehensive performance index set. The core of this process is to quantify the degree of deviation between the model output and the target result through a structured evaluation mechanism and integrate the results of multiple evaluation dimensions into a computable index set.

[0034] The model output set consists of standardized outputs obtained from the previous step of calling the language model interface. Each output corresponds one-to-one with the input text in the benchmark set. The expected outputs in the benchmark set are correct results pre-annotated by humans or the system, and are usually considered as the standard answers. The comparison process uses the standard answers as a reference to check the performance of each model output in terms of task completion and result quality.

[0035] The comparison process requires selecting appropriate analysis dimensions. Common task-related analysis dimensions include classification accuracy, recall, F1 score, and exact match rate, which are suitable for classification and matching tasks. Content quality analysis dimensions focus more on the naturalness and readability of the output, such as language fluency, consistency, and semantic similarity. These dimensions are typically achieved through text similarity calculations, scoring models, or manually labeled data-driven evaluation modules.

[0036] Each model output generates a corresponding metric value. Task-related metrics are often based on explicit label comparisons, such as whether the output category is completely consistent with the standard category; while content quality metrics may be obtained by calculating the cosine similarity between sentence embedding vectors or by using a specific quality scoring model.

[0037] To eliminate the dimensional differences between various indicators, normalization is required, mapping all indicators to the same numerical range, such as 0 to 1. Normalized values ​​facilitate horizontal comparisons and subsequent aggregation. Next, based on preset dimensional weights, the normalized indicators are weighted and summed to obtain a comprehensive performance score. The weights reflect the relative importance of different dimensions in a specific task; for example, in an information extraction task, the weight of task-related dimensions may be higher than that of content quality dimensions.

[0038] Finally, the overall performance scores of all models are aggregated to form a comprehensive performance index set. This set reflects the overall performance level of the output and provides a quantitative basis for subsequent parameter configuration adjustments.

[0039] This embodiment, through its comparison and indicator generation mechanism, avoids biased conclusions based on a single indicator, enabling multi-dimensional and weighted performance evaluation. This allows the evaluation results to reflect both the accuracy of task completion and the quality of language, thus providing a comprehensive and objective basis for subsequent parameter adjustments.

[0040] S50, adjust and update the prompt word parameter configuration set according to the comprehensive performance index set; In this embodiment, adjusting and updating the prompt word parameter configuration set based on the comprehensive performance index set is a data-driven parameter iteration process. Its goal is to analyze the performance of each parameter configuration to determine which parameter dimensions positively impact the model's output performance and which are deficient, thereby enabling targeted modifications to the prompt word parameter configuration set.

[0041] The comprehensive performance metric set is a numerical set generated in the previous step. It includes a comprehensive performance score formed by normalizing and weighting the numerical values ​​of task-related metrics and content quality metrics. These scores can intuitively reflect the gap between the model output and the expected output under different parameter configurations. The analysis process typically involves comparing the comprehensive performance scores corresponding to different parameter configurations to identify trends in performance.

[0042] In the adjustment and update phase, it's necessary to first identify the parameter dimensions. These include parameters such as the number of examples, example selection strategy, language style, and trigger phrases; each dimension can potentially affect the output. When a particular dimension consistently performs poorly across multiple combinations, it is marked as needing optimization. To avoid arbitrariness in the adjustment process, a set of optimization strategies is typically defined, such as increasing the range of parameters with lower weights, reducing the number of redundant samples, or changing the candidate set of trigger phrases.

[0043] Generating parameter adjustment instructions is the core of the adjustment process; these instructions are executable operational rules. For example, expanding the number of examples parameter from 3 to 5, or switching the language style from formal to concise. Once executed, these instructions modify the prompt word parameter configuration set, thereby generating an updated prompt word parameter configuration set.

[0044] To ensure the traceability of adjustments and updates, version registration is also required for the updated configuration set. During version registration, the system records the parameter values ​​before and after the modification, the corresponding overall performance score snapshot, and the executed adjustment commands. This mechanism allows for the review of historical versions at any time to verify the rationality of the adjustment path.

[0045] This embodiment achieves data-driven parameter optimization by adjusting and updating the set of prompt word parameter configurations based on a comprehensive performance index set, avoiding the inefficient process of blindly modifying parameters based on human experience. This mechanism can gradually converge to a better-performing parameter combination through multiple iterations, resulting in simultaneous improvements in both accuracy and quality of the model output. Furthermore, version registration ensures the transparency and traceability of the optimization path.

[0046] S60, after the prompt word parameter configuration set is updated, determine whether the termination condition is met; In this embodiment, determining whether the termination condition is met after the prompt word parameter configuration set is the dividing point of the entire iterative optimization process. It determines whether the system continues to execute the optimization loop or outputs the final result. The determination of the termination condition needs to be based on a comparison of historical and current performance data, combined with the iteration limit, to form a clear stopping criterion.

[0047] First, it's necessary to obtain the comprehensive performance metric set generated in the current iteration, as well as the comprehensive performance metric set generated in the previous iteration. The comprehensive performance metric set is a weighted aggregation of task-related metric values ​​and content quality metric values, directly reflecting the overall performance under the current prompt word parameter configuration set. Comparing the current set with the previous set reveals the trend of performance changes.

[0048] During the comparison process, the combined performance scores of the two sets are typically extracted, and the relative rate of change between them is calculated. The relative rate of change is used to measure whether the optimization has brought about a significant improvement. For example, if the score in this round is only slightly different from the score in the previous round, it indicates that the performance is close to convergence. To avoid endless iterations, a convergence threshold needs to be set, which can be a lower bound of change in percentage form, such as less than 1% or 0.5%.

[0049] In addition to performance convergence assessment, iteration count limitations must also be considered. The system maintains a counter, incrementing it after each update. When the cumulative iteration count reaches a preset maximum value, even if the performance change rate still exceeds the convergence threshold, the system is deemed to have met the termination condition, ensuring that computational resource consumption remains controllable.

[0050] Finally, when the relative rate of change is less than the convergence threshold, or the number of iterations reaches the maximum threshold, the result is "Termination condition met"; if neither is met, it is judged as "Termination condition not met", and the optimization loop continues to execute.

[0051] This embodiment introduces a termination condition check after updating the prompt word parameter configuration set, which effectively avoids resource waste and invalid iterations. While ensuring gradual performance optimization, it ensures automatic termination when performance stabilizes or the iteration limit is reached. This not only improves overall operating efficiency but also enhances the controllability and predictability of the optimization process.

[0052] S70: If the termination condition is not met, return the set of generated prompt word parameter configurations and execute the model call, output comparison, and parameter update operations in sequence; if the termination condition is met, output the target prompt word configuration and optimization report.

[0053] In this embodiment, the return execution mechanism when the termination condition is not met uses the comprehensive performance index set as the driving signal to establish an automatically routed execution channel for the regeneration of the prompt word parameter configuration set, model invocation, output comparison, and parameter update. The termination condition comes from the judgment result of the iteration control unit, which includes a joint judgment of the relative change rate of the set-level comprehensive performance score and the maximum iteration limit. The judgment output is a Boolean signal that directly controls the subsequent flow. Returning to generate the prompt word parameter configuration set means that based on the difference distribution between the parameter version of the previous round and the record of the comprehensive performance index set, new parameter combination entries are formed and written into the parameter pool. The entries maintain a one-to-one binding relationship with the benchmark test set so as to facilitate the subsequent batch production of prompt word instances. This return process needs to explicitly reference the version identifier of the prompt word parameter configuration set generated in the previous update and save the source version, change reason, and target test round in the metadata to ensure that the path is reproducible and rollback is feasible.

[0054] The sequential execution of model calls means combining the new set of prompt word parameter configurations with the input text of the benchmark set using a Cartesian combination to generate a sequence of prompt word instances, which is then submitted to the language model interface to obtain the model output set concurrently or in batches. This step requires adhering to the established rules for setting the running parameters or using the adjusted running parameters derived from the previous round of feedback. The mapping key between each prompt word instance and parameter entry must be recorded at the call layer to ensure that the model output set can be seamlessly backfilled into the parameter entries.

[0055] The sequential output comparison process involves aligning the newly obtained model output set with the corresponding expected outputs in the benchmark set, generating numerical indicators for task fit and content quality. These indicators are then normalized and weighted to form a new set of comprehensive performance indicators. Simultaneously, a set-level comprehensive performance score is generated to support the next termination condition determination. This comparison process must retain the mapping key to the prompt word parameter configuration set entries, forming a traceable path from parameters to indicators.

[0056] The meaning of sequentially performing parameter updates is to modify the entries in the prompt word parameter configuration set based on the new comprehensive performance index set. The modification actions include expanding or shrinking the range of example quantity, replacing the example selection strategy, switching the language style template, adding, deleting and rearranging the trigger phrase candidates, etc. The modification results are registered as the new parameter version and a snapshot is taken with the comprehensive performance index set of this round to form a basis for performance comparison between versions.

[0057] The output action upon meeting the termination condition comprises two parts. The first part, the target cue word configuration, selects the parameter entry with the optimal set-level overall performance score and meeting the constraints from all versions of the cue word parameter configuration set, and freezes it as a distributable configuration unit. The second part, the optimization report, compiles summary statistics of parameter versions for each round, the longitudinal trend of the overall performance index set, iteration counts, and resource consumption records, forming a structured document with visual attachments. The report should include a key list for reproducing the experiment, ensuring that external systems can directly replicate or audit it.

[0058] Throughout the routing process, the control flow is executed using a state machine. The state set includes two branches: return generation and model invocation, output comparison and parameter update, and termination. State transitions depend on decision signals, queue status, and resource quotas. Transition actions are bound to idempotent keys and rollback mechanisms. In case of call failure or comparison anomalies, retry or skipping can be performed by marking the entry, ensuring that the overall progress is not blocked by a single point. To reduce resource consumption, the execution channel can archive intermediate components after each batch, retaining only the prompt word parameter configuration set version, model output set index, and comprehensive performance index set summary, thus releasing the intermediate cache.

[0059] The control plane can employ either centralized scheduling or distributed orchestration. Centralized scheduling is suitable for environments with moderate data volumes and latency-sensitive objectives. A single scheduler maintains state machine transitions and task queues, facilitating unified rate limiting and log aggregation. Distributed orchestration is suitable for environments with large data volumes or those requiring cross-regional operation. Lightweight executors can be deployed across multiple execution nodes, with a message bus distributing prompt word instances and collecting model output sets. State snapshots are written to shared metadata storage. Both approaches can be configured with idempotent key strategies, using a combination of parameter version identifiers and input text identifiers as unique execution identifiers, directly hitting cached results when submitting repeatedly.

[0060] The return generation stage can employ grid exploration to cover the discrete parameter space, or a Bayesian optimization-based proposer to sample high-scoring regions from the previous round's comprehensive performance metric set. Alternatively, genetic recombination can be used to retain fragments of high-scoring parameter entries with minor perturbations. If the benchmark set is large, a bucketing strategy can be used to partition the input text by scene, length, or label, generating an independent subset of parameters for each partition. Then, cross-partition merging can be performed after the set-level comprehensive performance score converges.

[0061] The model invocation process can run at a fixed concurrency level, suitable for offline environments with stable resources; it can also run at an elastic concurrency level, monitoring interface latency and error rate in real time and dynamically adjusting the concurrency threshold; or it can run on a budget-driven basis, using a cost threshold as an external constraint, automatically reducing the temperature, shortening the maximum generation length, or reducing the number of prompt word instances when the accumulated cost approaches the threshold.

[0062] The output comparison process can choose to strictly match the evaluation that focuses on label consistency, or it can choose to evaluate both semantic similarity and readability. It can also add compliance clause checks and professional terminology coverage checks for financial or healthcare scenarios. All indicator values ​​enter a unified normalization channel, and the weight matrix is ​​provided by the configuration file or strategy engine. During execution, it supports adaptive adjustment by round number or by partition.

[0063] In the parameter update process, univariate fine-tuning can be adopted to observe causal relationships, or multivariate linkage can be adopted to accelerate the escape from local optima. In the univariate fine-tuning mode, only one dimension is changed in each round and the change level is limited, which is convenient for constructing response curves. In the multivariate linkage mode, the combination of dimensions with greater indicator elasticity is selected first for joint perturbation, while the baseline entries of the previous round are retained as control groups. After the update is completed, the version is uniformly registered and written into the mapping key.

[0064] The output action of the terminated branch can be extended to two implementation forms. The first is offline output, where the target prompt word configuration and optimization report are written to the artifact repository and a notification is triggered. The second is online output, where the target prompt word configuration is pushed to the configuration slot of the online inference service, gradually replacing the old configuration using a gray-scale weighting approach from low to high, while continuously collecting online feedback in the background for subsequent offline batch retraining. Both forms retain rollback scripts; if abnormal fluctuations occur in online feedback, a quick revert to the previous stable version can be made.

[0065] This embodiment introduces an automatic routing mechanism based on a comprehensive set of performance indicators to achieve continuous regeneration and evaluation loops when termination conditions are not met, as well as immediate output and archiving when termination conditions are met. This reduces the time and cost of repeated manual trial and error, and ensures full-link traceability and reproducibility through versioning and mapping keys. The stability and portability of target prompt word configuration are quantifiablely improved.

[0066] In one embodiment, step S10 above includes: S101, Receive a structured data file containing input text fields and expected output fields; S102, parse the format of the structured data file, identify and extract the input text sequence and the corresponding expected output sequence; S103, verify whether the field mapping relationship between the input text sequence and the expected output sequence conforms to the preset format constraints, and obtain the input text sequence and the expected output sequence after verification. S104, preprocess the input text sequence after verification to obtain a preprocessed input text sequence; S105, classify and label the expected output sequence after the verification is passed according to the task type, and construct a set of standard answers with classification labels; S106, Based on the preprocessed input text sequence and the standard answer set, generate a benchmark test set.

[0067] In this embodiment, when receiving a structured data file containing input text fields and expected output fields, source verification, integrity verification, and security verification must be performed simultaneously. A structured data file is a carrier with fixed field names and resolvable record boundaries. Its source can be exported from a business system, generated by a data synchronization task, or produced by an annotation platform. Common encapsulation formats include tabular text, hierarchical tagged text, or columnar storage snapshots. The receiving process first identifies the outer container and character encoding, establishes a mapping list from field names to memory fields, and records the file line number, primary key, timestamp, and source identifier for subsequent traceability and deduplication. The input text field is a column carrying the text to be processed, which may contain natural language questions, summaries, instructions, or descriptive paragraphs. The expected output field is a column carrying the standard answer or target label, which may contain discrete labels, short text, or structured fragments. The source of both types of field names is usually given by business definitions or annotation specifications. During reception, field name capitalization, hidden whitespace, and alias mapping should be checked to ensure a one-to-one correspondence between the input fields and logical fields.

[0068] When parsing the format of structured data files, row-level decomposition and column-level extraction must be performed without altering the original semantics. The parsing objects are the container structure, field order, escape conventions, and missing value representations. The parsing result is two sequences of equal length: the input text sequence and the expected output sequence. The input text sequence is constructed by concatenating the input text fields of each record in their original order; the expected output sequence is constructed by concatenating the expected output fields of each record in the same order. To ensure alignment, a record index is built during the parsing phase. The index entries store the source row number, primary key, and hash digest. Any subsequent reordering relies on the index for stable reconstruction. Parsing also needs to handle special characters and nested delimiters, such as separators within quotation marks, escape characters, and embedded text within newlines. This handling strategy can be achieved through pattern matching and a delimiter state machine working together. Finally, a two-dimensional table view from fields to values ​​is constructed in memory and then projected into the two sequences.

[0069] When verifying the field mapping relationship between the input text sequence and the expected output sequence, consistency judgment is required based on preset format constraints. Preset format constraints are a set of structured constraints used to ensure that two sequences are comparable, usable, and traceable. They are typically derived from a combined set of data annotation and evaluation specifications, and include requirements such as the existence of essential fields, equal record length, primary key uniqueness, upper limit for null values, illegal character masking list, and language encoding consistency. The verification process generates a verification result for each record, including a pass flag, defect type, and location information. Passing records proceed to subsequent processes, while failing records are isolated and remediation suggestions are output. To ensure reproducibility, all constraints are saved in versioned configuration form, and constraint versions and summaries of hit items are written to the verification log.

[0070] When preprocessing the validated input text sequence, the goal is to eliminate task-irrelevant formatting noise and standardize text morphology to ensure fair and comparable comparisons in subsequent tests. The preprocessed input text sequence is a normalized, equal-length sequence, with each item corresponding to a cleaned text. Preprocessing operations include whitespace normalization, control character removal, punctuation standardization, case merging, localized symbol replacement, unified encoding, and visible character set constraints. Language detection and unified segmentation can also be performed as needed. If the input text contains structural placeholders or confidential segments, a placeholder replacement table can be used to replace them with general symbols, while the restore key is retained in the mapping table for backfilling if necessary. In multilingual scenarios, a language label column can be introduced to store the inferred language for selecting appropriate similarity metrics and word segmentation strategies during subsequent evaluation, without changing the position of the original sequence.

[0071] When classifying and labeling the expected output sequence after successful verification according to task type, it is necessary to convert free text or original labels into a unified representation of the standard answer set. Task types come from upstream business definitions or labeling specifications and may include categories such as classification, matching, generation, and extraction. The goal of classification labeling is to merge the expected output into a comparable discrete or semi-structured space. For classification or matching, text labels can be mapped to specification items in the standard label table, and aliases, hierarchies, and parent-child relationships can be recorded. For generation or extraction, target text can be converted into controlled templates or structured fragments, such as key-value pairs, paragraph templates, or field lists. During classification labeling, invalid labels, ambiguous labels, and cross-domain labels need to be handled. Handling strategies include alias merging, hierarchical promotion, or discarding and recording the reasons. The final generated standard answer set and the input text sequence maintain a one-to-one correspondence through a record index. Each entry contains the standardization target, the original target, and the mapping trajectory to support interpretability review during subsequent evaluation.

[0072] When generating a benchmark set based on the preprocessed input text sequence and the standard answer set, the two equal-length sequences, along with the index, metadata, and task type, need to be packaged into a unified carrier that can be directly used downstream. The benchmark set not only includes the mapping between the input text and the standard answers but also record indexes, field summaries, and source identifiers for retrieval and tracing. The storage format can be a row-oriented table, a column-oriented file, or a collection of key-value pairs; the access interface needs to provide the ability to query by index, filter by task type, and paginate by batch. To ensure consistency, the generation process remains idempotent to any re-import, and hash summaries are compared and existing entries are reused during repeated construction. To ensure portability, the benchmark set carries a field dictionary, language tags, and annotation versions during export, enabling consistent reconstruction across different operating environments. After generation, a validation report is written, recording sequence length, missing rate, number of isolated records, and major transformation statistics, providing baseline information for subsequent steps.

[0073] This embodiment connects receiving, parsing, verifying, preprocessing, classification, labeling, and generation into a stable data link. The input text and the expected output are unified into a mapping structure of equal length, alignment, and traceability. Abnormal samples are isolated and recorded before entering the evaluation. The text form and label representation are standardized before entering the evaluation. This reduces the uncertainty caused by manual cleaning and temporary repair, reduces the evaluation bias caused by mismatch, and improves the stability and repeatability of subsequent parameter generation, interface calls, and comparison evaluation. Furthermore, indexes and version records ensure the feasibility of cross-environment migration and experiment reproduction.

[0074] In one embodiment, step S20 above includes: S201, Based on the task types included in the benchmark test set, initialize the prompt word parameter configuration framework; S202, set the exploration range for the example quantity parameter, example selection strategy parameter, language style parameter and trigger phrase parameter in the prompt word parameter configuration framework respectively; S203, Based on the exploration range, generate a set of prompt word parameter configurations containing multiple parameter combinations; S204, associate the prompt word parameter configuration set with the benchmark test set.

[0075] In this embodiment, when initializing the prompt word parameter configuration framework based on the benchmark test set, it is necessary to first read the task type identifier, input text distribution, standard answer format, and length statistics within the benchmark test set, and then establish an extensible configuration skeleton based on this information. The configuration skeleton is used to carry parameter dimensions, candidate space, and dependency constraints, including named fields such as example quantity parameters, example selection strategy parameters, language style parameters, and trigger phrase parameters, as well as mutual exclusion and priority relationships between fields. The task type is derived from the task definition items in the data collection or annotation specifications, such as labels for classification, matching, generation, and extraction; the input text distribution and standard answer format are derived from the field summary of the benchmark test set; the length statistics are derived from the aggregated information of text length and annotation length. After the skeleton is generated, a version identifier and a source identifier are written, forming a reusable configuration unit.

[0076] When setting the exploration range for the number of examples, constraints are imposed by the upper limit of the benchmark set capacity, the length of a single input, and the context carrying capacity. Candidate tiers ranging from single to multiple examples are given, and the length budget and example percentage upper limit for each tier are clearly defined. The number of examples is derived from the difference in context dependence between the few-sample and single-sample paradigms in the prompt word design. The length budget comes from the context carrying capacity of the underlying generation engine and the statistics of the average input length. The example percentage upper limit is derived from the constraint of avoiding examples from overwhelming the main input in the case of multiple samples. The exploration range is presented as a list of discrete tiers, maintaining a monotonic relationship between tiers to facilitate subsequent enumeration and filtering.

[0077] When setting the exploration range for selecting strategy parameters for examples, considering the varying requirements for representativeness and stability across different tasks, three candidate methods are proposed. One method is uniform sampling, sampling equally across intervals based on label distribution or scene partitioning to improve coverage. Another is semantic proximity, introducing nearest neighbor retrieval in the text vector space to prioritize examples semantically closer to the current input, thus improving fit. A third method is hard-example priority, utilizing poorly performing samples from historical evaluation records as example sources to enhance the sensitivity of parameter tuning. The three methods originate from the label balance principle, the semantic proximity principle, and the hard-example-driven principle, respectively. To avoid introducing unnecessary restrictions, all three methods can be included as candidates in parallel, and additional constraints can be superimposed during the combination phase, such as preventing the same input from being paired with duplicate examples.

[0078] When defining the scope of exploration for language style parameters, several style template groups are established based on task type and target audience. These can include formal writing groups emphasizing terminology completeness and sentence structure regularity; concise explanation groups emphasizing short sentences and direct expression; and end-user-oriented groups emphasizing readability and prompting tone. Style templates are derived from a style library, which consists of template phrases, sentence fragments, and tone markers. A mapping table exists between templates and task types to avoid inappropriate combinations of expressions. Each group includes a style granularity and a list of reserved placeholders to ensure consistency with the number of examples and trigger phrases when assembling the templates.

[0079] When setting the exploration scope for trigger phrase parameters, a candidate phrase set is set up around two functions: instruction guidance and output positioning. Instruction guidance phrases are used to clarify the task intent, such as "Please complete the classification judgment" or "Please give the summary conclusion"; output positioning phrases are used to fix the start or end position of the output, such as "The answer is" or "The conclusion is as follows". The phrase set is derived from previous evaluation and language practice. Each phrase entry in the set includes a trigger position, possible concatenation objects, and a list of conflicting phrases. The trigger position is used to define whether it appears at the beginning or end of the prompt word, the possible concatenation objects point to the style templates or example forms that can be combined with it, and the list of conflicting phrases is used to avoid semantic repetition or mutually canceling collocations.

[0080] When generating a set of prompt word parameter configurations containing multiple parameter combinations based on the exploration scope, it is necessary to enumerate and filter the four types of parameters without exceeding the length budget. The recommended combination order is to prioritize parameters related to the number of examples, followed by parameters related to example selection strategy, and then parameters related to language style and trigger phrases, to reduce invalid combinations. Combination generation can employ grid enumeration to cover the discrete candidate space, or stratified sampling to control the proportion of each interval in multi-distribution scenarios; it can also perform encrypted enumeration on local areas with high scores in the previous evaluation round to improve exploration efficiency. Each combination entry includes a length validator, which determines whether it meets the context carrying limit based on the sum of the input length and example length, and the trigger phrase length estimation; combinations that do not meet the limit are directly discarded. All validated combination entries are written into the prompt word parameter configuration set, with the entry structure including the values ​​of the four types of parameters, a length estimation summary, and a source version identifier.

[0081] When associating the prompt word parameter configuration set with the benchmark set, a bidirectional mapping needs to be established. On one hand, the mapping points each parameter combination entry to the input text index set in the benchmark set, supporting subsequent expansion of prompt word instances by combination entry; on the other hand, it points the input text index to the registered parameter combination entries, facilitating source tracing during the evaluation phase. The mapping storage uses a key-value structure, where the key is the concatenation key of the combination entry identifier and the input text identifier, and the value is the metadata digest and timestamp. To support cross-environment reproducibility, the association process simultaneously outputs snapshots of the combination list and the index list, containing a field dictionary, task type, and version identifier. After association, the prompt word parameter configuration set has comprehensive coverage of the benchmark set; any input can be combined with several valid combination entries to form a prompt word instance, which can then be used in subsequent invocation and evaluation processes.

[0082] This embodiment establishes a configuration skeleton around a benchmark test set, defines an executable exploration range for four types of parameters, controls the effectiveness of combinations with length verification and mutual exclusion constraints, and binds parameter combination entries to input text indices through bidirectional mapping, forming a sufficiently comprehensive and reproducible set of prompt word parameter configurations. This allows for the generation of multiple sets of configuration units that can be directly used for invocation and evaluation in a single process, reducing the time consumption and deviation risks associated with repeated manual trials. Subsequent invocations and comparisons can be carried out under unified constraints and a consistent context budget, fostering a more stable evaluation loop and a clearer direction for parameter tuning.

[0083] In one embodiment, step S30 above includes: S301, combine each parameter configuration in the prompt word parameter configuration set with each input text in the benchmark test set to generate a corresponding prompt word instance; S302 configures the runtime parameters for the language model interface call, including temperature parameters, maximum generation length parameters, and kernel sampling parameters; S303, the language model interface is called in batches in an asynchronous concurrent manner to process the prompt word instances, and timeout control and error retry mechanism are executed during the call process; S304, parse the response returned by the language model interface, and extract and organize the model output results from the response according to the output format requirements defined in the prompt word parameter configuration set to generate the model output set.

[0084] In this embodiment, when combining the prompt word parameter configuration set with the benchmark test set, a stable identifier is first generated for each parameter configuration in the prompt word parameter configuration set. Then, the input text index of the benchmark test set is read, and Cartesian pairing is performed according to the parameter configuration identifier and the input text index to obtain a prompt word instance sequence. Each prompt word instance includes the input text, the number of examples, example fragments generated by the example selection strategy, language style templates, trigger phrases, and output format requirements. A length estimation summary and source version are also recorded; the length estimation is used to verify context carrying limitations before submission. During the runtime parameter preparation phase, a one-time runtime parameter snapshot is built for the language model interface. This snapshot includes at least the temperature parameter, the maximum generation length parameter, and the kernel sampling parameter. The temperature parameter controls generation diversity, the maximum generation length parameter constrains the returned content range, and the kernel sampling parameter limits the probability and quality of candidate words. The runtime parameter snapshot is associated with the prompt word instance identifier to ensure replayability and comparison within the same batch. During the batch call phase, the prompt word instance sequence is pushed asynchronously and concurrently. Before submission, the sequence is fragmented and written to the task queue, which is then processed according to a concurrency threshold. The concurrency threshold is dynamically adjusted based on interface latency and quota to avoid congestion. Each task carries a timeout limit and a number of retries. Upon reaching a timeout, it is marked as a timeout event and enters the retry branch. The retry branch waits exponentially before resubmitting. If it still fails, the fault type is recorded and it enters the degradation channel. The degradation channel can shorten the maximum generation length or reduce concurrency to relieve resource pressure. To ensure traceability of the call process, each request is recorded with a request identifier, submission time, parameter configuration identifier, input text index, and a snapshot summary of runtime parameters. After the response is returned, the response time and quota consumption are recorded. In the response parsing phase, structure extraction is first performed according to the output format requirements. If the target is a structured fragment, the returned text is delimited, segmented, key-value extracted, and placeholders restored. If the target is a text paragraph, paragraph location, heading mark cleanup, and redundant truncation are performed. If the target is a tag value, tag mapping and illegal value filtering are performed. The parsing process continuously verifies length boundaries, character set, and encoding consistency, and writes standardized records using the parameter configuration identifier and input text index as a joint key. When assembling the model output set, all standardized records under the same parameter configuration identifier are aggregated to form a batch snapshot for evaluation. The snapshot contains a field dictionary, timestamp, response coverage, and failure reason distribution. When the coverage is lower than the threshold, a supplementary collection process is automatically triggered. The supplementary collection process only resubmits the call to complete the set for missing records. To ensure end-to-end consistency, all intermediate states retain a mapping table. The mapping table maintains a one-to-one traceable relationship between prompt word instances, runtime parameter snapshots, and standardized records. Any replay can reconstruct the same model output set by using the composite key.

[0085] This embodiment uses parameter configuration identifiers and input text indexes as joint keys to complete instantiation, parallel submission, timeout retries, and formatted parsing. The model output set is synchronously guaranteed in terms of coverage integrity, structural consistency, and traceability. Manual line-by-line trials are replaced by batch processes, and the comparison of running parameters has a replayable basis. Subsequent evaluation and update processes can make quantitative decisions based on stable inputs, thereby reducing unnecessary repeated attempts and shortening the overall optimization cycle.

[0086] In one embodiment, step S40 above includes: S401, Based on the task type corresponding to the benchmark test set, select the task matching analysis dimension and content quality analysis dimension from the preset indicator library; S402, Based on the task fit analysis dimension, determine the task fit index value of each output in the model output set and the corresponding expected output in the benchmark test set; S403, Based on the content quality analysis dimensions, determine the content quality index value for each output in the model output set; S404 normalizes all task fit index values ​​and content quality index values. S405, based on the preset weights of each analysis dimension, performs a weighted summation of the normalized task fit index values ​​and content quality index values ​​to generate a comprehensive performance score for each model output. S406 aggregates the overall performance scores of all models to form a set of overall performance indicators.

[0087] In this embodiment, the determination of the task fit analysis dimension and content quality analysis dimension is based on a pre-set indicator library. This library is a set of metrics organized for task types such as classification, matching, generation, and extraction, and can be sourced from internal enterprise evaluation standards, industry review guidelines, and domain annotation standards. The task fit analysis dimension measures the consistency between the output and the expected output in terms of goal achievement. For classification and matching, it focuses on label consistency, boundary consistency, and fragment alignment; for generation and extraction, it focuses on key point coverage, factual correspondence, and structural alignment. The content quality analysis dimension measures the readability and expression quality of the output itself, covering elements such as language fluency, paragraph coherence, terminology usage, and format completeness. In the healthcare business field, coverage of medical terminology and rigorous narration can be added; in the fintech business field, avoidance of compliance sensitivities and prudent wording can be added. The indicator library is managed by a mapping table from task type to dimension item. Each dimension item includes a metric definition, input prerequisites, and output format description, ensuring a consistent entry point for different tasks in the dimension selection process.

[0088] The determination of task fit index values ​​follows the principle of line-by-line alignment. The model output set and the benchmark set are mapped one-to-one using record indexes, and after alignment, they proceed to the measurement stage. For classification and matching tasks, label comparison and fragment alignment generate discrete hit and miss markers, which are then aggregated into sample-level task fit index values ​​based on the hit rate at the sample level. For generation and extraction tasks, key point localization or target field extraction is performed first, and then task fit index values ​​are generated based on the coverage and consistency of the key point set. The sample granularity remains unchanged during calculation, and all intermediate results are bound to input indexes and parameter configuration identifiers for subsequent traceability. If empty output, corrupted format, or missing target fields occur, the anomaly type is recorded and the process enters the missing data handling channel. The missing data channel uses placeholder values ​​and anomaly weighting strategies to avoid affecting subsequent aggregation, and the anomaly trajectory is retained in the evaluation log.

[0089] Content quality metrics are determined based on the inherent attributes of the text. Fluency and paragraph coherence are scored using quantifiable signals such as syntactic breakage rate and the proportion of redundant segments. Terminology usage is assessed using terminology lists and alias mappings to statistically analyze hits and misuses. Format integrity is verified using output format templates to check delimitation, tagging, and key-value pairing. A unified extractor and validator are used for cross-task content quality dimensions. The extractor is responsible for locating sentence units and symbol structures from the output text, while the validator assigns scores on a unified scale based on the dimension definitions. To accommodate the differentiated requirements of healthcare and fintech, domain-specific extended dimensions can be overlaid on top of the general dimensions. Examples include medical logic consistency or compliance-sensitive phrase masking. Extended dimensions and general dimensions follow the same extraction and scoring interfaces to ensure they ultimately enter the same aggregation channel.

[0090] Normalization is used to eliminate differences in the units and ranges of values ​​across different dimensions. All task fit metric values ​​and content quality metric values ​​are uniformly converted to a fixed range by a normalizer before entering the aggregation channel. The normalizer records the target range, anomaly truncation strategy, and missing value imputation strategy. The target range facilitates horizontal comparison, anomaly truncation reduces the impact of extreme values ​​on the overall result, and missing value imputation ensures that subsequent weighting operations do not fail due to missing dimensions. The normalizer's parameters and version information are written into the evaluation metadata with each batch, ensuring comparability and reproducibility between different batches.

[0091] Weighted summation is used to generate the overall performance score. Dimension weights are derived from a configuration table of task types to weight vectors, which is maintained through experience accumulation and offline backtesting. During aggregation, the normalized task fit index value and content quality index value of the same sample are weighted and summed to obtain the sample-level overall performance score. When domain-extended dimensions exist, the extended dimensions share the weight space with the general dimensions or use grouped weights. The weight vector is archived together with the batch number for easy interpretation of the score composition later. If a sample triggers an abnormal channel, the overall performance score is accompanied by an anomaly marker and confidence indicator to avoid misuse in subsequent decisions.

[0092] The comprehensive performance index set is formed based on sample-level comprehensive performance scores. An aggregator generates summary statistics and distribution information according to parameter configuration identifiers, task types, and data binning. The aggregator outputs two types of products: a detailed table, which stores the comprehensive performance score and dimensional decomposition of each sample; and a summary table, which stores the central tendency, dispersion, and quantile information from the perspective of parameter configuration, and provides a set-level comprehensive performance score. The detailed table is for tracing and diagnosis, while the summary table is for decision-making and comparison. Both types of products are bound to parameter configuration identifiers, benchmark test set versions, index library versions, and weight versions, forming a complete evaluation evidence chain. To reduce statistical bias, the aggregator supports outputting stratified results based on conditions such as task type, text length, and domain tags, facilitating more detailed configuration comparisons and optimizations within the stratified space.

[0093] This embodiment measures both task fit analysis and content quality analysis simultaneously, eliminates dimensional differences using a unified normalizer, completes interpretable aggregation using a weight configuration table, and outputs a comprehensive performance index set from both detailed and summary perspectives. The deviation between the model output set and the expected output is quantified into a stable, reproducible, and traceable scoring system. The evaluation results cover both the degree of goal completion and text quality attributes, providing consistent input for subsequent parameter generation, interface calls, and parameter updates. This reduces the bias and randomness caused by a single index and shortens the time window for diagnosing bottlenecks and identifying directions for improvement.

[0094] In one embodiment, step S50 above includes: S501, Based on the comprehensive performance index set, analyze the performance of each configuration in the prompt word parameter configuration set, and identify the parameter dimensions that need to be optimized; S502, Based on the parameter dimensions that need to be optimized, generate parameter adjustment instructions according to a preset optimization strategy; S503, according to the parameter adjustment instruction, modify the corresponding parameters in the prompt word parameter configuration set to generate an updated prompt word parameter configuration set; S504, perform version registration on the updated prompt word parameter configuration set, and record the adjustment details and corresponding performance index snapshots.

[0095] In this embodiment, the comprehensive performance index set serves as the tuning signal source, including fields such as sample-level comprehensive performance score, dimensional decomposition value, anomaly marker, and hierarchical statistics. A one-to-one traceable relationship is established through parameter configuration identifiers and input text indexes. The prompt word parameter configuration set consists of multiple parameter entries, each storing at least the example quantity parameter, example selection strategy parameter, language style parameter, trigger phrase parameter, length estimation summary, and source version. The parameter dimension points to the named set of the aforementioned adjustable fields, used to locate the specific location and value space requiring change. The preset optimization strategy is a set of executable change rules and boundary constraints, specifying which indicators trigger which changes and the allowed value ranges and mutual exclusion relationships. The parameter adjustment instruction is a single, actionable operation record, using parameter configuration identifiers and field keys as location keys, including the old value, new value, trigger evidence, and effective scope. Version registration is a time-oriented state management behavior; when the parameter set is modified, a new version identifier is generated, and adjustment details and performance index snapshots are recorded, ensuring historical traceability and rapid rollback.

[0096] When analyzing the performance of each configuration in the prompt word parameter configuration set, the sample-level comprehensive performance scores in the comprehensive performance index set are first aggregated into three statistical categories: central tendency, dispersion, and quantile, using the parameter configuration identifier as the key. Comparison values ​​are then output within hierarchical spaces such as task type, text length, and domain label. To avoid bias caused by differences in sample composition, weighted correction is introduced to align the hierarchical proportions to the baseline distribution before comparing the magnitude of the aggregated differences. Subsequently, sensitivity estimation is performed on the dimensional decomposition values. Single-factor perturbation or multi-factor regression is used to separate the marginal contributions of each parameter dimension from the comprehensive performance score, generating a ranking of parameter dimension importance and confidence intervals. When the sample size is insufficient or the variance is too large, a regression to robust statistics within the strata is automatically performed to reduce the impact of noise. Parameter dimensions requiring optimization are identified, and dimensions with negative contributions or confidence lower bounds crossing the zero line are selected in the ranking. Simultaneously, by combining outlier label distributions, spurious signals caused by parsing errors or missing outputs are eliminated.

[0097] When generating parameter adjustment instructions, a preset optimization strategy is used as the source of specification, mapping the "evidence-action-constraint" triple to executable records. Evidence comes from stratified comparisons and marginal contribution estimates of a comprehensive performance index set; actions involve new values ​​or narrowing of value ranges for specific fields; constraints include contextual constraints, upper limits on example proportions, mutually exclusive template lists, and compliant phrase whitelists. A grid fine-tuning strategy can be used to adjust the example quantity parameter step-by-step within adjacent levels; a difficult-example-first strategy can be used to switch the example selection strategy parameter to semantic nearest neighbor or difficult-sample skew; a style rewriting strategy can be used to switch to more easily interpretable template groups in the language style parameter; furthermore, candidate rearrangement can be performed on the trigger phrase parameter, placing high-hit phrases first while retaining the rest for fallback. Each parameter adjustment instruction is bound to an effective scope, which can be applied to the entire input or limited to a specific stratified bucket to avoid performance regression for non-target populations.

[0098] When executing instructions and generating an updated set of prompt word parameter configurations, the length of entries to be changed is first verified. The sum of the input length, example length, and phrase length is compared with the contextual carrying capacity limit. Entries exceeding the limit are either rejected or automatically pruned. Conflict detection is then performed on mutually exclusive relationships, such as tone conflicts between the language style template and the trigger phrase, or sample source conflicts between the example quantity parameter and the example selection strategy parameter. Conflicts are automatically resolved or marked as suspended according to the priority table. Entries that pass verification are written to a new set, while a read-only snapshot of the original set is retained. The two sets are linked by a version list. To ensure reproducibility, adjustment details are written for each change, including parameter configuration identifier, field key, old value, new value, trigger evidence hash, execution timestamp, and executor identifier.

[0099] Version registration occurs immediately after the set is written. The registration action generates a new version identifier, establishes a reference from the current version to the parent version, and simultaneously stores a performance metric snapshot. The performance metric snapshot consists of two parts: a reference snapshot before the change and an immediate snapshot taken after the change on small-sample replay or rapid validation. Both are aligned on the same sampling to assess short-term effects. Registration simultaneously writes the weight matrix version, metric library version, and benchmark set version, forming a consistent label across components. If subsequent validation shows performance degradation exceeding a threshold, a rollback to the parent version can be performed within a constant time using the version list. To support auditing and accountability, the registration record retains a serialized snapshot of the instruction set and the generator version, enabling external systems to replay the same adjustments.

[0100] This embodiment uses a comprehensive set of performance indicators as evidence to complete the location, instruction generation, change implementation, and version registration. Parameter updates shift from experience-driven to data-driven, and the scope and effective boundaries of changes can be strictly defined. False positives caused by abnormal samples and noise are suppressed by stratified controls and robust statistics. Through dual recording of adjustment details and performance indicator snapshots, any modification can be compared before and after on consistent sampling. Rollback and reproduction have clear paths, resulting in shorter diagnostic cycles, lower costs of repeated trial and error, and more stable cross-scenario migration performance.

[0101] In one embodiment, step S60 above includes: S601, after the prompt word parameter configuration set is updated, obtain the comprehensive performance index set of the current iteration and the comprehensive performance index set of the previous iteration; S602, determine the relative rate of change between the set-level comprehensive performance score of the current iteration's comprehensive performance index set and the set-level comprehensive performance score of the previous iteration's comprehensive performance index set; S603, determine whether the relative rate of change is less than a preset convergence threshold, and determine whether the current iteration number has reached a preset maximum iteration number threshold. S604, if the relative rate of change is less than the convergence threshold or the current iteration number reaches the maximum iteration number threshold, then the termination condition is determined to be met; otherwise, the termination condition is determined not to be met.

[0102] In this embodiment, after the prompt word parameter configuration set is updated, the comprehensive performance index set needs to be obtained and aligned in a versioned manner. The comprehensive performance index set of the current iteration is located through the summary table of the latest evaluation batch, and the comprehensive performance index set of the previous iteration is located through the parent version identifier; both are required to carry the index library version, weight version, benchmark test set version, and sampling snapshot identifier to ensure consistency of measurement caliber. To avoid comparison bias caused by sampling differences, the same subset is first selected from the two evaluation results for alignment and comparison according to the sampling snapshot identifier; if the corresponding snapshot was not generated in the historical batch, the alignment subset is generated according to the intersection of the record index, and the coverage and missing list are output at the same time. Missing samples are included in the statistics as placeholders and are marked with anomalies to ensure that the comparison is completed within a traceable range.

[0103] The set-level comprehensive performance score is determined based on the aggregation of sample-level comprehensive performance scores. The aggregator performs stratified calculations within the aligned subset by task type, text length, and domain label to obtain the central tendency, and synthesizes these into a single value using preset stratification weights, forming the set-level comprehensive performance score for the current iteration and the set-level comprehensive performance score for the previous iteration. To suppress misjudgments caused by occasional fluctuations, a smoother can be applied to the sample-level comprehensive performance scores before aggregation. The smoother performs gentle de-extreme value removal and local weighting within the stratification, without changing the ranking relationship, and the smoother version is recorded in the metadata for reproducibility.

[0104] The relative rate of change is calculated based on the ratio of the difference between two aggregate-level performance scores to the benchmark score. Before calculation, the benchmark score is pruned to a lower bound to avoid amplification effects caused by extremely small denominators. The difference is calculated by subtracting the absolute value of the previous score from the current score, using the pruned benchmark score as a scaling factor to obtain the dimensionless relative rate of change. The entire calculation process, along with the aligned subsets, hierarchical weights, and smoother versions, is written to a decision log. The log includes timestamps, participating batches, and hash summaries to ensure that any decision can be fully replayed.

[0105] The termination condition determination relies on two signals: the relative rate of change and the number of iterations. The convergence threshold is stored as a configuration item in the control plane, and the maximum number of iterations threshold is also recorded as a configuration item, both with version identifiers and effective times. The determination process first compares the relative rate of change with the convergence threshold to obtain a Boolean signal indicating whether convergence has occurred; then it compares the current number of iterations with the maximum number of iterations threshold to obtain a Boolean signal indicating whether the upper limit has been reached; the two are combined logically by OR to obtain the final determination result. To improve robustness, a continuous determination strategy can be enabled, i.e., outputting a result that meets the termination condition only when the convergence relationship is met in multiple consecutive comparisons; alternatively, a window determination strategy can be enabled, using relative rate of change quantiles within a sliding window instead of single-point values ​​to reduce the impact of single-batch anomalies. After the determination is completed, the result and all metadata involved in the determination are written to the state storage, triggering subsequent routing. If the termination condition is met, target configuration and report output are performed; otherwise, the process returns to the execution channel.

[0106] For example, the data standardization module automatically extracts the test corpus (input text) and its corresponding expected standard output from the user-uploaded Excel file, and constructs a benchmark test set accordingly. The input data adopts a standardized Excel spreadsheet format (containing two columns: "Input" and "Expected Output"), ensuring a uniform data source format and facilitating subsequent automatic batch processing and performance evaluation benchmark construction. This module's design guarantees the standardization of input data, providing a reliable starting point for subsequent Prompt tuning.

[0107] The Prompt parameterization generation module employs heuristic rules to automatically fine-tune and iteratively optimize the Prompt structure. This module explores Prompt configurations across multiple parameter dimensions, including the number of examples (from few to many), example selection strategies (e.g., random selection, prioritizing error-prone or representative cases), language style (detailed description or concise expression), and prompting words (e.g., adding directives like "be sure to be accurate" or "please analyze rigorously" to the Prompt). The module's built-in heuristic strategy dynamically adjusts these parameters based on feedback from the previous round of optimization: if increasing the number of examples significantly improves performance metrics, the next round continues to increase examples; if the improvement slows down or single-call time becomes too high after the number of examples increases to a certain extent, the example selection strategy is automatically switched to improve efficiency; simultaneously, the system appropriately adjusts the language style and prompting word settings based on metric changes, gradually approaching the optimal Prompt configuration combination. This rule-guided search process gradually narrows the exploration space of Prompt parameters, quickly locating effective Prompt configurations.

[0108] Parameter dimensions: Number of examples: small → medium → large; Example selection strategy: Random → Prioritize incorrect cases → Typical representativeness; Word choice and language style: detailed description → concise description; Adjustments to magic awakening words (trigger words): such as "Be sure to be accurate" and "Please analyze rigorously".

[0109] Example of a heuristic rule: If the increase in the number of examples in the previous round resulted in a significant improvement in the metrics, then the number should be increased moderately in the next round. If the number of examples increases to a certain extent and the metric growth becomes slow or takes too long, the example selection strategy will be automatically switched. The system automatically adjusts its language style and wake word usage based on changes in metrics, gradually approaching the optimal prompt.

[0110] The large language model automatic invocation module is responsible for batch and concurrently calling specified large language model APIs to obtain model inference results, and for timeout control of each call. Through an asynchronous parallel request mechanism, this module can collect batch output data at high speed, avoiding the delay of a single request from blocking the entire tuning process. At the same time, timeout control ensures that a single model call will not be suspended indefinitely, meeting the user's requirements for the timeliness of the tuning process.

[0111] The performance metrics evaluation and automatic feedback module automatically calculates the performance metrics for Prompt based on the model output and determines the optimization strategy for the next round accordingly. This module defines a multi-dimensional evaluation metric system, including exact matching accuracy for classification tasks and fuzzy matching accuracy for generation tasks (based on the cosine similarity between the output and the standard answer), and constructs a performance metric objective function that comprehensively considers multiple objectives.

[0112] Indicator definition and calculation formula: Exact match accuracy (classification task):

[0113] Where N is the total number of evaluation samples (from the benchmark test set); Let be the expected output label (true value) of the i-th sample; I( is the predicted label given by the model for the i-th sample; The indicator function takes the value 1 when the prediction is exactly the same as the true value, and 0 otherwise. The value range is: Accuracy∈[0,1]. It is applicable to both multi-class and binary classification. If there is a "rejection / empty prediction", it is usually included in the denominator as a mismatch.

[0114] Fuzzy matching accuracy (summarized from the inductive generation task, using Cosine similarity):

[0115] in, The vector representation of the expected output (such as TF-IDF vector, sentence vector, bag-of-words vector, etc.) has a dimension of n; This represents the vector representation of the model output, with the same dimension as y; n represents the vector dimension; its value range is: [ 1,1]. When performing text similarity calculations, negative values ​​are often truncated or linearly mapped to [0,1], for example (cos+1) / 2. If the norm of any vector is zero, a very small amount ε needs to be added to avoid the denominator being zero.

[0116] Comprehensive performance objective function (multi-objective optimization):

[0117] Where Accuracy is the exact match accuracy; F1 is the F1 score, which is usually 1. The diameter can be specified as micro / macro; α represents the average processing latency or average call time (average time per sample); α, β, γ represent non-negative weights, reflecting the balance between accuracy and recall / precision and time cost constraints, and need to be fixed and versioned under the same evaluation criteria.

[0118] Building upon this foundation, the module employs Bayesian optimization to automatically optimize the Prompt parameter configuration. Specifically, the system models the relationship between historical Prompt parameters and performance metrics using a Gaussian process, and utilizes the Expected Improvement (EI) criterion to determine the most promising parameter combination for the next round.

[0119] Formula expression:

[0120] Here, Prompt represents a prompt word parameter configuration (which can consist of the number of examples, example selection strategy, language style, trigger phrase, etc.); Score(Prompt) represents the overall score of this configuration in the current evaluation. This represents the best composite score ever observed. EI represents the expected improvement, measuring how much further the score can be improved beyond the current best. The calculation process involves using a Gaussian process to build a surrogate model for the score(Prompt), obtaining the mean μ(Prompt) and standard deviation σ(Prompt). Under the Gaussian assumption, EI has a closed-form solution.

[0121] in It is the standard normal distribution function. Its density function. Choose the configuration that maximizes EI as... .

[0122] The Prompt optimization report and knowledge accumulation module automatically generates historical reports for each round of optimization, including details of the adjustments made in each round of Prompt, trend analysis of performance metrics (such as accuracy, F1 score, and response time), and the final determined optimal Prompt solution. This module facilitates the traceability of results and the accumulation of knowledge during the optimization process. Enterprises can use this to build an internal Prompt optimization experience library, thereby reducing repetitive Prompt design work in subsequent similar tasks.

[0123] Report elements: parameter changes for each round, curves for each indicator (Accuracy / F1 / semantic similarity / latency / cost), optimal solution, failure analysis and risk warnings; automatic export to PDF / HTML / Excel. This embodiment calculates the overall performance score at the set level under the premise of version consistency, sampling alignment, and hierarchical aggregation. It uses a dual threshold of dimensionless relative change rate and iteration number for joint determination. The termination condition is reproducible, auditable, and insensitive to noise. False convergence and excessive iteration are suppressed simultaneously. The optimization loop can stop in time when the performance stabilizes or the resource limit is reached, thereby reducing unnecessary computational consumption and shortening the overall convergence time, providing a reliable trigger point for subsequent target configuration output and report archiving.

[0124] In one embodiment, a benchmark-driven prompt word tuning device is provided, which corresponds one-to-one with the benchmark-driven prompt word tuning method in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the benchmark-driven prompt word optimization device of the present invention. The modules include a data parsing module 10, a parameter configuration generation module 20, a model calling module 30, a performance comparison module 40, a parameter update module 50, a termination judgment module 60, and a result output module 70. Detailed descriptions of each functional module are as follows: Data parsing module 10 is used to parse input data, extract input text and corresponding expected output, and generate benchmark test set; Parameter configuration generation module 20 is used to generate a set of prompt word parameter configurations based on the benchmark test set and according to multiple parameter dimensions; The model invocation module 30 is used to call the language model interface to process the input text according to the prompt word parameter configuration set and the benchmark test set, and obtain the model output set. The performance comparison module 40 is used to compare the model output set with the expected output in the benchmark test set to generate a comprehensive performance index set. The parameter update module 50 is used to adjust and update the prompt word parameter configuration set according to the comprehensive performance index set; The termination judgment module 60 is used to determine whether the termination condition is met after the prompt word parameter configuration set is updated; The result output module 70 is used to return the set of generated prompt word parameter configurations if the termination condition is not met, and to perform model invocation, output comparison and parameter update operations in sequence; if the termination condition is met, it outputs the target prompt word configuration and optimization report.

[0125] Specific limitations regarding the benchmark-driven prompt tuning device can be found in the aforementioned limitations on the benchmark-driven prompt tuning method, and will not be repeated here. Each module in the aforementioned benchmark-driven prompt tuning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0126] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements server-side functions or steps of a benchmark-driven prompt word tuning method.

[0127] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a benchmark-driven prompt word tuning method.

[0128] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Parse the input data, extract the input text and the corresponding expected output, and generate a benchmark test set; Based on the benchmark test set, a set of prompt word parameter configurations is generated according to multiple parameter dimensions; Based on the prompt word parameter configuration set and the benchmark test set, the language model interface is called to process the input text and obtain the model output set. The model output set is compared with the expected output in the benchmark test set to generate a comprehensive performance index set. The prompt word parameter configuration set is adjusted and updated based on the comprehensive performance index set. After the set of prompt word parameter configurations is updated, determine whether the termination condition is met; If the termination condition is not met, the generated prompt word parameter configuration set is returned, and the model call, output comparison, and parameter update operations are executed in sequence; if the termination condition is met, the target prompt word configuration and optimization report are output.

[0129] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Parse the input data, extract the input text and the corresponding expected output, and generate a benchmark test set; Based on the benchmark test set, a set of prompt word parameter configurations is generated according to multiple parameter dimensions; Based on the prompt word parameter configuration set and the benchmark test set, the language model interface is called to process the input text and obtain the model output set. The model output set is compared with the expected output in the benchmark test set to generate a comprehensive performance index set. The prompt word parameter configuration set is adjusted and updated based on the comprehensive performance index set. After the set of prompt word parameter configurations is updated, determine whether the termination condition is met; If the termination condition is not met, the generated prompt word parameter configuration set is returned, and the model call, output comparison, and parameter update operations are executed in sequence; if the termination condition is met, the target prompt word configuration and optimization report are output.

[0130] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0131] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0132] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0133] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of this invention, and not to limit them; although this invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this invention, and should all be included within the protection scope of this invention. The user personal information involved in the embodiments of this application is all authorized (knowing and agreeing) by the relevant parties or fully authorized by all parties, and the executing entity can obtain it through various public, legal and compliant means. The collection, storage, use, processing, transmission, provision and disclosure of the information, data and signals involved all comply with the relevant laws and regulations of the relevant countries and regions, and do not violate public order and good morals.

Claims

1. A benchmark-driven method for optimizing prompt words, characterized in that, Includes the following steps: Parse the input data, extract the input text and the corresponding expected output, and generate a benchmark test set; Based on the benchmark test set, a set of prompt word parameter configurations is generated according to multiple parameter dimensions; Based on the prompt word parameter configuration set and the benchmark test set, the language model interface is called to process the input text and obtain the model output set. The model output set is compared with the expected output in the benchmark test set to generate a comprehensive performance index set. The prompt word parameter configuration set is adjusted and updated based on the comprehensive performance index set. After the set of prompt word parameter configurations is updated, determine whether the termination condition is met; If the termination condition is not met, the generated prompt word parameter configuration set is returned, and the model call, output comparison, and parameter update operations are executed in sequence; if the termination condition is met, the target prompt word configuration and optimization report are output.

2. The benchmark-driven prompt word optimization method as described in claim 1, characterized in that, Parse the input data, extract the input text and the corresponding expected output, and generate a benchmark test set, including: Receive a structured data file containing input text fields and expected output fields; The structured data file format is parsed to identify and extract the input text sequence and the corresponding expected output sequence; Verify whether the field mapping relationship between the input text sequence and the expected output sequence conforms to the preset format constraints, and obtain the input text sequence and the expected output sequence after verification. The input text sequence that has passed the verification is preprocessed to obtain a preprocessed input text sequence; Based on the task type, the expected output sequence after successful verification is classified and labeled to construct a set of standard answers with classification labels; A benchmark test set is generated based on the preprocessed input text sequence and the set of standard answers.

3. The benchmark-driven prompt word tuning method as described in claim 1, characterized in that, Based on the benchmark test set, a set of prompt word parameter configurations is generated according to multiple parameter dimensions, including: Based on the task types included in the benchmark test set, initialize the prompt word parameter configuration framework; The exploration scope is set for the example quantity parameter, example selection strategy parameter, language style parameter, and trigger phrase parameter in the prompt word parameter configuration framework, respectively. Based on the exploration scope, a set of prompt word parameter configurations containing multiple parameter combinations is generated; Associate the set of prompt word parameter configurations with the benchmark test set.

4. The benchmark-driven prompt word tuning method as described in claim 1, characterized in that, Based on the provided prompt word parameter configuration set and the benchmark test set, the language model interface is invoked to process the input text, resulting in a model output set, including: Each parameter configuration in the prompt word parameter configuration set is combined with each input text in the benchmark test set to generate a corresponding prompt word instance; Configure runtime parameters for the language model interface call, including temperature parameters, maximum generation length parameters, and kernel sampling parameters; The language model interface is called in batches in an asynchronous and concurrent manner to process the prompt word instances, and timeout control and error retry mechanism are executed during the call process; The response returned by the language model interface is parsed, and the model output results are extracted and organized from the response according to the output format requirements defined in the prompt word parameter configuration set to generate the model output set.

5. The benchmark-driven prompt word tuning method as described in claim 1, characterized in that, The model output set is compared with the expected output in the benchmark test set to generate a comprehensive performance index set, including: Based on the task type corresponding to the benchmark test set, select the task matching analysis dimension and content quality analysis dimension from the preset indicator library; Based on the task fit analysis dimensions, determine the task fit index value of each output in the model output set and the corresponding expected output in the benchmark test set; Based on the content quality analysis dimensions, determine the content quality index value for each output in the model output set; All task fit index values ​​and content quality index values ​​are normalized. Based on the preset weights of each analysis dimension, the normalized task fit index values ​​and content quality index values ​​are weighted and summed to generate a comprehensive performance score for each model output. The overall performance scores of all models are aggregated to form a set of overall performance indicators.

6. The benchmark-driven prompt word tuning method as described in claim 1, characterized in that, The prompt word parameter configuration set is adjusted and updated based on the comprehensive performance index set, including: Based on the comprehensive performance index set, analyze the performance of each configuration in the prompt word parameter configuration set, and identify the parameter dimensions that need to be optimized; Based on the parameter dimensions that need to be optimized, parameter adjustment instructions are generated according to a preset optimization strategy; According to the parameter adjustment instruction, the corresponding parameters in the prompt word parameter configuration set are modified to generate an updated prompt word parameter configuration set; The updated set of prompt word parameter configurations is version registered, and adjustment details and corresponding performance metric snapshots are recorded.

7. The benchmark-driven prompt word tuning method as described in claim 1, characterized in that, After the prompt word parameter configuration set is updated, determine whether the termination condition is met, including: After the prompt word parameter configuration set is updated, obtain the comprehensive performance index set of the current iteration and the comprehensive performance index set of the previous iteration; Determine the relative rate of change between the set-level comprehensive performance score of the current iteration's comprehensive performance index set and the set-level comprehensive performance score of the previous iteration's comprehensive performance index set; Determine whether the relative rate of change is less than a preset convergence threshold, and determine whether the current iteration number has reached a preset maximum iteration number threshold; If the relative rate of change is less than the convergence threshold or the current iteration count reaches the maximum iteration count threshold, then the termination condition is determined to be met; otherwise, the termination condition is determined not to be met.

8. A benchmark-driven prompt word tuning device, characterized in that, The benchmark-driven cue word tuning device includes: The data parsing module is used to parse input data, extract input text and corresponding expected output, and generate a benchmark test set. The parameter configuration generation module is used to generate a set of prompt word parameter configurations based on the benchmark test set and according to multiple parameter dimensions. The model invocation module is used to call the language model interface to process the input text based on the prompt word parameter configuration set and the benchmark test set, and obtain the model output set. The performance comparison module is used to compare the model output set with the expected output in the benchmark test set to generate a comprehensive performance index set. The parameter update module is used to adjust and update the prompt word parameter configuration set according to the comprehensive performance index set; The termination judgment module is used to determine whether the termination condition is met after the prompt word parameter configuration set is updated; The results output module is used to return the set of generated prompt word parameter configurations if the termination condition is not met, and to perform model invocation, output comparison and parameter update operations in sequence; if the termination condition is met, it outputs the target prompt word configuration and optimization report.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a benchmark-driven prompt word tuning program stored in the memory and executable on the processor. When executed by the processor, the benchmark-driven prompt word tuning program implements the steps of the benchmark-driven prompt word tuning method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a benchmark test driver prompt word tuning program, which, when executed by a processor, implements the steps of the benchmark test driver prompt word tuning method as described in any one of claims 1-7.

Images