Human-machine collaborative generation method, device and medium for task readiness assessment

Abstract

This application relates to the field of data processing technology and discloses a human-computer collaborative generation method, device, and medium for task readiness assessment. The method includes: obtaining structured description information of the target task and filling it into a preset prompt template to obtain model input; providing the model input to a language model to generate an assessment result including judgment values ​​for each preset assessment dimension and self-assessment confidence level; saving the result as an inactive draft after verification; determining a set of dimensions to be reviewed based on dimension category and confidence level, including unreviewed mandatory review dimensions and low-confidence dimensions; if the set is not empty, receiving the review result and removing it; if the set is empty, receiving a confirmation instruction to switch to an active state and obtaining an active assessment result, which is used to generate a readiness level to indicate the human-computer collaboration mode. This invention, through the constraint of draft state isolation and the requirement that the set to be reviewed must be cleared before it becomes effective, suppresses the indiscriminate adoption of model assessment errors, thereby determining the corresponding human-computer collaboration mode according to the task content.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a human-computer collaborative generation method, device, and medium for task readiness assessment. Background Technology

[0002] Before assigning tasks to AI agents, a multi-dimensional evaluation is typically required to determine suitability for the task and the appropriate human-machine collaboration method. With the advancement of large language models, automatically generating evaluation results across multiple dimensions using these models and determining the human-machine collaboration method accordingly has become a key technology for improving evaluation efficiency and consistency. A typical approach in this field involves inputting the task's structured information into a language model, which then outputs an evaluation result encompassing multiple dimensions. This result is either directly used as the final conclusion or, after human verification, serves as the basis for determining the human-machine collaboration method. This approach leads to the following problems: the reliability of the model's output results is low in dimensions involving compliance judgments and ethical reviews, as well as in dimensions where task input information is insufficient. Because the existing solution treats the model's output evaluation results as a whole as the object of effectiveness or overall confirmation, erroneous judgments generated by the model in its high uncertainty dimension can directly enter the effective state and affect downstream decisions. At the same time, the manual confirmation process, due to the lack of identification and guidance for specific high-risk dimensions, is prone to degenerating into an overall approval of the model's output results, failing to form a preventable intervention for specific high-risk dimensions at the data flow level.

[0003] Therefore, the technical challenge lies in how to improve the security and reliability of evaluation conclusions and reduce the impact of model output errors in certain dimensions on the final evaluation conclusions during the process of automatically generating evaluation results using language models. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a human-computer collaborative generation method, device and medium for task readiness assessment, which is used to improve the security and reliability of assessment conclusions and reduce the impact of model output errors in some dimensions on the final assessment conclusions during the process of automatically generating assessment results using language models.

[0005] According to one aspect of the present invention, a human-computer collaborative generation method for task readiness assessment is provided, the method comprising: Obtain structured description information of the target task, wherein the structured description information includes at least the input information, processing logic information, output information and constraint information of the target task; The structured description information is filled into a preset prompt template to obtain the model input; the prompt template contains evaluation constraints, which are used to indicate at least the preset evaluation dimensions that the language model needs to output and the value type of each evaluation dimension; The model input is provided to the language model so that the language model can generate an evaluation result corresponding to the target task under the constraints of the evaluation constraints. The evaluation result includes the judgment value of each preset evaluation dimension and the corresponding self-evaluation confidence level. The preset evaluation dimensions include at least a first type of dimension belonging to the preset mandatory review category and a second type of dimension quantified based on the evaluation indicators. The evaluation results are validated for data validity. Once the validation is successful, the evaluation results are saved as an inactive draft. Based on the category of each assessment dimension and the self-assessment confidence level corresponding to each assessment dimension, a set of dimensions to be reviewed is determined. The set of dimensions to be reviewed includes the first type of dimensions that have not been marked as reviewed and the second type of dimensions whose self-assessment confidence level is lower than a preset confidence threshold. If the set of dimensions to be reviewed is not empty, receive the manual review result of the judgment value of the target dimension in the set of dimensions to be reviewed; in response to the received manual review result, remove the target dimension from the set of dimensions to be reviewed. If the set of dimensions to be reviewed is empty, a manual confirmation instruction is received for the inactive draft, and in response to the manual confirmation instruction, the inactive draft is converted into an active state to obtain an effectiveness evaluation result. The effectiveness evaluation result is used to generate the readiness level of the target task, and the readiness level is used to indicate the human-machine collaboration mode in which the target task is executed by the intelligent agent.

[0006] Further, the step of filling the structured description information into a preset prompt template to obtain model input includes: Extract the input information, processing logic information, output information, and constraint information of the target task from the structured description information to generate task description text; The task description text is filled into the preset prompt template at the position corresponding to the task description to obtain the model input, which includes the task description text and the evaluation constraints contained in the preset prompt template. The evaluation constraints also include instructing the language model output to conform to the format requirements of a preset data structure.

[0007] Furthermore, the step of performing data validity verification on the evaluation results, and saving the evaluation results as an ineffective draft after the verification passes, includes: The original text returned by the language model is subjected to structured parsing to extract at least one candidate data object that conforms to a preset data structure; Perform pattern verification on each candidate data object, the pattern verification including dimensional integrity verification and value validity verification; If there is at least one candidate data object that passes the pattern verification, the first candidate data object that passes the pattern verification is selected as the evaluation result that passes the verification, and the evaluation result is saved as the ineffective draft. If no candidate data object passes the pattern verification, the data is rejected and the language model is triggered to regenerate the evaluation result.

[0008] Further, the first type of dimension includes a qualification determination dimension for determining whether the target task is allowed to be executed by the intelligent agent; the step of receiving the manual review result of the determination value of the target dimension in the set of dimensions to be reviewed includes: Receive the reference information provided by the user for the qualification determination dimension, as well as the confirmation or modification of the determination value for the qualification determination dimension; Upon receiving the reference information, the qualification determination dimension is removed from the set of dimensions to be reviewed.

[0009] Furthermore, after converting the ineffective draft into an effective state, the method further includes: For each evaluation dimension of the evaluation results, source traceability information is recorded. The source traceability information includes at least the suggested value of each evaluation dimension output by the language model, the self-assessed confidence level, whether each evaluation dimension has been manually reviewed, whether the judgment value has been manually modified, the identifier of the language model, and the generation timestamp. A source tag is attached to the evaluation result, and the source tag takes a first tag value, which is used to identify that the evaluation result is model-generated-user-confirmed data; The record set containing the evaluation results is split according to the source tag, and the records with the first tag value and the records with the second tag value are stored in different data sets respectively. The second tag value is used to identify independent human data.

[0010] Furthermore, the preset confidence threshold is determined in the following way: Obtain the distribution characteristics of the self-rated confidence scores of each evaluation dimension output by the language model on the sample task set; The preset quantile of the distribution feature is used as the preset confidence threshold so that the proportion of dimensions with self-rated confidence below the preset confidence threshold falls into the preset tail interval of the distribution feature.

[0011] Furthermore, the step of removing the target dimension from the set of dimensions to be reviewed in response to the received manual review result further includes: In response to the received manual review result, the target dimension is marked as reviewed; When the manual review result includes a modification to the judgment value of the target dimension, the target dimension will also be marked as modified, and the judgment value of the corresponding dimension in the ineffective draft will be updated to the modified judgment value.

[0012] Furthermore, the method also includes: Evaluation results for the same target task are generated by using multiple independent language models. After the evaluation results of the same dimension output by different language models are binned by source, conservative aggregation is performed, and the cross-model divergence degree between models is calculated as an estimate of the relevant error of the corresponding evaluation dimension. When the cross-model divergence of a certain evaluation dimension exceeds a preset divergence threshold, the evaluation dimension is incorporated into the set of dimensions to be reviewed.

[0013] According to another aspect of the present invention, a computer device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction that causes the processor to perform the operation of the human-machine collaborative generation method for task readiness assessment described above.

[0014] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction, the executable instruction causing a computer device to perform the operation of the human-computer collaborative generation method for task readiness assessment described in any of the preceding embodiments.

[0015] This invention establishes an isolation barrier between model output and effective conclusions at the data flow level by saving the verified evaluation results as an ineffective draft, preventing evaluation data that has not been reviewed from entering downstream consumption. By automatically calculating the set of dimensions to be reviewed based on dimension category and self-assessed confidence level, all first-category dimensions belonging to the preset mandatory review category are included, along with second-category dimensions with self-assessed confidence levels below a preset threshold. This achieves automatic identification and precise allocation of review resources for high-risk dimensions involving security and low-confidence dimensions where the model lacks sufficient understanding. By using the clearing of the set of dimensions to be reviewed as the sole trigger condition for the state transition of the evaluation result, a state machine barrier controlled by the set state is constructed in the computer system—during a non-empty set, the evaluation data is locked in a draft state, allowing only dimension-by-dimensional review operations; when the set is empty, a manual confirmation command is enabled to convert the draft to an effective state. This ensures that the model's output on mandatory review dimensions and low-confidence dimensions must be reviewed item by item by humans before flowing to the effective state, thereby preventing erroneous judgments of high-uncertainty dimensions in the data processing flow. The effectiveness assessment results are used to generate a readiness level to indicate the human-machine collaboration method, clarifying the applicable mode for tasks to be performed by intelligent agents.

[0016] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0017] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating the human-computer collaborative generation method for task readiness assessment provided in an embodiment of the present invention is shown. Figure 2 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention is shown. Detailed Implementation

[0018] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0019] Figure 1 A flowchart of a human-computer collaborative generation method for task readiness assessment provided in an embodiment of the present invention is shown. This method is executed by a computer device. The computer device can be a desktop computer, laptop computer, tablet computer, smart terminal, distributed device, etc. Figure 1 As shown, the method includes the following steps: S110. Obtain structured description information of the target task, wherein the structured description information includes at least the input information, processing logic information, output information and constraint information of the target task.

[0020] The target task refers to the specific business task that this solution needs to assess as suitable for execution by the intelligent agent, such as "approving employee expense reports" or "classifying customer complaints." The structured description information is a plural data structure that formally expresses the target task, organizing the key elements of the task in the form of structured fields. The input information describes the data or materials received during task execution, such as expense report data (including invoice photos, expense details, and amounts); the processing logic information describes the sequence of steps and methods or tools used in task execution, such as verifying the authenticity of invoices, checking whether expenses are within budget, and determining whether expense types meet standards; the output information describes the results generated after task execution, such as the approval result (approved / rejected / requires manual review); and the constraint information describes the restrictions that the task execution must comply with, such as requiring director approval for single transactions exceeding 5,000 yuan, and requiring compliance review for gift expenses. In one embodiment, the structured description information can be represented by an octet, including identifier, name, role, input, processing logic (including steps, tools used, cognitive level, determinism), output, constraints, and upstream dependencies. This invention is not limited to this octet; the structured description information can adopt any structured representation that can characterize the task's input, processing logic, output, and constraints, such as JSON objects, XML documents, relational database records, etc. By obtaining multi-dimensional structured information rather than relying solely on the task name or brief description, sufficient evaluation basis is provided for the language model, ensuring that the subsequently generated evaluation dimensions are adequately informed.

[0021] In this embodiment, by acquiring structured descriptive information that includes at least input information, processing logic information, output information, and constraint information, a standardized task representation is provided for subsequent steps. The sufficiency of this structured information is a prerequisite for ensuring the reliability of the evaluation results. When all the information in the task model is complete, the language model can generate evaluation results based on the complete task context, avoiding low confidence or incorrect judgments in certain dimensions due to missing information. Furthermore, this structured descriptive information serves as the sole data source for subsequent prompt template filling operations, ensuring input consistency and traceability throughout the evaluation process.

[0022] S120. Fill the structured description information into a preset prompt template to obtain model input; the prompt template includes evaluation constraints, which are used to indicate at least the preset evaluation dimensions that the language model needs to output and the value type of each evaluation dimension.

[0023] The preset prompt template is a text template pre-built during the system configuration phase. This prompt template is a fixed text framework prepared before the language model is invoked; its purpose is to standardize the input format and output requirements of the language model, rather than being dynamically generated each time it is invoked. The prompt template contains two parts: the first part is the evaluation constraints, declared in the form of system prompt text; the second part is a task description placeholder area, reserved for filling in structured descriptive information for the specific target task.

[0024] The evaluation constraints refer to the instruction text used to constrain and guide the language model to generate evaluation results according to predetermined requirements. Evaluation constraints include at least the following instructions: Dimension declaration, explicitly listing all preset evaluation dimensions that the language model needs to output, and the value type (e.g., Boolean or ordinal), value range (e.g., true / false, or integers from 1 to 5), and value polarity (e.g., a larger value for feasibility readiness indicates greater readiness, and a larger value for risk indicates higher risk); Format requirements, explicitly instructing the language model on the data format to output the evaluation results (e.g., JSON format). By pre-setting the evaluation constraints in the prompt template, each time the language model is invoked, the evaluation constraints, along with the task description, are provided to the model as input. Guided by the evaluation constraints, the model generates structured evaluation results that meet the requirements.

[0025] In one specific implementation, the preset evaluation dimensions include 11 dimensions, belonging to three categories: Eligibility Category (Boolean, belonging to the first category), including E1 Regulatory Permissibility, E2 Ethical-Relational Nature, and E3 Accountability Attributability, each taking a Boolean value; Tolerance Category (Ordinal, belonging to the second category), including T1 Reversibility, T2 Scope of Impact, and T3 Verification Delay, each taking an integer from 1 to 5; and Feasibility Category (Ordinal, belonging to the second category), including F1 Task Clarity, F2 Model Adaptability, F3 Context Adaptability, F4 Tool / Data Adaptability, and F5 Governance Adaptability, each taking an integer from 1 to 5. The evaluation constraints included in the prompt template declare the above 11 dimensions and their value types and ranges, and include format requirements: each dimension must simultaneously output a judgment value, judgment basis text, and a self-assessment confidence value between 0 and 1, returned as a JSON object.

[0026] During the method execution phase, the structured description information obtained in S110 is filled into the placeholder area corresponding to the task description in the prompt template. After filling, the complete content of the prompt template becomes the model input. This model input includes both the task description extracted from the structured description information and the preset evaluation constraints in the prompt template. Thus, the model input is provided to the language model as a whole, and the language model generates an evaluation result that conforms to the preset structure under the constraints of the evaluation constraints.

[0027] It should be noted that, in one implementation, the system only performs the operation of filling the prompt template with the structured description information and triggering the language model call when the task name, input information, processing logic information, output information, and constraint information in the structured description information are all complete; if any key element of the above information is missing, the model call will not be initiated. This prerequisite ensures the integrity of the task signals required for evaluation and avoids the language model generating unreliable evaluation results without sufficient evidence due to insufficient information.

[0028] In this embodiment, by pre-setting a prompt template and filling it with structured descriptive information, the evaluation criteria are injected in one go, and the task information is expressed in a standardized manner. Compared with the existing technology where manual scoring leads to low consistency among evaluators, the evaluation constraints enable the language model to consistently apply the same scoring criteria to produce all dimensions at once, reducing score fluctuations caused by differences in user cognition, improving the throughput of the evaluation, and enabling readiness assessment to cover task sets of enterprise scale.

[0029] S130. The model input is provided to the language model so that the language model generates an evaluation result corresponding to the target task under the constraints of the evaluation constraints. The evaluation result includes the judgment value of each preset evaluation dimension and the corresponding self-evaluation confidence level. The preset evaluation dimensions include at least a first type of dimension belonging to the preset mandatory review category and a second type of dimension quantified based on the evaluation indicators.

[0030] The language model refers to a neural network model capable of receiving natural language text as input and generating natural language text as output. Through pre-training on large-scale text corpora, the language model acquires language understanding and text generation capabilities, enabling it to generate structured output with specific format and content according to the requirements of the given prompt text, under constraints. In one embodiment, the language model is a large language model, i.e., a large language model with billions of parameters trained on massive amounts of diverse data. Such models possess emergent capabilities (such as context learning and thought chain reasoning), and can effectively perform zero-sample or small-sample dimensionality determinations while adhering to the evaluation constraints in the prompt template. Specific instances of the language model can include commercial or open-source large language models; this invention is not limited to specific model providers or model versions. The operation of providing model input to the language model includes sending the model input as prompt text to the language model via an application programming interface (API) and receiving the generated text returned by the language model. "Providing" refers to the action of sending model input data to the language model so that it can be processed.

[0031] The evaluation result refers to the structured output generated by the language model after receiving model input. The evaluation result includes the following contents for each preset evaluation dimension: judgment value, that is, the specific score or judgment result of the language model for each preset evaluation dimension, taking true / false values ​​for Boolean dimensions and integer values ​​within a preset range for ordinal dimensions; judgment basis, that is, a brief explanation or rationale for each judgment value output by the language model in natural language, which is a non-empty string; self-assessed confidence level, that is, the language model's self-assessment of its own confidence in the judgment value of this dimension, which is a value between 0 and 1. The higher the confidence level, the more reliable the model believes the judgment is, and the lower the confidence level, the less reliable the model believes the judgment is.

[0032] The preset mandatory review category refers to a pre-designated category of dimensions. Regardless of their self-assessed confidence level, all dimensions must be reviewed by the user item by item before taking effect. The first category includes eligibility threshold dimensions (such as legal permissibility, ethical-relational nature, and accountability attributability). These dimensions involve determining the safety of task execution; the model's output on these dimensions cannot be fully trusted and must be confirmed or authorized by the user. The second category of dimensions, quantified based on evaluation indicators, includes risk tolerance dimensions (such as reversibility, scope of impact, and verification delay) and feasibility readiness dimensions (such as task clarity, model fit, context fit, tool / data fit, and governance fit). These dimensions are quantified using ordinal values ​​and only require manual review when their self-assessed confidence level is below a preset threshold.

[0033] In this embodiment, by providing model input to a language model and enabling it to generate evaluation results under evaluation constraints, the language model completes the judgment of all preset evaluation dimensions in one go. Compared with purely manual evaluation, the language model can consistently apply the scoring criteria declared in the evaluation constraints, avoiding score fluctuations caused by differences in the cognition of different evaluators. Simultaneously, this step requires the language model to not only output the judgment value for each dimension but also the corresponding self-assessed confidence level. This self-assessed confidence level is a key signal for determining the set of dimensions to be reviewed in subsequent steps—the language model explicitly expresses its uncertainty level in each dimension through the self-assessed confidence level, enabling the system to automatically identify high-risk dimensions requiring user intervention for review based on this signal, thereby achieving an orderly transition from "model generation" to "human review."

[0034] S140. Perform data validity verification on the evaluation results. If the verification is successful, save the evaluation results as an inactive draft.

[0035] The data validity check verifies whether the evaluation results output by the language model conform to a preset pattern in terms of data structure. The preset pattern refers to the list of dimensions, value types, and output format declared in the evaluation constraints of the prompt template in step S120. The data validity check includes dimension integrity check and value validity check: the dimension integrity check checks whether all preset evaluation dimensions are complete; if any dimension is missing, the check fails. The value validity check checks whether the judgment value type and range of each dimension meet the preset requirements—the judgment value for Boolean dimensions must be a Boolean value, the judgment value for ordinal dimensions must be an integer within a preset range, the self-assessment confidence level must be a value between 0 and 1, and the judgment basis must be a non-empty string. If any dimension value is out of bounds or the type is incorrect, the check fails. If the check passes, it indicates that the evaluation results output by the language model are fully compliant in format and can be used as the data basis for subsequent processes; if the check fails, the evaluation results are rejected and the language model can be regenerated.

[0036] The "unsubmitted draft" refers to an intermediate data record saved after successful verification. This unsubmitted draft has a clear status identifier, which is set to the "draft" state. The draft state is achieved by setting a status identifier field in the data record. The change in the status identifier field value (draft → submitted) constitutes a two-state state machine. The state transition conditions of this state machine are determined by the empty / non-empty state of the dimension set to be reviewed. Records in the draft state do not enter the downstream consumption stage—downstream modules must verify the record status before reading the readiness assessment record; records in a non-confirmed state are considered non-existent and not included in the calculation. For example, in the data loading stage of the inter-evaluator consistency measurement, draft state records are skipped directly and not included in the measurement dataset. The introduction of this draft state establishes a barrier at the data flow level, ensuring that the model output will not enter the downstream decision-making or statistical stages in any form before manual review and confirmation.

[0037] In this embodiment, by validating the evaluation results, it is ensured that the evaluation results entering subsequent processes fully conform to the preset pattern in terms of data structure, avoiding process anomalies caused by format errors or missing dimensions. Simultaneously, the validated evaluation results are saved as an ineffective draft rather than being directly implemented, establishing a barrier between model output and effective conclusions at the data flow level. This draft-state mechanism solves the technical problem in existing technologies where "model output is directly adopted as effective conclusions, lacking means to block unverified data"—the model's output in its high-uncertainty dimensions remains in a draft state until it is verified and confirmed by subsequent steps, preventing downstream modules from consuming it, thus eliminating the indiscriminate inflow of erroneous judgments at the state machine level.

[0038] S150. Based on the category of each assessment dimension and the self-assessment confidence level corresponding to each assessment dimension, determine the set of dimensions to be reviewed. The set of dimensions to be reviewed includes the first type of dimensions that have not been marked as reviewed and the second type of dimensions whose self-assessment confidence level is lower than a preset confidence threshold.

[0039] The set of dimensions to be reviewed refers to the collection of evaluation dimensions that need to be reviewed by the user dimension by dimension in the current step. Its calculation is achieved by iterating through the category labels and self-assessed confidence scores of each evaluation dimension and performing pure function operations. This operation does not rely on the user's subjective judgment but is automatically filtered based on a preset dimension category configuration table and confidence thresholds. This set is automatically calculated by the system based on dimension category conditions and self-assessed confidence scores. The dimension category conditions are: dimensions belonging to the first category of the preset mandatory review category (i.e., qualification gate dimensions E1, E2, and E3), regardless of their self-assessed confidence scores, are all included in the set of dimensions to be reviewed. The reason why the first category of dimensions is not exempt from the confidence threshold is that these dimensions involve the safety judgment of task execution—errors in regulatory permissibility judgment may lead to the incorrect execution of non-compliant tasks, errors in ethical review judgment may lead to ethical risks, and errors in accountability judgment may lead to unclear responsibility attribution—the consequences of model misjudgments in these dimensions are serious. Therefore, each item must be confirmed and authorized by the user, and the decision to review cannot be based solely on the model's self-assessed confidence score. The self-assessed confidence level is determined as follows: Dimensions belonging to the second category (i.e., tolerance gate dimensions T1-T3 and feasibility gate dimensions F1-F5) with a self-assessed confidence level lower than a pre-set confidence threshold are included in the set of dimensions to be reviewed. This condition utilizes the language model's self-assessed confidence level signal to automatically filter out dimensions that the model itself considers insufficiently understood, thus achieving precise allocation of review resources. Dimensions already marked as reviewed (regardless of the criteria used to include them and complete the review process) are automatically excluded and not included in the set again.

[0040] The preset confidence threshold is a configurable empirical parameter. In one embodiment, this threshold is set to 0.70. The threshold is calibrated as follows: The self-assessment confidence scores of the language model output on the sample task set are collected, and their empirical distribution is obtained; the threshold is set to a preset quantile (e.g., the 10th percentile) of the empirical distribution, so that the proportion of dimensions with self-assessment confidence scores below this threshold falls into a preset tail interval of the empirical distribution. In this embodiment, the empirical distribution of 88 self-assessment confidence scores (8 tasks multiplied by 11 dimensions) is: minimum 0.60, 10th percentile 0.70, median 0.80, mean 0.79, maximum 0.95; the threshold is calibrated to 0.70, so that the proportion of low-confidence dimensions subject to mandatory review is approximately 8%, concentrated in dimensions that are difficult for structured task models to determine independently, such as the governance adaptation dimension and the reversibility dimension. This threshold can be recalibrated according to changes in the actual language model used or the evaluation task; this scheme is not limited to a specific value.

[0041] In this embodiment, the set of dimensions to be reviewed is automatically calculated based on two conditions: dimension category and self-assessed confidence level. This achieves efficient allocation of review resources. For the first category of dimensions involved in security determination, an unconditional mandatory review strategy is executed to ensure the security of the evaluation results. For other dimensions, the model's self-assessed confidence level is used as a heuristic gating signal, and only dimensions where the model's understanding is insufficient are reviewed. This ensures the reliability of the evaluation results while avoiding indiscriminate manual review of all dimensions. An empty set of dimensions to be reviewed is a hard condition that enables subsequent confirmation operations, constituting the core of the preventable effectiveness control mechanism.

[0042] S160. If the set of dimensions to be reviewed is not empty, receive the manual review result of the judgment value of the target dimension in the set of dimensions to be reviewed; in response to the received manual review result, remove the target dimension from the set of dimensions to be reviewed.

[0043] When the set of dimensions to be reviewed is not empty, the system is in the review in progress state. In this state, the confirmation entry is disabled, meaning the system does not respond to or allow cross-dimensional confirmation operations for the entire ineffective draft; users are only allowed to perform reviews one by one for the specific dimensions listed in the set. The system displays the list of dimensions to be reviewed to the reviewer, with each dimension showing the model's original suggested value, the basis for judgment, and the self-assessed confidence level, so that the reviewer can make a judgment.

[0044] The manual review result refers to the data submitted by the reviewing user after performing operations on the target dimension. For each dimension to be reviewed, the reviewing user can perform the following operations: Confirm – Check “Reviewed” to indicate acceptance of the judgment value given by the model; the judgment value remains unchanged. Modify – Modify the judgment value of the dimension, and optionally fill in the reason for the modification. When modifying, the system simultaneously marks the dimension as reviewed and as modified. Supplementary Basis – For qualification judgment dimensions in the first category (such as regulatory permission dimensions), the reviewing user must provide the legal or standard references on which they are based, such as filling in “Based on Article 12 of the ‘Enterprise Financial Reimbursement Compliance Management Measures’”, to enhance the traceability of compliance audits. If the legal or standard references on which they are based are missing, the dimension must not be removed from the set of dimensions to be reviewed. After receiving the manual review result, the system removes the target dimension from the set of dimensions to be reviewed, updates the review mark and judgment value of the dimension (if modified), and triggers the recalculation of the set. The review progress dynamically advances with each operation until the set is empty.

[0045] In this embodiment, mandatory item-by-item manual intervention for high-risk dimensions is achieved by receiving the results of manual review for each dimension while the set of dimensions to be reviewed is not empty. This mechanism solves the technical problem in the prior art where "the manual confirmation process degenerates into an overall approval of the model suggestion, failing to prevent unverified results from entering the effective state at the data flow level." During periods when the set is not empty, the system does not provide one-click batch confirmation operations across dimensions; the confirmation entry is disabled. Confirmation is only enabled after the user completes item-by-item review of all dimensions to be reviewed. That is, enabling and disabling the confirmation entry is achieved by the system automatically controlling the user interface operation permissions based on the state of the set of dimensions to be reviewed—when the set is not empty, the system refuses to respond to confirmation requests for the entire evaluation record, only responding to review operation requests for a single dimension; when the set is empty, the system enables the confirmation operation. Providing supporting information for qualification dimension requirements ensures that dimensions involving security and compliance judgments have traceable audit trails, meeting the accountability requirements for human-machine collaboration in regulated scenarios.

[0046] S170. If the set of dimensions to be reviewed is empty, a manual confirmation instruction is received for the ineffective draft, and in response to the manual confirmation instruction, the ineffective draft is converted into an effective state to obtain an effectiveness evaluation result. The effectiveness evaluation result is used to generate the readiness level of the target task, and the readiness level is used to indicate the human-machine collaboration mode in which the target task is executed by the intelligent agent.

[0047] When the set of dimensions to be reviewed is empty, it indicates that all dimensions requiring review have been manually reviewed, and the system enables the confirmation operation. At this time, the user can perform an explicit confirmation operation. In response to this confirmation instruction, the system changes the status of the evaluation record from draft to confirmed and effective, obtaining the effective evaluation result. The effective evaluation result refers to the set of judgment values ​​for each preset evaluation dimension that have been manually reviewed and confirmed dimension by dimension. The judgment value for each dimension may be the original suggested value of the model (if it has not been modified after user confirmation) or the value modified by the user (if the user reassigned the value during the review).

[0048] The effectiveness evaluation results are used to generate the readiness level of the target task. The readiness level is a comprehensive evaluation conclusion obtained by aggregating and mapping the judgment values ​​of each preset evaluation dimension in the effectiveness evaluation results according to preset gate-level aggregation rules. In one implementation, a multi-gate assessment framework is used for aggregation: the eligibility gate logically combines each of the first-category dimensions (eligibility judgment dimensions E1, E2, E3) one by one; if any dimension's judgment value is negative, the system short-circuits to the highest manually retained level. The tolerance gate takes the worst value of each of the risk tolerance dimensions (T1, T2, T3) in the second-category dimensions to obtain the risk level. The feasibility gate takes the minimum value of the feasibility readiness dimensions (F2, F3, F4, F5) in the second-category dimensions to obtain the shortcoming value, and caps it in segments according to the task clarity dimension F1—when F1 is less than or equal to 2, the shortcoming value is capped at 2; when F1 is equal to 3, it is capped at 3—to obtain the feasibility readiness score. The capability gap is defined as a constant minus the feasibility readiness score. Finally, the risk level and capability gap are mapped to the readiness level through a preset decision matrix. This aggregation rule is an externally established framework, and this invention does not claim protection for the aggregation rule itself.

[0049] The readiness level corresponds to a preset human-machine collaboration mode. The human-machine collaboration mode refers to the division of labor between the agent and the user during the actual task execution phase. In one embodiment, the human-machine collaboration modes include: autonomous execution—the task can be completely executed independently by the agent without user intervention; collaborative execution—the task is completed collaboratively by the agent and the user, with the agent performing initial processing, and key steps or final decisions requiring user confirmation; and manual retention—the task is not suitable for execution by the agent and must be completed independently by the user.

[0050] In this embodiment, by setting an empty set of dimensions to be reviewed as a hard condition for enabling the manual confirmation operation, a complete data flow loop from "model generation" to "manual review" to "confirmation and effectiveness" is established. The user can only execute the confirmation operation and make the evaluation result effective if and only after all dimensions requiring review have undergone manual review. This hard enable constraint ensures the security and reliability of the effective evaluation result at the data flow level—the model's outputs in its eligibility and low-confidence dimensions must be reviewed or reassigned by the user dimension-by-dimensionally before entering the effective state, thereby preventing the indiscriminate adoption of model errors in high-uncertainty dimensions. The effective evaluation result is used to generate a readiness level, which provides a quantitative decision basis for the human-machine collaboration method of the task—clarifying whether the task can be executed autonomously by the agent, requires human-machine collaboration, or must be reserved for user execution.

[0051] In one embodiment, the decision matrix is ​​further supplemented with several lower bound rules. For example, when the judgment value of the context adaptation dimension is lower than a preset level, the readiness level is not allowed to be mapped to fully autonomous execution; when the judgment value of the model adaptation dimension is lower than a preset level, the readiness level is at least cooperative execution, etc. These lower bound rules serve as a safety fallback strategy to ensure that even if the overall score is high, the task will not be incorrectly assigned to fully autonomous execution when the readiness of certain key dimensions is insufficient.

[0052] In one embodiment, filling the structured description information into a preset prompt template to obtain model input includes: S111. Extract the input information, processing logic information, output information and constraint information of the target task from the structured description information to generate task description text; S112. Fill the task description text into the position corresponding to the task description in the preset prompt template to obtain the model input. The model input includes the task description text and the evaluation constraints contained in the preset prompt template. S113, wherein the evaluation constraint further includes instructing the language model output to conform to the format requirements of a preset data structure.

[0053] In this embodiment, the task description text is a text paragraph in natural language form, which is the conversion of each field in the structured description information into a text paragraph. The extraction process includes: reading the field values ​​of input information, processing logic information, output information and constraint information from the structured description information, and organizing these field values ​​into a coherent natural language description according to a preset text format. For example, for the task of "Approving Employee Reimbursement Forms", the generated task description text can be in the form of: "Task Name: Approving Employee Reimbursement Forms. Input Information: Reimbursement Form (including invoice photo, expense details, amount). Processing Logic Information: (1) Verify the authenticity of the invoice; (2) Verify whether the expense is within the budget; (3) Determine whether the expense type meets the company's travel standards; (4) Mark the expense exceeding the standard as an exception. Output Information: Approval Result (Pass / Reject / Requires Manual Review). Constraint Information: A single amount exceeding 5,000 yuan requires director approval; expenses involving gifts require compliance review." The location corresponding to the task description in the preset prompt template refers to the placeholder area for task information reserved during the template's construction. This placeholder area is identified by placeholders or specific markers within the template. The filling operation involves writing the generated task description text into this placeholder area, replacing the placeholders. After filling, the complete content of the prompt template becomes the model input. This model input includes both the task description text and the evaluation constraints contained in the preset prompt template. Both parts coexist in the same model input text, eliminating the need for additional concatenation operations after filling. The format requirements refer to the part of the evaluation constraints that instructs the language model on the data format to output the evaluation results, such as "return strictly in JSON format" or "each dimension must output three fields: judgment value, judgment basis, and self-assessment confidence level."

[0054] In this embodiment, by converting structured task information into natural language task description text and filling it into a preset prompt template, the conversion of structured data into text input that the language model can understand is realized. The task description text presents the key elements of the task in natural language, which is convenient for the language model to understand and process. At the same time, the format requirements in the evaluation constraints clearly constrain the language model output to conform to the preset data structure, providing a verification benchmark for the data validity verification in step S140. This allows the verification program to automatically check the model output according to the preset data structure pattern, rather than relying on manual verification one by one.

[0055] In one embodiment, the step of performing data validity verification on the evaluation results, and saving the evaluation results as an ineffective draft after the verification passes, includes: S121. Perform structured parsing on the original text returned by the language model to extract at least one candidate data object that conforms to a preset data structure; S122. Perform pattern verification on each of the candidate data objects, wherein the pattern verification includes dimensional integrity verification and value validity verification; S123. If there is at least one candidate data object that passes the pattern verification, the first candidate data object that passes the pattern verification is selected as the evaluation result that passes the verification, and the evaluation result is saved as the ineffective draft. S124. If there are no candidate data objects that pass the pattern verification, then refuse to save and trigger the language model to regenerate the evaluation results.

[0056] In this embodiment, the original text refers to the complete output text returned by the language model after receiving the model input. Since the language model may attach prose wrappings (e.g., "The following is the evaluation result:") or code fence markers (e.g., starting with "```json" and ending with "```", where "```" is a backtick) in the actual output, the original text is not necessarily a pure structured data format. The structured parsing in step S121 adopts a fault-tolerant extraction strategy, and tries the following methods to extract candidate data objects in turn: (1) extract the text within the code fence, that is, locate the content between "```json" and "```" markers in the original text as JSON candidates; (2) if method (1) fails, extract the longest substring with matching curly braces at the beginning and end as JSON candidates; (3) if method (2) fails, use the entire original text as a JSON parsing candidate. Each candidate is parsed into an object that conforms to the preset data structure, and the one that is successfully parsed is the candidate data object. This fault-tolerant extraction strategy makes the model output containing prose wrappings or code fences still robustly parsable, avoiding the interruption of the entire evaluation process due to minor deviations in the output format.

[0057] Step S122, the pattern validation, is a strict validation phase. The following checks are performed on each candidate data object: Dimensional completeness check—checking whether the candidate data object contains all preset evaluation dimensions (e.g., 11 dimensions E1-E3, T1-T3, F1-F5). If any dimension is missing, the candidate validation fails. Value validity check—checking whether the judgment value type and range of each dimension meet preset requirements (Boolean dimensions must be Boolean values, ordinal dimensions must be integers in the range of 1 to 5), whether the self-assessed confidence level is in the range of 0 to 1, and whether the judgment criterion is a non-empty string. If any dimension value exceeds the limit or the type is incorrect, the candidate validation fails.

[0058] In step S123, if multiple candidates pass pattern validation, the first one that passes completely is selected as the final evaluation result. In step S124, if all candidates fail pattern validation, the pattern error information of the closest candidate is recorded and rejected from the database. At the same time, the language model is triggered to regenerate the evaluation result. The system can re-execute the process of steps S120 to S140, and provide the original model input (or the slightly adjusted model input) to the language model again to generate a new evaluation result.

[0059] In this embodiment, a two-stage "easy entry, strict verification" process is employed—the fault-tolerant parsing stage loosely extracts multiple candidate JSONs, while the strict verification stage performs a complete dimensional integrity check and value validity check on each candidate. This improves robustness to changes in the language model's output format (avoiding direct failure due to output containing prose or code fences) and ensures that the final rigorous verification guarantees the complete compliance of the evaluation results before they are entered into the database. This mechanism safeguards the data input quality of subsequent processes and avoids process interruptions or data contamination caused by format errors.

[0060] In one embodiment, the first type of dimension includes a qualification determination dimension for determining whether the target task is allowed to be executed by an intelligent agent; the step of receiving the manual review result of the determination value of the target dimension in the set of dimensions to be reviewed includes: S131. Receive the reference information provided by the user for the qualification determination dimension, and confirm or modify the determination value of the qualification determination dimension; S132. After receiving the reference information, remove the qualification determination dimension from the set of dimensions to be reviewed.

[0061] In this embodiment, the qualification determination dimension refers to the dimension in the first category used to determine the legality and compliance of task execution. In one embodiment, the qualification determination dimension includes the legality dimension (E1), the ethical-relationship nature dimension (E2), and the accountability attributability dimension (E3). The legality dimension is used to determine whether the execution of the task complies with relevant laws, regulations, and rules, such as whether financial approval complies with the company's reimbursement compliance management regulations; the ethical-relationship nature dimension is used to determine whether the task involves ethical or relationship issues that must be judged by the user; and the accountability attributability dimension is used to determine whether the decision-making at each step in the task execution process can be traced back to the specific responsible person.

[0062] Since the eligibility criteria directly determine whether a task is "permitted" to be executed by the agent, and the output of the language model in such dimensions cannot be fully trusted, this solution requires reviewing users to provide supporting information when reviewing eligibility criteria. This supporting information refers to external regulatory basis supporting the eligibility criteria, such as legal provision numbers, corporate policy clauses, or industry standard names. For regulatory-permissive dimensions, the supporting information includes citations of the relevant laws or standards, such as "based on Article 12 of the 'Enterprise Financial Reimbursement Compliance Management Measures'". After receiving the supporting information from the user, the system confirms that the citation meets preset conditions (e.g., not empty, verifiable) and removes the eligibility criteria dimension from the set of dimensions to be reviewed. If the user does not provide supporting information or the provided citation is empty, the dimension must not be removed from the set of dimensions to be reviewed, thus ensuring that the review of eligibility criteria dimensions has traceable audit trails.

[0063] In this embodiment, by providing supporting information for the eligibility criteria, the dimensions involved in security compliance determination have clear external regulatory basis and traceable audit records. This mechanism ensures the security of the assessment results while meeting the audit and accountability requirements of human-machine collaborative decision-making in regulated scenarios—the assessment records can clearly trace which regulations or standards the reviewing user used to make the confirmation or modification decision for key dimensions such as regulatory permissibility.

[0064] In one embodiment, after converting the ineffective draft to an effective state, the method further includes: S141. Record source traceability information for each evaluation dimension of the evaluation result. The source traceability information includes at least the suggested value of each evaluation dimension output by the language model, the self-assessment confidence level, whether each evaluation dimension has been manually reviewed, whether the judgment value has been manually modified, the identifier of the language model, and the generation timestamp. S142. Add a source tag to the evaluation result. The source tag takes a first tag value, which is used to identify that the evaluation result is model generation-user confirmation data. S143. Based on the source marker, the record set containing the evaluation result is split into different data sets, and the records with the first marker value and the records with the second marker value are stored in different data sets respectively. The second marker value is used to identify independent human data.

[0065] In this embodiment, firstly, for each evaluation dimension in the evaluation results, the system records the following source tracing information: the model suggested value, i.e., the original judgment value of that dimension output by the language model, which is distinguished from the final judgment value after manual review (which may be modified), facilitating the tracing of the source of the evaluation conclusion; the self-assessed confidence level, i.e., the original self-assessment score of the language model for the judgment value of that dimension; whether it has been manually reviewed, i.e., whether the dimension has been reviewed by the user; whether the judgment value has been manually modified, i.e., whether the final judgment value is consistent with the model suggested value, if not, it is recorded as modified; the language model identifier, i.e., the identifier information of the language model that actually returned the evaluation result, which in one embodiment is recorded as a combination of the model name and the snapshot identifier, in the form of "<model name>@<snapshot>", based on the identifier value returned by the model interface; and the generation timestamp, i.e., the time record of the generation of the evaluation result. In one embodiment, the confirmer information is also recorded, taken from the evaluator identifier in the evaluation record. This dimension-by-dimensional audit information provides a complete traceability chain for the evaluation results.

[0066] The source tag is a label field appended to the entire evaluation record, used to distinguish the source nature of the evaluation result. The first tag value is used to identify the evaluation result as model-generated-user-confirmed data, indicating that the evaluation result was first generated by the language model, and then reviewed and confirmed by the user dimension by dimension. In one embodiment, for example, the first tag value can be "llm_assisted" or a string with an equivalent meaning.

[0067] The system then categorizes and stores the record set containing the evaluation results based on the source label: records with a first label value are stored in the first data set, and records with a second label value are stored in the second data set, achieving physical or logical isolation between data from different sources. The second label value is used to identify independent human-generated data, i.e., evaluation data that has not been generated by a language model and is entirely completed independently by the user. Through this splitting process, when performing inter-evaluator consistency measurements or reliability / validity statistical aggregation, the system can filter the record set based on the source label during the data loading stage—only records with the second label value (independent human source) constitute the measurement input set, while model-generated and user-confirmed records with the first label value are not included in the measurement aggregation. Thus, model-generated data and independent human-generated data are processed separately and not mixed during the measurement loading stage, avoiding contamination of the measurement results by model-generated items.

[0068] In this embodiment, by recording source traceability information and adding source tags dimension by dimension, the generation, review, and modification process of each evaluation record has a complete audit record that can be checked. When evaluating reliability / validity, records from different sources are stored separately and filtered based on the source tags, preventing model-generated data from being mixed with independent human data in the same measurement input set, thus avoiding contamination of the measurement results by model-generated items. Using only independent human evaluation records as the measurement input set ensures the authenticity and reliability of the measurement results.

[0069] In one embodiment, the preset confidence threshold is determined in the following manner: S151. Obtain the distribution characteristics of the self-rated confidence of each evaluation dimension output by the language model on the sample task set; S152. Using the preset quantile of the distribution feature as the preset confidence threshold, so that the proportion of dimensions with self-rated confidence below the preset confidence threshold falls into the preset tail interval of the distribution feature.

[0070] In this embodiment, the sample task set refers to a pre-selected set of representative task instances used to calibrate a pre-set confidence threshold. In one embodiment, the sample task set contains eight real tasks, covering business scenarios of different types and complexities. For each task in the sample task set, steps S110 to S130 of this scheme are executed to obtain the self-rated confidence scores for each dimension output by the language model. After collecting the self-rated confidence scores for all dimensions of all tasks, their empirical distribution characteristics are calculated, including statistics such as minimum, quantile, median, mean, and maximum.

[0071] The preset quantile is determined based on the required review ratio. A lower review ratio (i.e., a smaller proportion of low-confidence dimensions included in the review set) indicates higher confidence in the model, but may miss some dimensions that the model actually misclassifies; a higher review ratio indicates more rigorous review, but increases the workload of manual review. Based on empirical distribution data, a quantile is selected as a threshold so that the proportion of dimensions with self-rated confidence below this threshold falls into the preset tail interval of the empirical distribution (e.g., the tail 10% to 20%). In one implementation, for example, an evaluation of eight real tasks is performed based on a commercial large language model (tested code M, the model used is based on the actual identifier value returned by the model interface). A total of 88 self-assessed confidence scores (8 tasks × 11 dimensions) are collected, with the empirical distribution as follows: minimum 0.60, 10th percentile 0.70, median 0.80, mean 0.79, and maximum 0.95. Among them, the dimensions below 0.70 account for about 8%, concentrated in the control and governance adaptation dimension and the reversibility dimension—that is, dimensions that are difficult for the structured task model to determine independently. Accordingly, the preset confidence threshold is set to 0.70, which is near the 10th percentile of the empirical distribution, so that the proportion of low confidence dimensions included in the mandatory review is controlled at about 8%.

[0072] In this embodiment, a pre-set confidence threshold is determined using a calibration method based on empirical distribution quantiles. This ensures that the threshold selection is scientifically based and reproducible, rather than relying on subjective settings. This threshold can be recalibrated according to the actual language model used, the type of assessment task, and the required level of review rigor, providing flexibility and configurability. It should be noted that the self-reported confidence level is a self-reported signal from the model and serves only as a heuristic gating signal to trigger manual review; it does not guarantee the accuracy of the judgment—even in high-confidence dimensions, all dimensions belonging to the first category are still unconditionally subject to mandatory review.

[0073] In one embodiment, the step of removing the target dimension from the set of dimensions to be reviewed in response to the received manual review result further includes: S161. In response to the received manual review result, mark the target dimension as reviewed; S162. When the manual review result includes a modification to the judgment value of the target dimension, the target dimension is also marked as modified, and the judgment value of the corresponding dimension in the ineffective draft is updated to the modified judgment value.

[0074] In this embodiment, firstly, marking the operation as reviewed is a prerequisite for removing the dimension set to be reviewed. After each manual review result is received, the system sets the reviewed flag of the corresponding dimension to true. This flag is read when the dimension set to be reviewed is recalculated later. Dimensions marked as reviewed are automatically excluded and no longer included in the set.

[0075] When a reviewer modifies the judgment value of a target dimension (e.g., changing the judgment value for F5 governance adaptation from 2 to 3), the system sets the modified flag for that dimension to true and updates the judgment value field for that dimension in the draft before it takes effect from the original model recommendation value to the user's modified value. Thereafter, the final judgment value for that dimension is based on the modified value, and both the original model recommendation value and the modified flag are retained in the source traceability information for subsequent audit traceability.

[0076] In this embodiment, by marking the status as reviewed and modified, and synchronously updating the judgment values ​​in the draft before modification, a complete record of the review operation and consistent maintenance of the draft data are achieved. The "reviewed" mark ensures that each dimension only needs to be manually reviewed once, avoiding duplicate operations; the "modified" mark records the user's intervention in the model's suggested values, providing key information for source auditing—by distinguishing between "confirmation only" and "modification" review behaviors, auditors can clearly understand which dimensions' final judgment values ​​come from the model and which come from the user's reassignment. The synchronous update of the judgment values ​​in the draft ensures that the evaluation results that take effect after review reflect the user's final confirmed values, rather than the model's original suggested values.

[0077] In one embodiment, the method further includes: S171. Evaluation results for the same target task are generated by using multiple independent language models. S172. After binning the evaluation results of the same dimension output by different language models according to their sources, perform conservative aggregation and calculate the cross-model divergence degree between models as an estimate of the relevant error of the corresponding evaluation dimension. S173. When the cross-model divergence of a certain evaluation dimension exceeds a preset divergence threshold, the evaluation dimension is incorporated into the set of dimensions to be reviewed.

[0078] In this embodiment, which is an optional implementation, the multiple independent language models refer to language models from different training families or different vendors. Their independence ensures that the output differences of each model can reflect the uncertainty or training bias of the model itself, rather than the different sampling fluctuations of the same model. In one embodiment, two or more commercial large language models from different providers can be selected.

[0079] First, binning by source refers to categorizing and organizing the evaluation results generated by different language models for the same target task according to their model sources. This ensures that for the same preset evaluation dimension, the judgment values ​​from various models are grouped together, facilitating subsequent aggregation and comparison. Conservative aggregation refers to synthesizing the judgment values ​​of different models in the same dimension using a conservative (i.e., less ready) approach: For Boolean-type eligibility dimensions, conjunctive conservative aggregation is used—if any model judges it negative, the aggregation result for that dimension is negative; only if all models judge it positive is the aggregation result positive. For ordinal dimensions, the least ready value for each model in that dimension is taken; that is, for the feasibility readiness dimension, the minimum score of each model is taken; for the risk ordinal dimension, the maximum score of each model is taken, or the median score of each model is taken. Self-assessed confidence is the average of the self-assessed confidence scores of each model in that dimension. The average self-assessed confidence score of each model in that dimension is used as the aggregated confidence value to comprehensively reflect the average assessment of the reliability of the judgments of multiple models in that dimension.

[0080] Cross-model divergence is a quantitative metric that measures the degree of difference in judgment values ​​among different models on the same dimension, and is used to estimate the evaluation error of that dimension. In one implementation, cross-model divergence can be calculated using methods such as dimension-level consistency rate or ordinal weighted consistency measure.

[0081] When the cross-model divergence of a certain dimension exceeds a preset divergence threshold, it indicates that there is significant disagreement among different models regarding the judgment of that dimension, and the reliability of the evaluation results for that dimension is low. In this case, that dimension is added to the set of dimensions to be reviewed as an additional mandatory manual review trigger condition. This means that even if the self-assessed confidence level of that dimension is higher than the preset confidence threshold and it does not belong to the first category of dimensions, it will still be forcibly included in the manual review due to excessive cross-model divergence. The preset divergence threshold can be configured according to the review resources and required reliability level in the actual application scenario. Evaluation data generated by different models are identified and managed separately through a source tag field and are not mixed with single-model data or independent manual data.

[0082] In this embodiment, multiple independent language models are introduced for parallel evaluation, and cross-model divergence is used as an error estimate to supplement the detection of uncertainty in single-model evaluation. The self-assessed confidence of a single model may not fully reflect the actual risk of misjudgment due to model self-assessment bias, while the degree of divergence among multiple models provides an external reliability signal independent of model self-assessment. When the cross-model divergence is high, this dimension is forcibly included in manual review, effectively supplementing the limitations of single-model self-assessment confidence and further improving the reliability of the evaluation results. Reducing the weight of dimensions with significant divergence or forcing manual review ensures that the multi-model approach improves evaluation accuracy while maintaining the precise allocation of manual review resources.

[0083] Figure 2 The diagram shows a structural schematic of a computer device provided in an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computer device.

[0084] like Figure 2 As shown, the computer device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0085] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. Communication interface 404 is used to communicate with other network elements such as clients or other servers. The processor 402 executes program 410, specifically performing the relevant steps in the above-described embodiment of the human-machine collaborative generation method for task readiness assessment.

[0086] Specifically, program 410 may include program code, which includes computer-executable instructions.

[0087] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0088] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0089] Specifically, program 410 can be called by processor 402 to enable computer devices to perform relevant steps in the human-computer collaborative generation method embodiment for task readiness assessment.

[0090] This invention provides a computer-readable storage medium storing at least one executable instruction that, when executed on a computer device, causes the computer device to perform the human-computer collaborative generation method for task readiness assessment in any of the above method embodiments.

[0091] This invention provides a computer program that can be called by a processor to cause a computer device to execute the human-computer collaborative generation method for task readiness assessment in any of the above method embodiments.

[0092] This invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed on a computer, cause the computer to perform the human-computer collaborative generation method for task readiness assessment in any of the above method embodiments.

[0093] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0094] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0095] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0096] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0097] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A human-computer collaborative generation method for task readiness assessment, characterized in that, The method includes: Obtain structured description information of the target task, wherein the structured description information includes at least the input information, processing logic information, output information and constraint information of the target task; The structured description information is filled into a preset prompt template to obtain the model input; the prompt template contains evaluation constraints, which are used to indicate at least the preset evaluation dimensions that the language model needs to output and the value type of each evaluation dimension; The model input is provided to the language model so that the language model can generate an evaluation result corresponding to the target task under the constraints of the evaluation constraints. The evaluation result includes the judgment value of each preset evaluation dimension and the corresponding self-evaluation confidence level. The preset evaluation dimensions include at least a first type of dimension belonging to the preset mandatory review category and a second type of dimension quantified based on the evaluation indicators. The evaluation results are validated for data validity. Once the validation is successful, the evaluation results are saved as an inactive draft. Based on the category of each assessment dimension and the self-assessment confidence level corresponding to each assessment dimension, a set of dimensions to be reviewed is determined. The set of dimensions to be reviewed includes the first type of dimensions that have not been marked as reviewed and the second type of dimensions whose self-assessment confidence level is lower than a preset confidence threshold. If the set of dimensions to be reviewed is not empty, receive the manual review result of the judgment value of the target dimension in the set of dimensions to be reviewed; in response to the received manual review result, remove the target dimension from the set of dimensions to be reviewed. If the set of dimensions to be reviewed is empty, a manual confirmation instruction is received for the inactive draft, and in response to the manual confirmation instruction, the inactive draft is converted into an active state to obtain an effectiveness evaluation result. The effectiveness evaluation result is used to generate the readiness level of the target task, and the readiness level is used to indicate the human-machine collaboration mode in which the target task is executed by the intelligent agent.

2. The method according to claim 1, characterized in that, The step of filling the structured description information into a preset prompt template to obtain model input includes: Extract the input information, processing logic information, output information, and constraint information of the target task from the structured description information to generate task description text; The task description text is filled into the preset prompt template at the position corresponding to the task description to obtain the model input, which includes the task description text and the evaluation constraints contained in the preset prompt template. The evaluation constraints also include instructing the language model output to conform to the format requirements of a preset data structure.

3. The method according to claim 1, characterized in that, The process of validating the evaluation results and saving them as an inactive draft after successful validation includes: The original text returned by the language model is subjected to structured parsing to extract at least one candidate data object that conforms to a preset data structure; Perform pattern verification on each candidate data object, the pattern verification including dimensional integrity verification and value validity verification; If there is at least one candidate data object that passes the pattern verification, the first candidate data object that passes the pattern verification is selected as the evaluation result that passes the verification, and the evaluation result is saved as the ineffective draft. If no candidate data object passes the pattern verification, the data is rejected and the language model is triggered to regenerate the evaluation result.

4. The method according to claim 1, characterized in that, The first type of dimension includes a qualification determination dimension for determining whether the target task is allowed to be executed by the intelligent agent; The step of receiving the manual review result for the judgment value of the target dimension in the set of dimensions to be reviewed includes: Receive the reference information provided by the user for the qualification determination dimension, as well as the confirmation or modification of the determination value for the qualification determination dimension; Upon receiving the reference information, the qualification determination dimension is removed from the set of dimensions to be reviewed.

5. The method according to claim 1, characterized in that, After converting the ineffective draft into an effective state, the method further includes: For each evaluation dimension of the evaluation results, source traceability information is recorded. The source traceability information includes at least the suggested value of each evaluation dimension output by the language model, the self-assessed confidence level, whether each evaluation dimension has been manually reviewed, whether the judgment value has been manually modified, the identifier of the language model, and the generation timestamp. A source tag is attached to the evaluation result, and the source tag takes a first tag value, which is used to identify that the evaluation result is model-generated-user-confirmed data; The record set containing the evaluation results is split according to the source tag, and the records with the first tag value and the records with the second tag value are stored in different data sets respectively. The second tag value is used to identify independent human data.

6. The method according to claim 1, characterized in that, The preset confidence threshold is determined in the following way: Obtain the distribution characteristics of the self-rated confidence scores of each evaluation dimension output by the language model on the sample task set; The preset quantile of the distribution feature is used as the preset confidence threshold so that the proportion of dimensions with self-rated confidence below the preset confidence threshold falls into the preset tail interval of the distribution feature.

7. The method according to claim 1, characterized in that, The step of removing the target dimension from the set of dimensions to be reviewed in response to the received manual review result further includes: In response to the received manual review result, the target dimension is marked as reviewed; When the manual review result includes a modification to the judgment value of the target dimension, the target dimension will also be marked as modified, and the judgment value of the corresponding dimension in the ineffective draft will be updated to the modified judgment value.

8. The method according to claim 1, characterized in that, The method further includes: Evaluation results for the same target task are generated by using multiple independent language models. After the evaluation results of the same dimension output by different language models are binned by source, conservative aggregation is performed, and the cross-model divergence degree between models is calculated as an estimate of the relevant error of the corresponding evaluation dimension. When the cross-model divergence of a certain evaluation dimension exceeds a preset divergence threshold, the evaluation dimension is incorporated into the set of dimensions to be reviewed.

9. A computer device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the human-machine collaborative generation method for task readiness assessment as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on a computer device, causes the computer device to perform the operation of the human-computer collaborative generation method for task readiness assessment as described in any one of claims 1-8.

Images