Evaluation method and system based on hierarchical chain reasoning

Abstract

The invention provides an evaluation method and system based on hierarchical chain reasoning. The method comprises the steps of obtaining target task scheme data; inputting the target task scheme data into a large language model to obtain a target evaluation result, the large language model being obtained based on reward function training in different evaluation scenes; each evaluation subsystem in the reward function is determined and obtained based on orthogonality test; the reasoning scoring process corresponding to each evaluation subsystem is constructed according to the index type in the evaluation scene and chain reasoning; evaluation subsystem scoring constraints are set in the total score calculation process of the reward function, and the evaluation subsystem scoring constraints are used for constraining evaluation subsystem scores output in the reasoning scoring process of the evaluation subsystem. According to the method, the robustness and reliability of the evaluation result can be effectively improved, meanwhile, the award function definition is clear, executable and explainable, and an accurate and effective quantitative evaluation tool is provided for large language model output evaluation and agent training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an evaluation method and system based on hierarchical chain reasoning. BACKGROUND

[0002] In the current development process of large language models (LLM), reward evaluation mechanisms are crucial in model training and optimization. Past research and practice show that the reinforcement learning framework of LLM verifiable rewards is effective in structured reference answer tasks such as mathematical reasoning and coding. It can precisely design reward functions or models based on the characteristics of structured data, efficiently guide model learning, and improve task completion accuracy and efficiency. For example, when solving mathematical operations or code writing problems, the model can determine the output results based on clear rules and reference answers and obtain rewards to achieve self-optimization.

[0003] However, when the scope of research expands to real-world fields such as medicine, psychology, economics, and education, the situation becomes complex and challenging. Tasks in these fields often lack structured and standardized reference answers, and existing reward evaluation mechanisms have many shortcomings. From the perspective of index design, most existing reward functions rely only on a single index or a small number of heuristic indexes, making it difficult to fully consider the completeness, execution loop, and standardization of the output results. For example, in the medical diagnosis and treatment recommendation generation task, only the accuracy of the diagnosis conclusion is used to evaluate the reward, while the completeness of the diagnosis basis, the executable loop of the treatment plan, and the degree of adherence to medical standards are ignored, which undoubtedly makes the reward evaluation biased.

[0004] Therefore, there is an urgent need for an evaluation method and system based on hierarchical chain reasoning to solve the above problems. SUMMARY

[0005] To solve the problems in the prior art, the present application provides an evaluation method and system based on hierarchical chain reasoning.

[0006] The present application provides an evaluation method based on hierarchical chain reasoning, comprising: Obtaining target task scheme data to be evaluated in the current evaluation scenario; inputting the target task scheme data into a large language model corresponding to the evaluation scene to obtain a target evaluation result of the target task scheme data output by the large language model, wherein the large language model is obtained based on a reward function under different evaluation scenes; each evaluation sub-system in the reward function is determined based on orthogonality test; a reasoning score process corresponding to each evaluation sub-system is constructed according to an index type in the evaluation scene and chain reasoning; and an evaluation sub-system score constraint is set in a total score calculation process of the reward function, and the evaluation sub-system score constraint is used to constrain an evaluation sub-system score output by the reasoning score process of the evaluation sub-system.

[0007] According to the application, a layered chain reasoning-based evaluation method is provided, and the reward function includes an evaluation system layer, a chain reasoning score layer and a total score calculation constraint layer, wherein: The evaluation system layer is provided with a plurality of evaluation sub-systems corresponding to evaluation scene types, and a Pearson correlation coefficient between each evaluation sub-system is less than a preset correlation coefficient threshold; The chain reasoning score layer is used to determine a score judgment condition and a score mechanism of each to-be-scored index in the evaluation sub-system based on the reasoning score process corresponding to the evaluation sub-system, and to perform chain reasoning on each to-be-scored index according to the score judgment condition and the score mechanism to obtain an index score result corresponding to each to-be-scored index; The total score calculation constraint layer is used to aggregate and process the index score results of each to-be-scored index in the evaluation sub-system based on a preset index aggregation rule to obtain an evaluation sub-system total score corresponding to each evaluation sub-system, to constrain each evaluation sub-system total score based on a preset evaluation sub-system score veto threshold corresponding to the evaluation sub-system total score, and to sum up the constrained evaluation sub-system total scores to obtain the target evaluation result of the target task scheme data.

[0008] According to the application, a layered chain reasoning-based evaluation method is provided, and the method further includes: constructing a new evaluation sub-system based on evaluation scene evaluation task adjustment information; determining that there is a nonlinear relationship between the new evaluation sub-system and each evaluation sub-system in the current evaluation system layer, and calculating a mutual information value between the new evaluation sub-system and each evaluation sub-system based on mutual information entropy; in a case where the mutual information value is determined to be less than a preset mutual information threshold, adding the new evaluation sub-system to the current evaluation system layer to obtain a new evaluation system layer.

[0009] According to the evaluation method based on hierarchical chain reasoning provided by the present invention, the chain reasoning scoring layer is further used to perform anomaly verification on the condition judgment process corresponding to the target task scheme data in the scoring judgment conditions, and re-execute the condition judgment process with anomalies.

[0010] According to the evaluation method based on hierarchical chain reasoning provided by the present invention, the total score calculation constraint layer is further used to adjust the preset index aggregation rules according to the change information of the evaluation scenario, and to aggregate the index score results based on the adjusted preset index aggregation rules. The total score calculation constraint layer is also used to filter out indicators to be scored that have a score result less than a preset indicator score threshold during the aggregation process.

[0011] According to the evaluation method based on hierarchical chain reasoning provided by the present invention, the preset evaluation subsystem scoring rejection threshold includes a first evaluation subsystem scoring rejection threshold and a second evaluation subsystem scoring rejection threshold, wherein the first evaluation subsystem scoring rejection threshold is greater than the second evaluation subsystem scoring rejection threshold. The constraint on the total score of each evaluation subsystem based on the preset evaluation subsystem score rejection threshold corresponding to the total score of each evaluation subsystem includes: When it is determined that the total score of the evaluation subsystem is greater than or equal to the score rejection threshold of the first evaluation subsystem, the total score of the evaluation subsystem is retained and summed. When it is determined that the total score of the evaluation subsystem is less than the first evaluation subsystem scoring rejection threshold and greater than or equal to the second evaluation subsystem scoring rejection threshold, a risk warning label is marked on the total score of the evaluation subsystem, and the total score of the evaluation subsystem after the risk warning label is marked is summed. When the total score of the evaluation subsystem is determined to be less than the rejection threshold of the second evaluation subsystem, the total score of the evaluation subsystem is reset to zero.

[0012] According to the evaluation method based on hierarchical chain reasoning provided by the present invention, the method further includes: The target task plan data is preprocessed and verified, and based on the preprocessing and verification results, a task plan data anomaly log corresponding to the target task plan data is generated. If the orthogonality test results between the evaluation subsystems do not meet the preset correlation coefficient threshold, the indicators in the evaluation subsystems are adjusted. If an index with an abnormal score is identified in the evaluation subsystem, the consistency of the reasoning and scoring process of each index to be scored in the evaluation subsystem is checked, and the scoring anomaly location information is obtained based on the consistency check result. If data anomalies are found during the total score calculation process, data completion processing is performed on the evaluation subsystem where the data anomalies occurred.

[0013] The present invention also provides an evaluation system based on hierarchical chain reasoning, comprising: The task input module is used to obtain data on the target task scheme to be evaluated in the current evaluation scenario; The evaluation module is used to input the target task scheme data into a large language model corresponding to the evaluation scenario, and obtain the target evaluation result of the target task scheme data output by the large language model. The large language model is trained based on reward functions under different evaluation scenarios; each evaluation subsystem in the reward function is determined based on orthogonality testing; the reasoning scoring process corresponding to each evaluation subsystem is constructed based on the indicator type and chain reasoning in the evaluation scenario; the total score calculation process of the reward function includes evaluation subsystem scoring constraints, which are used to constrain the evaluation subsystem scores output by the reasoning scoring process of the evaluation subsystem.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the evaluation method based on hierarchical chain reasoning as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the evaluation method based on hierarchical chained reasoning as described above.

[0016] The evaluation method and system based on hierarchical chain reasoning provided by this invention, by constructing a hierarchical, multi-dimensional and traceable reward function system, ensures that the index scores have logical causal relationships, effectively improves the robustness and reliability of the evaluation results, and at the same time makes the reward function clearly defined, executable and interpretable, providing a precise and effective quantitative evaluation tool for the evaluation of large language model output and the training of intelligent agents. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating the evaluation method based on hierarchical chain reasoning provided by this invention; Figure 2 This is a diagram illustrating the implementation framework of the orthogonal multidimensional reward function based on hierarchical chain reasoning provided by the present invention. Figure 3 A schematic diagram of the structure of the evaluation system based on hierarchical chain reasoning provided by the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] In complex task domains lacking standard answers (such as medicine, psychology, and economics), current large language model reward evaluation exposes many problems that urgently need to be addressed, specifically in the following aspects: Firstly, existing methods have significant flaws in the correlation of indicators, with frequent redundancy or conflict between indicators. Redundant indicators not only increase computational costs but also cause the scoring results to be overly biased towards specific aspects due to repeated consideration of information in the same dimension, seriously affecting the objectivity of the overall assessment. Indicator conflicts, on the other hand, can cause confusion in the scoring logic, resulting in highly volatile, unstable, and difficult-to-interpret scoring results. Taking a psychological counseling scenario as an example, when evaluating the counseling suggestions given by the evaluation model, if the weights of the two indicators, "emotional resonance of the suggestion" and "rational guidance of the suggestion," are set unreasonably, and there is an overlapping and ambiguous area in their evaluation criteria, the scores for the same counseling suggestion may fluctuate greatly.

[0021] Secondly, the traceability of the scoring process is a major pain point for existing technologies. Due to the lack of a step-by-step traceability mechanism, it is difficult to quickly and accurately pinpoint the root cause of problems once the scoring results are unsatisfactory, which greatly hinders model optimization and improvement. For example, in the field of education, when evaluating the output of student essay grading models, it is impossible to clearly define the specific content and rules upon which each scoring item is based. Subsequent optimization work on the model is like the blind men and the elephant, making it difficult to be targeted and effective.

[0022] Thirdly, the lack of a robust constraint mechanism is also a significant shortcoming of existing reward evaluation methods. In many scenarios, some key violations or seriously substandard schemes may still receive high scores due to the absence of effective constraints, which greatly reduces the effectiveness and reliability of reward evaluation. Taking financial investment strategy evaluation as an example, if there is no rigid constraint mechanism similar to a veto for strategy schemes with significant potential risks or violations of basic investment regulations, the adoption of such schemes could lead to incalculable losses for investors.

[0023] In summary, current LLM reward evaluation technologies have many shortcomings in terms of the comprehensiveness of indicator design, the rationality of indicator correlation, the traceability of the scoring process, and the completeness of hard constraint mechanisms. Therefore, there is an urgent need for an innovative and more adaptable multi-dimensional reward function design method to fill this technological gap and promote the effective application and development of LLM in more complex real-world scenarios.

[0024] Figure 1 The flowchart of the evaluation method based on hierarchical chain reasoning provided by the present invention is shown below. Figure 1 As shown, this invention provides an evaluation method based on hierarchical chain reasoning, comprising: Step 101: Obtain the target task scheme data to be evaluated in the current evaluation scenario.

[0025] In this invention, the data corresponding to the specific target task scheme that needs to be evaluated in the current specific evaluation scenario is first determined. Different evaluation scenarios cover a wide range, such as agent training scenarios, policy evaluation scenarios, scheme optimization scenarios, and multi-stage task scoring scenarios.

[0026] Taking the training scenario of an intelligent agent as an example, the target task plan data may be a series of behavioral records generated by the intelligent agent after performing a series of actions in a specific task environment, including information such as action selection and state changes. In the scenario of financial investment strategy evaluation, the target task plan data may be data such as asset allocation plans and transaction records of a certain investment strategy under different market conditions. This data is the basis for subsequent evaluation and provides raw materials for the entire evaluation process.

[0027] Step 102: Input the target task scheme data into the large language model corresponding to the evaluation scenario to obtain the target evaluation result of the target task scheme data output by the large language model. The large language model is trained based on reward functions under different evaluation scenarios. Each evaluation subsystem in the reward function is determined based on orthogonality testing. The reasoning scoring process corresponding to each evaluation subsystem is constructed based on the indicator type and chain reasoning in the evaluation scenario. The total score calculation process of the reward function is set with evaluation subsystem scoring constraints, which are used to constrain the evaluation subsystem score output by the reasoning scoring process of the evaluation subsystem.

[0028] In this invention, a large language model is used to evaluate target task scheme data and derive results. This large language model is trained based on reward functions under different evaluation scenarios. Specifically, the reward function system of this invention consists of multiple independent and logically orthogonal subsystems. Logical orthogonality means that the subsystems are independent of each other, evaluating the target task scheme from different dimensions and avoiding redundancy and conflict between indicators. For example, when evaluating a complex project scheme, it may be evaluated from orthogonal subsystems such as technical feasibility, economic cost, and social impact.

[0029] In this invention, the indicators within each subsystem are executed step-by-step through a rigorous chain-of-thought (COT) process of "definition → conditional judgment → scoring." First, each indicator is clearly defined. For example, when evaluating the code quality of a software project, the indicator "code comment coverage" is defined as the ratio of the number of lines of comments to the total number of lines of code. Then, conditional judgments are made to determine whether the target task plan data meets the conditions of this indicator based on pre-set standards, such as whether the code comment coverage reaches 30%. Finally, a score is assigned based on the judgment results. If the condition is met, a corresponding score is given; otherwise, different levels of scores are given based on the discrepancy. This chain-of-thought process ensures that each conclusion relies on verifiable evidence from the previous step, resulting in a coherent scoring logic.

[0030] Since each evaluation subsystem is determined based on orthogonality testing, this testing ensures that each subsystem evaluates the target task plan from different and independent perspectives, avoiding redundant evaluation of the same aspect and improving the comprehensiveness and accuracy of the evaluation. For example, when evaluating a marketing plan, different orthogonal subsystems such as market research, target customer positioning, and marketing channel selection can be used for evaluation. These subsystems complement each other, jointly constituting a comprehensive evaluation of the marketing plan.

[0031] Furthermore, the reasoning and scoring processes corresponding to each assessment subsystem are constructed based on the indicator types and chain reasoning within the assessment scenario. Indicators are constructed according to a hierarchical COT logic of "top-down expansion and bottom-up aggregation." High-level indicators consist of several sub-indicators. For example, when assessing a company's overall competitiveness, the high-level indicator "Company Overall Competitiveness" can be composed of sub-indicators such as "Technological Innovation Capability," "Market Expansion Capability," and "Financial Management Capability." After the sub-indicators are scored, they are aggregated from bottom to top to form the high-level indicator score. That is, each sub-indicator is scored first, and then the sub-indicator scores are aggregated according to certain rules to obtain the high-level indicator score. Each indicator score is associated with input evidence, ensuring that the source of the score is clear and verifiable, achieving full-chain traceability of the scoring process. For example, when assessing the "Technological Innovation Capability" sub-indicator, the associated input evidence might be data such as the company's R&D investment amount, number of patents, and frequency of new product launches. Scoring is conducted using this evidence, and the scoring basis can be verified at any time.

[0032] In this invention, the total score calculation of the reward function employs a dual mechanism of "weighted fusion + veto". The total score is calculated as follows: Total Score = (Score of each subsystem × Corresponding Weight) × Veto Gate (triggered, resets to zero / downgrades). Subsystem weights can be allocated based on importance. For example, when evaluating a large engineering project, if engineering quality is crucial to the project's success, the weight of the "Engineering Quality Assessment Subsystem" might be set higher; while less important aspects, such as the weight of the subsystem evaluating the greening of the project site, might be lower.

[0033] Furthermore, the veto gate (i.e., the scoring constraint of the assessment subsystem) sets hard constraints on compliance, key risk control, and other indicators. If a key indicator fails to meet the requirements, the total score will be downgraded or reduced to zero. For example, in the assessment of financial investment strategies, if an investment strategy violates basic investment regulations, even if the strategy performs well in other aspects, its total score may be reduced to zero because the veto gate has been triggered. This reflects the strict control over these key hard constraints and ensures the validity and reliability of the assessment results.

[0034] Through the above steps, the large language model trained based on a specific reward function can comprehensively, accurately, and traceably evaluate the target task scheme data from multiple independent and orthogonal dimensions, and obtain reasonable evaluation results. It is applicable to a variety of evaluation scenarios and has good scalability. Subsystems, indicators, and weights can be customized according to different task requirements. Local adjustments (such as a certain layer or a certain subsystem) can also be made when the data distribution drifts without destroying the overall framework.

[0035] For different evaluation scenarios, this invention illustrates the evaluation method based on hierarchical chain reasoning through the following embodiments: In one embodiment, a bank's risk control department evaluates the risk control strategy of the credit approval intelligent agent by loading three orthogonal subsystems: "Completion Assessment (A), Compliance Assessment (B), and Execution Closed-Loop Assessment (C)". The system automatically verifies and displays that the correlation coefficients of the three subsystems are all <0.3, avoiding duplicate scoring of "risk factor coverage" and "data compliance" (in previous manual assessments, "high data compliance scores masked the lack of risk factors," and the current subsystem orthogonality ensures that each dimension is scored independently, preventing the situation of "a passing total score but missing key dimensions").

[0036] Then, traceability is achieved through chain reasoning: when evaluating a new credit agent strategy, in the scoring stage of the "user data compliance" indicator, the system can view the "three-stage chain evaluation" evidence chain: Evid-B1 clearly states "must comply with relevant legal provisions", Evid-B2 shows "sensitive data detection tool scanned two bank card numbers that were not anonymized", and Evid-B3 records "35 points according to the rules", thus forming the corresponding evidence log, without the need to repeatedly check the original strategy document.

[0037] If, in a certain assessment, the total score of Subsystem B (such as compliance) is 58 points, which is lower than the preset threshold of 60 points, the system will automatically trigger a veto, reducing the final total score to "unqualified". Through the assessment method provided by this invention, the bank's risk control assessment efficiency has been improved by 40%, the rate of missed compliance issues has been reduced to 0, and each assessment report can be directly used as the basis for regulatory inspections, solving the core pain points of "untraceable scoring and difficulty in controlling compliance risks".

[0038] In another embodiment, when a medical technology company was developing a "chronic disease management intelligent agent", it needed to evaluate whether the treatment plan output by the intelligent agent "covered the patient's medical history, medication contraindications, and follow-up plan". However, the previous "direct scoring" model was used, and the "completeness of the plan" was scored by the evaluator based on experience. However, it was impossible to explain "whether the deduction was due to missing medical history or missed follow-up", which made it difficult for the R&D team to optimize. At the same time, the "completeness of the plan" and "medication safety" indicators were often scored repeatedly (e.g., "missing medication contraindications" deducted both completeness and safety scores), resulting in poor stability of the scoring results.

[0039] After adopting the hierarchical chain reasoning-based evaluation method provided by this invention, when evaluating a certain version of chronic disease management program, the system first decomposes "program completeness" (high-level indicator) into three low-level sub-indicators: "medical history coverage", "medication contraindication reminders" and "follow-up plan completeness", with scores of 60, 85 and 55 respectively. This directly identifies "incomplete medical history coverage and missing follow-up plans" as the main problems, and provides feedback to the R&D team to "prioritize the addition of a 5-year medical history collection module for hypertension patients and design an automatic follow-up reminder function".

[0040] Meanwhile, the addition of the "Drug Safety" subsystem to the system, which is orthogonal to the original "Protocol Completeness" subsystem through "mutual information entropy" (mutual information value 0.08 < 0.1), ensures that "omitted contraindications" only deduct points in the "Drug Safety" subsystem and no longer affect the "Completeness" score. This makes the scores of each subsystem independent, and the R&D team can clearly determine "whether it is insufficient completeness or safety risk".

[0041] Furthermore, a four-stage reasoning process of "indicator definition → condition judgment → anomaly verification → scoring mechanism" was adopted. In the scoring of the "medication contraindication reminder" indicator, the "anomaly verification" step called the medical terminology dictionary and found that the system mistakenly identified "aspirin contraindication population (gastric ulcer patients)" as "applicable population". At this time, a re-judgment was automatically triggered, and the score of this indicator was finally corrected from 85 points to 50 points, which improved the scoring accuracy by 60%.

[0042] The evaluation method provided by this invention increases the pass rate of the above-mentioned chronic disease management intelligent agent from 65% to 92%, shortens the R&D iteration cycle by 30%, solves the key problems of "inaccurate scoring and lack of optimization direction", and accelerates the product launch process.

[0043] In another embodiment, during a promotional season, the existing evaluation process of an e-commerce platform's operations department needs to assess whether the strategy of the "intelligent recommendation agent" covers "discount rules, inventory warnings, and user preferences." However, the "average score aggregation" rule used in non-promotional seasons fails during promotional seasons (e.g., "discount rule completeness" is more important than "user preference matching," but it is still calculated based on the average score). At the same time, the evaluation process requires manual switching between "promotional season / non-promotional season" indicators, which is inefficient and often leads to scoring deviations due to "incorrect indicator weight configuration."

[0044] After adopting the evaluation method provided by this invention, a "scenario-based aggregation rule" is set in the system. During promotional seasons, the weight of "completeness of discount rules" is 0.7 and the weight of "user preference matching" is 0.3; in non-promotional seasons, the average score is used. When evaluating a recommendation strategy during a promotional season, the system automatically calculates the total score of the "solution completeness" subsystem according to the promotional season rules: (90 points for completeness of discount rules × 0.7) + (70 points for user preference matching × 0.3) = 84 points, accurately reflecting the core requirement of "prioritizing discount rules," thus improving evaluation efficiency by 80%.

[0045] Furthermore, the portable framework allows for rapid expansion of metrics: for example, before a promotional season, the platform can add a "logistics timeliness reminder" assessment requirement, and then directly add a "logistics adaptability" subsystem to the system, importing two sub-metrics: "logistics timeliness commitment" and "delivery plan for remote areas," without having to reconstruct the overall framework, thus quickly responding to temporary needs during the promotional season.

[0046] In addition, this embodiment sets the rejection threshold for the "Inventory Warning" subsystem to three levels: ≥80 points (no warning), 60-79 points (inventory warning), and <60 points (unqualified). When evaluating a strategy during the pre-sale period of a promotional season, if this subsystem scores 65 points, the system will mark it as "Inventory Warning".

[0047] The evaluation method provided by this invention achieves the scenario adaptation requirement of "rapid response during promotional seasons and stable evaluation during non-promotional seasons", and solves the problems of "low evaluation efficiency and poor adaptability".

[0048] The hierarchical chain reasoning-based evaluation method provided by this invention constructs a hierarchical, multi-dimensional, and traceable reward function system to ensure that the index scores have logical causal relationships, effectively improving the robustness and reliability of the evaluation results. At the same time, it makes the reward function clearly defined, executable, and interpretable, providing a precise and effective quantitative evaluation tool for the evaluation of large language model output and the training of intelligent agents.

[0049] Based on the above embodiments, the reward function includes an evaluation system layer, a chain reasoning scoring layer, and a total score calculation constraint layer, wherein: The evaluation system layer is provided with multiple evaluation subsystems corresponding to the evaluation scenario type, and the Pearson correlation coefficient between each evaluation subsystem is less than a preset correlation coefficient threshold. The chain-based reasoning scoring layer is used to determine the scoring judgment conditions and scoring mechanisms for each indicator to be scored in the evaluation subsystem based on the reasoning scoring process corresponding to the evaluation subsystem; and to perform chain-based reasoning on each indicator to be scored according to the scoring judgment conditions and the scoring mechanism to obtain the indicator score results corresponding to each indicator to be scored. The total score calculation constraint layer is used to aggregate the index scores of each of the indicators to be scored in the evaluation subsystem based on preset index aggregation rules to obtain the total score of each evaluation subsystem; constrain the total score of each evaluation subsystem based on the preset evaluation subsystem scoring rejection threshold corresponding to the total score of each evaluation subsystem; and sum the constrained total scores of the evaluation subsystem to obtain the target evaluation result of the target task scheme data.

[0050] In this invention, the reward function consists of three key layers: the evaluation system layer, the chain reasoning scoring layer, and the total score calculation constraint layer. Each layer has a clear division of labor and cooperates with each other to jointly complete the evaluation of the target task plan data and finally output the target evaluation result.

[0051] Specifically, the evaluation system layer sets up multiple evaluation subsystems based on different evaluation scenario types. These evaluation subsystems are built for different evaluation dimensions and focuses under specific evaluation scenarios. For example, when evaluating a software project, a "functional integrity evaluation subsystem," a "performance efficiency evaluation subsystem," and a "security evaluation subsystem" may be set up to comprehensively evaluate the software project from multiple core aspects.

[0052] In this invention, the orthogonality of the various evaluation subsystems is verified through statistical tests, such as the Pearson correlation coefficient. For example, the correlation coefficient between any two subsystems is required to be less than a preset correlation coefficient threshold (this threshold can be set to 0.3 to meet the requirements of low correlation / low mutual information). This effectively avoids information redundancy between different subsystems, ensuring that each subsystem can independently and effectively evaluate the target task solution from a specific perspective, thus improving the accuracy and comprehensiveness of the evaluation. For example, if the correlation coefficient between the "functional integrity evaluation subsystem" and the "performance efficiency evaluation subsystem" is too high, it indicates that these two subsystems are considering certain factors repeatedly during the evaluation process. By controlling the correlation coefficient, this problem can be avoided, making the evaluation more scientific and reasonable.

[0053] The chain-reasoning scoring layer serves as the core processing layer, corresponding one-to-one with each subsystem in the evaluation system layer. Each evaluation subsystem incorporates a built-in "multi-stage chain-review" logic, namely "indicator definition → condition judgment → scoring mechanism." Specifically, firstly, each indicator to be scored is clearly defined. For example, in the "functional integrity evaluation subsystem," the indicator "functional coverage" is defined as the ratio of the actual functions implemented in the software to the functions specified in the requirements specification. Then, condition judgments are made based on pre-set standards to determine whether the target task solution data meets the conditions of this indicator, such as whether the functional coverage reaches 80%. Finally, a scoring mechanism is determined based on the judgment results; if the condition is met, a corresponding score is given; otherwise, different levels of scores are given based on the gap.

[0054] In the specific reasoning process of this invention, the high-level indicators of each subsystem are first broken down. Taking the high-level indicator "completeness" of subsystem A as an example, it is broken down into low-level sub-indicators, such as "element coverage," "logical coherence," and "information completeness." After these low-level sub-indicators are scored, they are aggregated into the high-level indicator score according to preset aggregation rules (such as weighted summation, average score, etc.). For example, if the score for "element coverage" is 8 points, "logical coherence" is 7 points, and "information completeness" is 9 points, then if the average score rule is used, the score for the high-level indicator "completeness" is (8+7+9)÷3≈8 points. Furthermore, each step generates traceable evidence records, such as recording the scoring basis and calculation process for each low-level sub-indicator, facilitating subsequent review and auditing, and ensuring the transparency and reliability of the scoring process. Simultaneously, the various sub-indicator systems are executed sequentially, and the sub-indicators within an indicator are also calculated sequentially, forming a chain-like reasoning scoring model to ensure the coherence and rigor of the scoring logic.

[0055] The total score calculation constraint layer comprises two core modules. The first is the "weighted fusion module," which assigns weights to each subsystem according to its importance. In this invention, subsystem A is assigned a weight of 0.4, subsystem B a weight of 0.3, and subsystem C a weight of 0.3. This approach highlights the impact of important subsystems in the overall evaluation, making the evaluation results more consistent with reality. For example, if subsystem A represents the most critical core indicator evaluation of the target task plan, then assigning a higher weight ensures that the core indicator plays a dominant role in the total score calculation. The data is first processed by the weighted fusion module to calculate the basic total score, that is, by weighting and summing the scores of each subsystem according to their corresponding weights to obtain the total score of the evaluated subsystem.

[0056] In this invention, the total score of the evaluation subsystem is input into a "veto module," which presets a veto threshold for key subsystems. For example, the threshold for subsystem B is set to 60 points. If the total score of a key subsystem is lower than its preset veto threshold, a constraint mechanism is triggered to adjust the total score, such as downgrading or resetting it to zero. Finally, the total scores of the evaluated subsystems after constraint are summed to obtain the target evaluation result of the target task solution data. For example, if the weighted and fused base total score is 80 points, but the total score of subsystem B is only 50 points, which is lower than its veto threshold of 60 points, after triggering the veto mechanism, the final target evaluation result may be adjusted according to specific rules, such as being reduced to 60 points or lower, to ensure that the evaluation result strictly follows the requirements of key indicators and guarantees the effectiveness and reliability of the evaluation.

[0057] This invention, through the collaborative work of the above evaluation system layer, chain reasoning scoring layer, and total score calculation constraint layer, enables the reward function to comprehensively, accurately, and scientifically evaluate the target task plan data, providing strong support for decision-making.

[0058] Based on the above embodiments, the method further includes: Based on the assessment scenario and task adjustment information, a new assessment subsystem is constructed; Based on the determination that there is a non-linear relationship between the newly added evaluation subsystem and each evaluation subsystem in the current evaluation system layer, the mutual information value between the newly added evaluation subsystem and each evaluation subsystem is calculated based on mutual information entropy. If the mutual information value is determined to be less than the preset mutual information threshold, the newly added evaluation subsystem is added to the current evaluation system layer to obtain a new evaluation system layer.

[0059] In this invention, different evaluation scenarios and tasks in actual evaluation work have different characteristics and requirements. For example, in financial scenarios, in addition to conventional evaluation dimensions, there may be a greater focus on aspects such as the execution efficiency of the strategy. Therefore, it is necessary to determine whether it is necessary to add an evaluation subsystem based on the information fed back from the specific evaluation scenario and task. For instance, in financial scenarios, if it is found that the existing evaluation subsystem cannot comprehensively measure the efficiency of the strategy, adjustments need to be made to meet this need.

[0060] Taking a financial scenario as an example, a "strategy efficiency assessment" subsystem is constructed to evaluate strategy efficiency. Within this subsystem, specific sub-indicators are further refined, such as "response time" (measuring the time interval from strategy triggering to execution, reflecting the strategy's rapid response capability) and "resource utilization" (reflecting the degree of system resource consumption during strategy execution, such as memory and CPU utilization). This construction allows the newly added assessment subsystem to quantitatively evaluate strategy efficiency from multiple perspectives.

[0061] In this invention, before incorporating a new evaluation subsystem into the existing evaluation system layer, it is necessary to first determine the relationship between the new evaluation subsystem and the existing evaluation subsystems. In complex scenarios, many indicators are not simply linearly correlated, but rather exhibit non-linear relationships. For example, in a financial scenario, the relationship between "resource utilization rate" (an indicator in the new subsystem) and "compliance" (a possible indicator in the existing subsystem) may not be linearly correlated; there may be some complex, non-intuitive correlation pattern. Determining this non-linear relationship provides a basis for subsequently selecting an appropriate orthogonality verification method.

[0062] Since the Pearson correlation coefficient is primarily used to measure linear correlations and is not effective in validating nonlinear correlations, this invention employs mutual information entropy to calculate the mutual information value between the newly added evaluation subsystem and each existing evaluation subsystem. Mutual information entropy measures the degree of interdependence between two variables, regardless of whether this dependency is linear or nonlinear. Taking "resource utilization" and "compliance" as examples, a specific calculation method (involving probability distribution and other information) yields their mutual information value, which reflects the amount of information shared between these two indicators.

[0063] In this invention, to ensure good orthogonality between the newly added evaluation subsystem and the various subsystems in the existing evaluation system layer, and to avoid information redundancy, a mutual information threshold needs to be pre-set. For example, the mutual information value is required to be less than 0.1. This threshold can be determined based on actual evaluation needs and tolerance for information redundancy. The lower the threshold, the stricter the requirement for orthogonality, and the better the information redundancy control effect.

[0064] Furthermore, when the calculated mutual information values ​​between the newly added evaluation subsystem and each existing evaluation subsystem are all less than the preset mutual information threshold (0.1), it indicates that the information sharing between the newly added subsystem and the existing subsystems is very low, exhibiting good orthogonality and not introducing excessive redundant information. In this case, the newly added "strategy efficiency evaluation" subsystem can be added to the current evaluation system layer, forming a new evaluation system layer together with the original evaluation subsystems. For example, if the original evaluation system layer had 3 orthogonal subsystems, adding the new subsystem would create 4 orthogonal subsystems, enabling a more comprehensive and accurate evaluation of evaluation tasks in financial scenarios.

[0065] Through the above steps, this invention can flexibly adjust the evaluation system layer under complex and ever-changing evaluation scenarios and task requirements, ensure the orthogonality between evaluation subsystems, and improve the quality and effectiveness of the evaluation system.

[0066] Based on the above embodiments, the chain-based reasoning scoring layer is also used to perform anomaly verification on the condition judgment process corresponding to the target task scheme data in the scoring judgment conditions, and re-execute the condition judgment process with anomalies.

[0067] In this invention, the chain-based reasoning scoring layer employs a three-stage reasoning logic: "indicator definition → condition judgment → scoring mechanism." First, the evaluation indicators are clearly defined, specifying the concrete content and standards measured by each indicator. For example, in a medical agent evaluation scenario, the indicator "blood pressure judgment accuracy" is defined, clearly indicating that it measures the accuracy of the agent's judgment of a patient's blood pressure. Next, the condition judgment stage proceeds. Based on pre-set rules and standards, the target task plan data (such as the agent's judgment of the patient's blood pressure) is judged to see if it meets the corresponding conditions, such as whether the agent correctly judges the patient's blood pressure as "hypertension," "normal blood pressure," or "hypotension." Finally, based on the condition judgment results, a corresponding score is given according to a predetermined scoring mechanism, such as 1 point for a correct judgment and 0 points for an incorrect judgment.

[0068] Furthermore, this invention adds an "anomaly verification" step to the original three-stage logic, forming a four-stage reasoning logic of "indicator definition → condition judgment → anomaly verification → scoring mechanism". The purpose of this new step is to further improve the accuracy of scoring in high-reliability scenarios (such as medical scenarios).

[0069] For example, in a medical agent evaluation scenario, the "anomaly detection" step might utilize a professional resource like a medical terminology dictionary. This dictionary contains accurate and standardized medical terms along with their definitions and explanations. By comparing the terms used in the conditional judgment process with the dictionary, it can determine if any misclassifications exist. For instance, if the agent misclassifies a patient's "hypotension" as "hypotension" during the conditional judgment process, this obvious terminology error can be detected during the anomaly detection phase by comparing it with the medical terminology dictionary.

[0070] If an anomaly is detected during the anomaly verification process (such as misjudgment of terminology), the system will automatically trigger a re-judgment mechanism. This means that the judgment process for the anomaly is executed again, and the target task solution data is re-evaluated based on the correct standards and rules. For example, if "hypertension" is misjudged as "hypotension," the system will re-examine the agent's judgment of the patient's blood pressure, eliminate erroneous interference, and provide an accurate judgment.

[0071] In this invention, the chain-reasoning scoring layer, in addition to its original functions, has been given the responsibility of anomaly verification in the conditional judgment process. In the medical agent evaluation scenario, this involves using resources such as a medical terminology dictionary to check for anomalies in the agent's conditional judgment process for outcomes such as patient blood pressure. When anomalies are detected in the conditional judgment process (such as misjudgment of terminology affecting accuracy), the chain-reasoning scoring layer automatically re-executes the conditional judgment process to ensure that the final score is based on accurate and reliable conditions. This improves the accuracy of the entire scoring process and meets the stringent requirements for accuracy in high-reliability scenarios (such as medical settings).

[0072] Based on the above embodiments, the total score calculation constraint layer is also used to adjust the preset index aggregation rules according to the change information of the evaluation scenario, and to aggregate the index score results based on the adjusted preset index aggregation rules. The total score calculation constraint layer is also used to filter out indicators to be scored that have a score result less than a preset indicator score threshold during the aggregation process.

[0073] In this invention, the importance of various indicators varies across different evaluation scenarios. For example, in e-commerce agent evaluation, the requirements for agent capabilities differ between promotional seasons and non-promotional seasons, rendering the originally fixed indicator aggregation rules inadequate for practical needs. Therefore, the total score calculation constraint layer needs to adjust the preset indicator aggregation rules according to the changing information of the evaluation scenario to adapt to the differences in indicator importance under different scenarios, making the evaluation results more accurate and reasonable.

[0074] Taking subsystem A (such as completeness) in e-commerce intelligent agent evaluation as an example, during promotional seasons, consumers pay more attention to the complete presentation of promotional information. At this time, the "element coverage" indicator is more critical for measuring the agent's completeness. Therefore, the weight of "element coverage" is set to 0.7, and the weight of "logical coherence" is set to 0.3. This weight allocation highlights the importance of "element coverage" during promotional seasons. In non-promotional seasons, there are no special requirements for the completeness of promotional information, and the importance of the two indicators is relatively balanced. The score of subsystem A is calculated using an average score. This method of flexibly adjusting the indicator aggregation rules according to changes in the scenario can better reflect the actual performance of the intelligent agent in different scenarios.

[0075] After adjusting the preset indicator aggregation rules, the total score calculation constraint layer will aggregate the indicator scores based on the adjusted rules. For example, in the e-commerce intelligent agent evaluation mentioned above, according to the different rules for promotional seasons or non-promotional seasons, the scores of each sub-indicator (element coverage, logical coherence) are calculated according to their respective weights to obtain the final score of subsystem A, thereby completing the aggregation processing of the entire evaluation indicator system and obtaining a comprehensive and accurate evaluation result.

[0076] During the evaluation process, some indicators scoring too low may indicate that the agent has serious deficiencies in that area and can no longer meet the basic requirements. In order to more efficiently identify these unqualified situations and avoid low-scoring indicators from having an unreasonable impact on the overall evaluation results, the total score calculation constraint layer of this invention also sets a "threshold filtering" rule. By setting a preset indicator score threshold, indicators with scores below the threshold are filtered out.

[0077] Taking e-commerce intelligent agent evaluation as an example, assuming a preset score threshold of 50 points, when calculating the score of a subsystem (e.g., subsystem B), if any sub-indicator (e.g., information accuracy) scores less than 50 points, the total score calculation constraint layer will directly determine that the sub-indicator is unqualified, and this low-scoring sub-indicator will not be included in the calculation when aggregating the scores of subsystem B. In other words, the impact of this unqualified sub-indicator on the score of subsystem B is no longer considered; the qualification of subsystem B is determined directly based on the scores of other qualified sub-indicators. This filtering operation can quickly identify intelligent agent evaluation aspects with obvious problems, improving evaluation efficiency and accuracy.

[0078] Through the filtering operations described above, the total score calculation constraint layer can focus more on the aspects where the agent performs well, while clearly identifying serious problems. The final evaluation results can more intuitively reflect whether the agent meets the basic requirements in different scenarios, as well as the strengths and weaknesses of each capability, providing a more targeted basis for subsequent improvements and optimizations.

[0079] The total score calculation constraint layer in this invention can better adapt to the needs of different evaluation scenarios and improve the accuracy and reliability of evaluation results by adjusting the preset index aggregation rules according to changes in the evaluation scenario and filtering low-scoring indicators during aggregation processing.

[0080] Based on the above embodiments, the preset evaluation subsystem scoring rejection threshold includes a first evaluation subsystem scoring rejection threshold and a second evaluation subsystem scoring rejection threshold, wherein the first evaluation subsystem scoring rejection threshold is greater than the second evaluation subsystem scoring rejection threshold. The constraint on the total score of each evaluation subsystem based on the preset evaluation subsystem score rejection threshold corresponding to the total score of each evaluation subsystem includes: When it is determined that the total score of the evaluation subsystem is greater than or equal to the score rejection threshold of the first evaluation subsystem, the total score of the evaluation subsystem is retained and summed. When it is determined that the total score of the evaluation subsystem is less than the first evaluation subsystem scoring rejection threshold and greater than or equal to the second evaluation subsystem scoring rejection threshold, a risk warning label is marked on the total score of the evaluation subsystem, and the total score of the evaluation subsystem after the risk warning label is marked is summed. When the total score of the evaluation subsystem is determined to be less than the rejection threshold of the second evaluation subsystem, the total score of the evaluation subsystem is reset to zero.

[0081] In this invention, the preset evaluation subsystem scoring rejection threshold includes a first evaluation subsystem scoring rejection threshold and a second evaluation subsystem scoring rejection threshold, with the first evaluation subsystem scoring rejection threshold being greater than the second evaluation subsystem scoring rejection threshold. This setting constructs a multi-gradient constraint framework, dividing different processing intervals for evaluation results at different levels. For example, in the e-commerce intelligent agent evaluation scenario, if subsystem B (compliance) is taken as the evaluation object, the first rejection threshold can be set to 80 points, and the second rejection threshold to 60 points, thus forming three intervals: score ≥ 80 points, 60 points ≤ score < 80 points, and score < 60 points, with each interval corresponding to a different processing method.

[0082] When the total score of an evaluation subsystem reaches or exceeds the first evaluation subsystem's score rejection threshold, it indicates that the subsystem performed well in the evaluation and there are no obvious risk issues. In this case, the total score of the evaluation subsystem is retained and included in subsequent summation processing. For example, in the above evaluation of e-commerce intelligent agent subsystem B (compliance), if the score is ≥80 points, no constraints are triggered, and this score is directly calculated together with the scores of other qualified subsystems to reflect that the intelligent agent has achieved a high level of compliance without affecting the overall evaluation result.

[0083] If the total score of the evaluated subsystem falls between the first and second rejection thresholds, it indicates that the subsystem has some problems, but has not yet reached the level of serious non-compliance. In this case, a risk warning label is marked on the total score of the evaluated subsystem to remind relevant personnel of the potential risks associated with the subsystem. After labeling, the score is still included in the summation process to calculate the final total score. For example, if the score of e-commerce intelligent agent subsystem B is between 60 and 80, a "compliance warning" is marked to inform the evaluators that the intelligent agent has some areas that need attention and improvement in terms of compliance, but the overall evaluation can still continue.

[0084] When the total score of an evaluation subsystem falls below the rejection threshold of the second evaluation subsystem, it indicates a serious problem with that subsystem, potentially having a significant negative impact on the overall evaluation results. In this case, the total score of that subsystem is reset to zero, meaning its score is not included in the final total score. For example, if the score of e-commerce intelligent agent subsystem B is less than 60 points, the total score is downgraded to "unqualified" and directly reset to zero, indicating that the intelligent agent has serious deficiencies in compliance and cannot meet basic requirements, thus severely affecting the entire evaluation result.

[0085] In addition to the aforementioned constraint method based on the total score of a single evaluation subsystem, this invention also supports "joint veto of multiple subsystems." For example, setting "subsystem B < 60 points and subsystem C < 70 points triggers total score to zero" means that when multiple subsystems simultaneously score below the corresponding threshold, even if a single subsystem does not meet the criteria for individual veto, the severity of the problems in multiple subsystems will be considered, triggering the severe measure of resetting the total score to zero. This joint veto mechanism can more comprehensively evaluate the agent's performance in multiple key aspects, avoiding inaccurate overall evaluation results due to the neglect of problems in some subsystems, and further meeting diverse risk control needs.

[0086] This invention, through its multi-gradient constraints and multi-subsystem joint veto approach, enables more refined and rigorous management and control of the evaluation subsystem, ensuring that the evaluation results truly and accurately reflect the actual situation of the evaluated object, and effectively enhancing the risk prevention and control capabilities of the evaluation system.

[0087] Based on the above embodiments, the method further includes: The target task plan data is preprocessed and verified, and based on the preprocessing and verification results, a task plan data anomaly log corresponding to the target task plan data is generated. If the orthogonality test results between the evaluation subsystems do not meet the preset correlation coefficient threshold, the indicators in the evaluation subsystems are adjusted. If an index with an abnormal score is identified in the evaluation subsystem, the consistency of the reasoning and scoring process of each index to be scored in the evaluation subsystem is checked, and the scoring anomaly location information is obtained based on the consistency check result. If data anomalies are found during the total score calculation process, data completion processing is performed on the evaluation subsystem where the data anomalies occurred.

[0088] In this invention, during the task input phase, if the uploaded target task plan data (such as strategy plan P) has problems, preprocessing verification will be triggered. Data problems mainly include two types: format errors and missing core information. For example, garbled text is a format error; not specifying the type of credit product to which the strategy applies is a missing core information.

[0089] If the system detects the above-mentioned problems after performing preprocessing verification at the task input layer, it will output corresponding prompts, such as "Input scheme format error / core information missing, please complete and re-upload," and terminate the subsequent process to prevent erroneous data from entering subsequent processing stages and affecting the accuracy of the evaluation results.

[0090] After the operator corrects the problem according to the prompts, a new task ID needs to be generated to avoid confusion with abnormal task data. The system will generate a task plan data anomaly log, recording the problems with the input data and the handling process, which facilitates subsequent traceability and review, and also helps to analyze the patterns of data anomalies, providing a reference for optimizing the input process and improving data quality.

[0091] Orthogonality testing is used to ensure that the various evaluation subsystems are independent of each other, avoid indicator redundancy, and guarantee the accuracy and effectiveness of the evaluation results. When the orthogonality test results show that the correlation coefficient between two subsystems is greater than or equal to a preset threshold (e.g., 0.3), it indicates that there is a certain degree of correlation between the two subsystems, which may indicate indicator duplication or mutual influence. At this point, the "redundancy optimization" branch is triggered. For example, in a financial scenario, the correlation coefficient between the newly added "strategy efficiency assessment" and "execution closed-loop assessment" reaches 0.35, exceeding the preset threshold.

[0092] At this point, the system will automatically adjust the metrics of one of the subsystems. Adjustments can be made by deleting duplicate or highly correlated metrics, or by adding independent metrics. For example, the "Result Feedback Time" metric can be deleted from "Strategy Efficiency Evaluation," and a "CPU Utilization" metric can be added. The correlation coefficient can then be recalculated until it meets the requirement of a correlation coefficient < 0.3. If the requirement is still not met after three adjustments, the system will output "Subsystem configuration abnormal, please redesign the metrics," prompting the operator to perform further optimization of the subsystem.

[0093] In the chain-reasoning scoring process, if a contradiction arises in the chain-reasoning of a certain indicator, it indicates an anomaly in the scoring. For example, if subsystem A scores 100 points for "risk factor coverage" but 0 points for "rule logical completeness" without reasonable evidence, this constitutes an anomaly. In this case, the system triggers a "consistency check" at the chain-reasoning scoring layer, automatically backtracking all evidence records for that subsystem. By analyzing the logical relationships between the evidence, the system identifies the cause of the contradiction, such as the possibility that a risk factor was mistakenly omitted during condition judgment, leading to an unreasonable score.

[0094] If the system can automatically correct the problem (such as adding missed risk factors), the scoring will be re-executed; if it cannot be automatically corrected, the system will output "Scoring abnormal, please manually review the evidence record". The operator can then review the evidence record in a targeted manner based on the location information, find the problem and handle it, and continue the process after manual confirmation.

[0095] During the weighted fusion calculation of the total score, if data anomalies occur, such as a missing score for a certain subsystem, data completion processing will be triggered. Data anomalies may affect the accuracy and validity of the total score calculation, so timely handling is necessary. The system triggers the "data completion" branch at the total score calculation and constraint layer. Different completion strategies are adopted depending on whether the subsystem with missing scores is a critical subsystem. If the subsystem with missing scores is a non-critical subsystem (such as subsystem C), the historical average score of that subsystem is used for completion, which can minimize the impact on the total score calculation result. If it is a critical subsystem (such as subsystem B), the system outputs "Critical subsystem score is missing, total score cannot be calculated, please re-execute the scoring" to ensure the validity of the total score calculation.

[0096] Figure 2 The implementation framework diagram of the orthogonal multidimensional reward function based on hierarchical chain reasoning provided by this invention can be referred to. Figure 2 As shown, in the evaluation system, the first layer is the task input layer, serving as the data entry point for the entire evaluation system and responsible for receiving the task data to be evaluated. The task input layer supports two types of input formats: structured data, such as JSON format parameters of agent policy schemes. This type of data has a clear structure and fields, making it easy for the system to directly parse and process; and unstructured data, such as text schemes output by large language models. This type of data has a more flexible format and requires the system to have certain text processing capabilities to extract effective information. After data input, the system automatically generates a unique task ID, such as "Task-20240501-001". This ID is used throughout the entire evaluation process to associate data generated in subsequent layers, facilitating full-process tracking and management of each task.

[0097] The second layer is the orthogonal subsystem layer, which contains a built-in set of configurable orthogonal subsystems, including multiple core subsystems. These subsystems are designed for different aspects of the assessment task, such as integrity assessment subsystems, compliance assessment subsystems, and execution closure-loop assessment subsystems, comprehensively assessing the task solution from multiple dimensions.

[0098] In this invention, to ensure the independence of each subsystem and avoid information redundancy, the orthogonality of each subsystem is verified through statistical tests. For example, Pearson correlation coefficient is used to calculate the correlation coefficient, requiring that the correlation coefficient between any two subsystems be less than 0.3, satisfying the low correlation / low mutual information requirement. This ensures that each subsystem evaluates different information dimensions, improving the accuracy and reliability of the evaluation results.

[0099] The third layer is the chain-based reasoning and scoring layer, serving as the core processing layer. This layer corresponds one-to-one with each subsystem of the orthogonal subsystem layer, providing independent reasoning and scoring logic for each subsystem. (See reference...) Figure 2 As shown, the "three-stage chain evaluation" logic is used for explanation: each reasoning unit has a built-in logic of "indicator definition → condition judgment → scoring mechanism". First, the indicator is defined to determine the specific content and standards of the evaluation; then, judgment is made based on the defined conditions to determine whether the content to be evaluated meets the corresponding conditions; finally, a specific score is given according to the scoring mechanism. Moreover, each step generates traceable evidence records to ensure the transparency and explainability of the scoring process.

[0100] This invention first breaks down the high-level indicators of each subsystem into low-level sub-indicators. For example, the "completeness" of subsystem A is broken down into low-level sub-indicators such as "element coverage," "logical coherence," and "information completeness." After the low-level sub-indicators are scored, they are aggregated into high-level indicator scores according to preset aggregation rules (such as weighted summation or average score). Similarly, traceable evidence records are generated at each step.

[0101] Furthermore, the various sub-indicator systems are executed sequentially, and the various indicators within an indicator are also calculated sequentially. That is, a chain-reasoning scoring method is adopted to ensure the orderliness and logic of the evaluation process.

[0102] The fourth layer is the total score calculation and constraint layer. The weighted fusion module assigns weights according to the importance of each subsystem. In this invention, subsystem A has a weight of 0.4, subsystem B has a weight of 0.3, and subsystem C has a weight of 0.3. Through weighted fusion, the scores of each subsystem are combined to calculate the basic total score, reflecting the importance of different subsystems in the overall evaluation.

[0103] In this invention, a veto threshold for key subsystems can be preset; for example, the threshold for subsystem B is 60 points. After calculating the basic total score, it is passed to a veto module to determine whether a constraint is triggered. If the score of the key subsystem is lower than the veto threshold, the total score will be adjusted, and may even be reset to zero, to ensure that the evaluation results meet the key requirements and avoid serious non-compliance.

[0104] This invention first calculates the basic total score through a weighted fusion module, then passes it to a veto module to determine whether a constraint is triggered, and finally outputs the adjusted total score, providing the final quantitative indicator for the evaluation results.

[0105] The fifth layer is the results output layer, used to generate a structured evaluation report, containing at least three parts. The core results section includes information such as the final total score, task ID, and evaluation duration, intuitively presenting the key evaluation results. The subsystem details section displays the scores and sub-indicator decompositions of each subsystem, facilitating a deeper understanding of the evaluation status of each subsystem. The evidence chain section connects the evidence records of the chain-reasoning scoring layer and the aggregation logs of the hierarchical aggregation layer, providing detailed evidence and traceability information for the evaluation results.

[0106] Furthermore, the exported reports support both PDF and Excel formats to meet the needs of different scenarios. For example, PDF format is suitable for formal report presentation and archiving, while Excel format is convenient for data analysis and further processing.

[0107] Furthermore, the evaluation scenario for risk control strategies for intelligent agents in the financial field (such as "risk identification strategies for credit approval intelligent agents") is explained in detail, and the complete implementation steps are as follows: Step 1: Task Input and Subsystem Initialization (corresponding to) Figure 2 (First and second layers in the middle) By uploading the risk control strategy plan P to be evaluated (unstructured text format, including the applicable scenarios of the strategy, risk identification rules and decision logic, etc.) through the task input layer, the system will automatically generate the task ID "Task-20240501-001". Then, the system reads the scenario configuration file and loads three sets of orthogonal subsystems specifically for the financial field (Subsystem A: Strategy integrity assessment, including two sub-indicators: "risk factor coverage" and "rule logic integrity"; Subsystem B: Compliance assessment, including two sub-indicators: "regulatory policy compliance" and "user data compliance"; Subsystem C: Execution closure assessment, including two sub-indicators: "anomaly handling closure" and "result feedback closure"), and automatically verifies the orthogonality of the subsystems (calling the Pearson coefficient calculation interface, the results show that the correlation coefficients between A and B are 0.25, A and C are 0.28, and B and C are 0.22, all <0.3, which meets the requirements).

[0108] Step 2: Chain reasoning scoring (corresponding to) Figure 2 The third layer, taking the "User Data Compliance" indicator of Subsystem B's "Compliance Assessment" as an example): Phase 1: Indicator Definition (Generate Evid-B1 evidence record, associated with task ID), "User Data Compliance: Assess whether there is any user sensitive data (such as bank card number, credit information) in the risk control strategy that has not been desensitized, and whether it complies with the relevant requirements for the processing of sensitive personal information"; Phase 2: Condition Judgment (Generate Evid-B2 evidence record, including detection tool logs). The system calls the "Sensitive Data Detection Tool" to scan scheme P, and identifies that "the policy contains an example of a user's bank card number (6222XXXX1234) that has not been desensitized". At the same time, it searches for relevant clauses and determines that "the compliance conditions are not met". Phase 3 scoring mechanism (generating evidence record Evid-B3, associated with scoring rules), based on preset scoring rules ("80-100 points for meeting compliance conditions, 40-79 points for 1 instance of non-anonymization, 0-39 points for 2 or more instances of non-anonymization"). Scheme P has 1 instance of non-anonymization, so this indicator scores 55 points. The scoring basis is "bank card number not anonymized, triggering medium deduction rule".

[0109] Repeat the above "three-stage chain evaluation" process to complete the scoring of another sub-indicator of subsystem B, "regulatory policy compliance" (score 65 points), all sub-indicators of subsystem A ("risk factor coverage" 80 points, "rule logic integrity" 75 points), and all sub-indicators of subsystem C ("anomaly handling closed loop" 90 points, "result feedback closed loop" 85 points). All evidence records are associated with the task ID.

[0110] Step 3: Hierarchical aggregation (corresponding to) Figure 2 (the third layer) Subsystem A aggregation: Using the "weighted summation" rule ("risk factor coverage" weight 0.6, "rule logic integrity" weight 0.4), the total score of subsystem A = 80 × 0.6 + 75 × 0.4 = 48 + 30 = 78 points, generating an aggregation log Agg-A (including recording weight configuration and calculation process). Subsystem B aggregation: Using the "average score" rule, the total score of subsystem B = (55+65) / 2 = 60 points, generating the aggregation log Agg-B; Subsystem C aggregation: Using the "weighted summation" rule ("anomaly handling closed loop" weight 0.5, "result feedback closed loop" weight 0.5), the total score of subsystem C = 90×0.5 + 85×0.5 = 45 + 42.5 = 87.5 points, generating the aggregation log Agg-C; Furthermore, the aggregation results are synchronized to the hierarchical aggregation layer in real time and displayed in association with the corresponding subsystem modules.

[0111] Step 4: Total Score Calculation and Constraint Judgment (corresponding to) Figure 2 (the fourth layer) Weighted fusion calculation of the basic total score: According to the preset weights (A: 0.4, B: 0.3, C: 0.3), the basic total score = 78 × 0.4 + 60 × 0.3 + 87.5 × 0.3 = 31.2 + 18 + 26.25 = 75.45 points; Veto decision: Subsystem B (compliance) has a total score of 60 points, which is equal to the preset veto threshold (60 points). It is determined that "veto will not be triggered". The final total score is retained as 75.45 points (rounded to 75 points). A veto decision log (including the recorded threshold and decision result) is generated.

[0112] Step 5: Output the results (corresponding) Figure 2 (the fifth layer) The system automatically generates an evaluation report, the core contents of which include: Task ID "Task-20240501-001", evaluation object "Credit Approval Intelligent Agent Risk Control Strategy Solution P", final total score of 75 points (rating "Good"); sub-system details (A: 78 points, B: 60 points, C: 87.5 points) and sub-indicator breakdown; The report embeds a complete chain of evidence: users can click on the "User Data Compliance" indicator score to view Evid-B1 (indicator definition), Evid-B2 (detection log), and Evid-B3 (scoring rules); clicking on the total score of subsystem A will allow users to view Agg-A (aggregated log). The operator selects PDF format to export the report, and the system automatically adds a watermark ("Assessment Report No.: Report-20240501-001"), completing the entire assessment process.

[0113] This invention breaks through the limitations of existing technologies that rely on "stacked indicators and no explicit dependencies between steps". It sets the scoring process to be executed strictly in sequence, with each conclusion relying on verifiable evidence from the previous step. This solves the problems of loose scoring logic and lack of verification in existing methods, ensuring the rigor and verifiability of the scoring process.

[0114] Meanwhile, the design of orthogonal multi-set scoring system architecture effectively reduces indicator redundancy and information leakage, avoids the risks of duplicate scoring and bias amplification in existing methods, and improves the objectivity of evaluation results.

[0115] Furthermore, it employs a three-stage chain of "definition → fulfillment of conditions → scoring mechanism" to execute each indicator and enforces evidence alignment. Compared to the existing technology's "direct scoring or subjective labeling" approach, it solves the problem of the lack of verifiable constraints in scoring, ensuring that the source of each score has clear evidence.

[0116] This invention's indicators unfold hierarchically from top to bottom and summarize from bottom to top, ensuring both the hierarchical nature of the scoring and improving the interpretability and reproducibility of the solutions, facilitating the identification of optimization directions. Furthermore, the total score is designed as a "weighted fusion × veto gate," balancing the importance of each subsystem through weight allocation and preventing non-compliant solutions from obtaining high total scores through a veto mechanism. This solves the problems of unstable scoring and constraint failure in existing technologies, enhancing the reliability of the evaluation results.

[0117] The evaluation system based on hierarchical chain reasoning provided by the present invention will be described below. The evaluation system based on hierarchical chain reasoning described below can be referred to in correspondence with the evaluation method based on hierarchical chain reasoning described above.

[0118] Figure 3 This is a schematic diagram of the structure of the evaluation system based on hierarchical chain reasoning provided by the present invention, as shown below. Figure 3 As shown, this invention provides an evaluation system based on hierarchical chain reasoning, including a task input module 301 and an evaluation module 302. The task input module 301 is used to acquire target task scheme data to be evaluated in the current evaluation scenario. The evaluation module 302 is used to input the target task scheme data into a large language model corresponding to the evaluation scenario to obtain the target evaluation result of the target task scheme data output by the large language model. The large language model is trained based on reward functions under different evaluation scenarios. Each evaluation subsystem in the reward function is determined based on orthogonality testing. The reasoning scoring process corresponding to each evaluation subsystem is constructed based on the indicator type and chain reasoning in the evaluation scenario. The total score calculation process of the reward function includes evaluation subsystem scoring constraints, which constrain the evaluation subsystem scores output by the reasoning scoring process of the evaluation subsystem.

[0119] The hierarchical chain reasoning-based evaluation system provided by this invention constructs a hierarchical, multi-dimensional, and traceable reward function system to ensure that the index scores have logical causal relationships, effectively improving the robustness and reliability of the evaluation results. At the same time, it makes the reward function clearly defined, executable, and interpretable, providing a precise and effective quantitative evaluation tool for the evaluation of large language model outputs and the training of intelligent agents.

[0120] The system provided in this embodiment of the invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0121] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4As shown, the electronic device may include: a processor 401, a communications interface 402, a memory 403, and a communication bus 404, wherein the processor 401, communications interface 402, and memory 403 communicate with each other via the communication bus 404. The processor 401 can call logical instructions in the memory 403 to execute an evaluation method based on hierarchical chain reasoning. This method includes: acquiring target task scheme data to be evaluated in the current evaluation scenario; inputting the target task scheme data into a large language model corresponding to the evaluation scenario to obtain the target evaluation result of the target task scheme data output by the large language model, wherein the large language model is trained based on reward functions under different evaluation scenarios; each evaluation subsystem in the reward function is determined based on orthogonality testing; the reasoning scoring process corresponding to each evaluation subsystem is constructed based on the indicator type and chain reasoning in the evaluation scenario; the total score calculation process of the reward function includes evaluation subsystem scoring constraints, which are used to constrain the evaluation subsystem score output by the reasoning scoring process of the evaluation subsystem.

[0122] Furthermore, the logical instructions in the aforementioned memory 403 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the evaluation method based on hierarchical chain reasoning provided by the above methods, the method comprising: acquiring target task scheme data to be evaluated in the current evaluation scenario; inputting the target task scheme data into a large language model corresponding to the evaluation scenario to obtain a target evaluation result of the target task scheme data output by the large language model, wherein the large language model is trained based on reward functions under different evaluation scenarios; each evaluation subsystem in the reward function is determined based on orthogonality testing; the reasoning scoring process corresponding to each evaluation subsystem is constructed according to the indicator type and chain reasoning in the evaluation scenario; the total score calculation process of the reward function is set with evaluation subsystem scoring constraints, the evaluation subsystem scoring constraints being used to constrain the evaluation subsystem score output by the reasoning scoring process of the evaluation subsystem.

[0124] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the evaluation method based on hierarchical chain reasoning provided in the above embodiments. The method includes: acquiring target task scheme data to be evaluated in the current evaluation scenario; inputting the target task scheme data into a large language model corresponding to the evaluation scenario to obtain a target evaluation result of the target task scheme data output by the large language model, wherein the large language model is trained based on reward functions under different evaluation scenarios; each evaluation subsystem in the reward function is determined based on orthogonality testing; the reasoning scoring process corresponding to each evaluation subsystem is constructed according to the indicator type and chain reasoning in the evaluation scenario; the total score calculation process of the reward function is set with evaluation subsystem scoring constraints, which are used to constrain the evaluation subsystem score output by the reasoning scoring process of the evaluation subsystem.

[0125] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0126] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An evaluation method based on hierarchical chain reasoning, characterized in that, include: Obtain data on the target task schemes to be evaluated in the current evaluation scenario; The target task scheme data is input into a large language model corresponding to the evaluation scenario to obtain the target evaluation result of the target task scheme data output by the large language model. The large language model is trained based on reward functions under different evaluation scenarios. Each evaluation subsystem in the reward function is determined based on orthogonality testing. The reasoning scoring process corresponding to each evaluation subsystem is constructed based on the indicator type and chain reasoning in the evaluation scenario. The total score calculation process of the reward function is set with evaluation subsystem scoring constraints, which are used to constrain the evaluation subsystem score output by the reasoning scoring process of the evaluation subsystem.

2. The evaluation method based on hierarchical chain reasoning according to claim 1, characterized in that, The reward function includes an evaluation system layer, a chain reasoning scoring layer, and a total score calculation constraint layer, wherein: The evaluation system layer is provided with multiple evaluation subsystems corresponding to the evaluation scenario type, and the Pearson correlation coefficient between each evaluation subsystem is less than a preset correlation coefficient threshold. The chain-based reasoning scoring layer is used to determine the scoring judgment conditions and scoring mechanisms for each indicator to be scored in the evaluation subsystem based on the reasoning scoring process corresponding to the evaluation subsystem; and to perform chain-based reasoning on each indicator to be scored according to the scoring judgment conditions and the scoring mechanism to obtain the indicator score results corresponding to each indicator to be scored. The total score calculation constraint layer is used to aggregate the index scores of each of the indicators to be scored in the evaluation subsystem based on preset index aggregation rules to obtain the total score of each evaluation subsystem; constrain the total score of each evaluation subsystem based on the preset evaluation subsystem scoring rejection threshold corresponding to the total score of each evaluation subsystem; and sum the constrained total scores of the evaluation subsystem to obtain the target evaluation result of the target task scheme data.

3. The evaluation method based on hierarchical chain reasoning according to claim 2, characterized in that, The method further includes: Based on the assessment scenario and task adjustment information, a new assessment subsystem is constructed; Based on the determination that there is a non-linear relationship between the newly added evaluation subsystem and each evaluation subsystem in the current evaluation system layer, the mutual information value between the newly added evaluation subsystem and each evaluation subsystem is calculated based on mutual information entropy. If the mutual information value is determined to be less than the preset mutual information threshold, the newly added evaluation subsystem is added to the current evaluation system layer to obtain a new evaluation system layer.

4. The evaluation method based on hierarchical chain reasoning according to claim 2, characterized in that, The chain-based reasoning scoring layer is also used to perform anomaly verification on the condition judgment process corresponding to the target task scheme data in the scoring judgment conditions, and to re-execute the condition judgment process with anomalies.

5. The evaluation method based on hierarchical chain reasoning according to claim 2, characterized in that, The total score calculation constraint layer is also used to adjust the preset index aggregation rules according to the change information of the evaluation scenario, and to aggregate the index score results based on the adjusted preset index aggregation rules. The total score calculation constraint layer is also used to filter out indicators to be scored that have a score result less than a preset indicator score threshold during the aggregation process.

6. The evaluation method based on hierarchical chain reasoning according to claim 2, characterized in that, The preset evaluation subsystem scoring rejection threshold includes a first evaluation subsystem scoring rejection threshold and a second evaluation subsystem scoring rejection threshold, wherein the first evaluation subsystem scoring rejection threshold is greater than the second evaluation subsystem scoring rejection threshold. The constraint on the total score of each evaluation subsystem based on the preset evaluation subsystem score rejection threshold corresponding to the total score of each evaluation subsystem includes: When it is determined that the total score of the evaluation subsystem is greater than or equal to the score rejection threshold of the first evaluation subsystem, the total score of the evaluation subsystem is retained and summed. When it is determined that the total score of the evaluation subsystem is less than the first evaluation subsystem scoring rejection threshold and greater than or equal to the second evaluation subsystem scoring rejection threshold, a risk warning label is marked on the total score of the evaluation subsystem, and the total score of the evaluation subsystem after the risk warning label is marked is summed. When the total score of the evaluation subsystem is determined to be less than the rejection threshold of the second evaluation subsystem, the total score of the evaluation subsystem is reset to zero.

7. The evaluation method based on hierarchical chain reasoning according to any one of claims 2 to 6, characterized in that, The method further includes: The target task plan data is preprocessed and verified, and based on the preprocessing and verification results, a task plan data anomaly log corresponding to the target task plan data is generated. If the orthogonality test results between the evaluation subsystems do not meet the preset correlation coefficient threshold, the indicators in the evaluation subsystems are adjusted. If an index with an abnormal score is identified in the evaluation subsystem, the consistency of the reasoning and scoring process of each index to be scored in the evaluation subsystem is checked, and the scoring anomaly location information is obtained based on the consistency check result. If data anomalies are found during the total score calculation process, data completion processing is performed on the evaluation subsystem where the data anomalies occurred.

8. An evaluation system based on hierarchical chain reasoning, characterized in that, include: The task input module is used to obtain data on the target task scheme to be evaluated in the current evaluation scenario; The evaluation module is used to input the target task scheme data into a large language model corresponding to the evaluation scenario, and obtain the target evaluation result of the target task scheme data output by the large language model. The large language model is trained based on reward functions under different evaluation scenarios; each evaluation subsystem in the reward function is determined based on orthogonality testing; the reasoning scoring process corresponding to each evaluation subsystem is constructed based on the indicator type and chain reasoning in the evaluation scenario; the total score calculation process of the reward function includes evaluation subsystem scoring constraints, which are used to constrain the evaluation subsystem scores output by the reasoning scoring process of the evaluation subsystem.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the evaluation method based on hierarchical chain reasoning as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the evaluation method based on hierarchical chain reasoning as described in any one of claims 1 to 7.

Images