Policy network optimization method and related device

Abstract

The invention discloses a policy network optimization method and a related device, and relates to the technical field of artificial intelligence, and the method employs a reward function at least comprising a basic reward and a cognitive reward to carry out deep reinforcement learning on a policy network, is multi-target collaborative reinforcement learning, can synchronously improve the scoring accuracy and interpretability in an inference chain, and improves the learning efficiency. The generation of the reasoning chain is optimized, a new solution is provided for realizing more intelligent and more transparent model evaluation, and the method has wide application prospects and important practical values in the fields of intelligent question answering, decision assistance, content auditing and the like.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a strategy network optimization method and related apparatus. Background Technology

[0002] With the increasing application of large-scale models in diverse tasks such as mathematical reasoning, ethical judgment, and emotional dialogue, accurate reward modeling has become a core element in ensuring the quality of model output.

[0003] In existing large-scale reward modeling techniques, traditional policy network optimization methods focus solely on rating accuracy, leading to an imbalance between rating accuracy and interpretability in the inference chain, resulting in "correct but unreliable" ratings. Experiments show that traditional models often exhibit "correct ratings but weak explanations" in complex tasks. For example, rating ethical responses may meet expert standards, but the reasoning process may fail to mention key ethical principles, leading to insufficient user trust in the rating results. This deficiency can pose serious application risks, especially in scenarios with extremely high interpretability requirements, such as financial risk control and medical decision-making.

[0004] Therefore, how to provide a new strategy network optimization method to simultaneously improve the scoring accuracy and interpretability of the inference chain has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, this application provides a policy network optimization method and related apparatus to simultaneously improve the scoring accuracy and interpretability of the inference chain. The specific solution is as follows:

[0006] The first aspect of this application provides a method for optimizing policy networks, including:

[0007] Obtain sample information, which includes query text, model answers, and expert annotation information;

[0008] The inference chain for generating the model's answer is generated based on the query text;

[0009] Based on the inference chain and the expert annotation information, deep reinforcement learning is performed on the policy network to obtain the target policy network. The reward function used in the deep reinforcement learning includes at least a basic reward and a cognitive reward. The basic reward is used to indicate the degree of matching between the overall prediction score of the policy network on the model's answer quality based on the inference chain and the expert annotation information. The cognitive reward is used to indicate the degree of matching between the inference chain and the expert annotation information in terms of the accuracy of evidence citation and the comprehensiveness of evaluation dimension analysis.

[0010] In one possible implementation, the inference chain for generating the model's answer based on the query text includes:

[0011] Based on the task type corresponding to the query text, generate a set of evaluation dimensions specific to the target task type corresponding to the query text;

[0012] Based on the set of evaluation dimensions specific to the target task type, the reasoning chain for the model's answer is generated.

[0013] In one possible implementation, generating a set of evaluation dimensions specific to the target task type corresponding to the query text, based on the task type corresponding to the query text, includes:

[0014] Perform task semantic understanding on the query text to determine the target task type corresponding to the query text;

[0015] The evaluation dimensions for the target task type are initialized to generate a set of evaluation dimensions specific to the target task type.

[0016] In one possible implementation, generating the inference chain for the model's answer based on the set of evaluation dimensions specific to the target task type includes:

[0017] For each evaluation dimension, target evidence is retrieved from the professional knowledge base corresponding to the target task type; based on the target evidence and the model's response, an analysis report for the evaluation dimension is generated; based on the target evidence and the information of the evaluation dimension, a score for the evaluation dimension is generated.

[0018] Based on the analysis reports and scores of each of the evaluation dimensions, the reasoning chain for the model's answer is generated.

[0019] In one possible implementation, generating the reasoning chain for the model's answer based on the analysis reports and scores of each of the evaluation dimensions includes:

[0020] Obtain user feedback on their scores for each of the evaluation dimensions;

[0021] Based on the user's feedback on the scores of each of the evaluation dimensions, the initial weights of each of the evaluation dimensions are calibrated to obtain the calibrated weights of each of the evaluation dimensions.

[0022] Based on the calibrated weights of each of the evaluation dimensions, the scores of each of the evaluation dimensions are adjusted to obtain the calibrated scores of each of the evaluation dimensions.

[0023] Based on the analysis reports of each of the evaluation dimensions and the calibrated scores of each of the evaluation dimensions, the reasoning chain for the model's answer is generated.

[0024] In one possible implementation, the reward function used in the deep reinforcement learning also includes:

[0025] Regularization penalty, which indicates the degree of deviation of the calibrated weights of each of the evaluation dimensions from the initial weights of each of the evaluation dimensions.

[0026] In one possible implementation, the degree of matching between the reasoning chain and the expert annotation information in terms of the accuracy of evidence citation and the comprehensiveness of the evaluation dimension analysis is determined as follows:

[0027] Using the expert annotation information as a reference, the inference chain and the expert annotation information are compared at the text level, and the overlap evaluation index between the inference chain and the expert annotation information in a preset dimension is calculated as the matching degree.

[0028] The preset dimensions include any one or more of the following: key information coverage, logical structure similarity, and evidence citation matching degree.

[0029] A second aspect of this application provides a policy network optimization apparatus, comprising:

[0030] The acquisition unit is used to acquire sample information, which includes query text, model answer, and expert annotation information.

[0031] The inference chain generation unit is used to generate an inference chain for the model's answer based on the query text;

[0032] A deep reinforcement learning unit is used to perform deep reinforcement learning on a policy network based on the inference chain and the expert annotation information to obtain a target policy network. The reward function used in the deep reinforcement learning includes at least a basic reward and a cognitive reward. The basic reward is used to indicate the degree of matching between the overall prediction score of the policy network on the model's response quality based on the inference chain and the expert annotation information. The cognitive reward is used to indicate the degree of matching between the inference chain and the expert annotation information in terms of the accuracy of evidence citation and the comprehensiveness of evaluation dimension analysis.

[0033] In one possible implementation, the inference chain generation unit includes:

[0034] The evaluation dimension set generation unit is used to generate an evaluation dimension set specific to the target task type corresponding to the query text, based on the task type corresponding to the query text.

[0035] The inference chain generation subunit is used to generate the inference chain of the model's answer based on the set of evaluation dimensions specific to the target task type.

[0036] In one possible implementation, the evaluation dimension set generation unit is specifically used for:

[0037] Perform task semantic understanding on the query text to determine the target task type corresponding to the query text;

[0038] The evaluation dimensions for the target task type are initialized to generate a set of evaluation dimensions specific to the target task type.

[0039] In one possible implementation, the inference chain generating sub-unit is specifically used for:

[0040] For each evaluation dimension, target evidence is retrieved from the professional knowledge base corresponding to the target task type; based on the target evidence and the model's response, an analysis report for the evaluation dimension is generated; based on the target evidence and the information of the evaluation dimension, a score for the evaluation dimension is generated.

[0041] Based on the analysis reports and scores of each of the evaluation dimensions, the reasoning chain for the model's answer is generated.

[0042] In one possible implementation, the inference chain generating sub-unit is specifically used for:

[0043] Obtain user feedback on their scores for each of the evaluation dimensions;

[0044] Based on the user's feedback on the scores of each of the evaluation dimensions, the initial weights of each of the evaluation dimensions are calibrated to obtain the calibrated weights of each of the evaluation dimensions.

[0045] Based on the calibrated weights of each of the evaluation dimensions, the scores of each of the evaluation dimensions are adjusted to obtain the calibrated scores of each of the evaluation dimensions.

[0046] Based on the analysis reports of each of the evaluation dimensions and the calibrated scores of each of the evaluation dimensions, the reasoning chain for the model's answer is generated.

[0047] In one possible implementation, the reward function used in the deep reinforcement learning also includes:

[0048] Regularization penalty, which indicates the degree of deviation of the calibrated weights of each of the evaluation dimensions from the initial weights of each of the evaluation dimensions.

[0049] In one possible implementation, the deep reinforcement learning unit is specifically used for:

[0050] Using the expert annotation information as a reference, the inference chain and the expert annotation information are compared at the text level, and the overlap evaluation index between the inference chain and the expert annotation information in a preset dimension is calculated as the matching degree.

[0051] The preset dimensions include any one or more of the following: key information coverage, logical structure similarity, and evidence citation matching degree.

[0052] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the strategy network optimization method of the first aspect or any implementation thereof.

[0053] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0054] The memory is used to store computer programs;

[0055] The processor is used to execute the computer program so that the electronic device can implement the strategy network optimization method of the first aspect or any implementation thereof.

[0056] The fifth aspect of this application provides a computer-readable storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the strategy network optimization method described in the first aspect or any implementation thereof.

[0057] By employing the above technical solutions, the policy network optimization method and related apparatus provided in this application use a reward function that includes at least basic rewards and cognitive rewards to perform deep reinforcement learning on the policy network. This is a multi-objective collaborative reinforcement learning method that can simultaneously improve the accuracy and interpretability of scoring in the inference chain and optimize the generation of the inference chain. It provides a new solution for achieving more intelligent and transparent model evaluation and has broad application prospects and important practical value in fields such as intelligent question answering, decision support, and content moderation. Attached Figure Description

[0058] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0059] Figure 1 A flowchart illustrating a policy network optimization method provided in an embodiment of this application;

[0060] Figure 2 A schematic diagram of the structure of a policy network optimization device provided in an embodiment of this application;

[0061] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0062] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0063] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0064] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0065] With the increasing application of large-scale models in diverse tasks such as mathematical reasoning, ethical judgment, and emotional dialogue, accurate reward modeling has become a core element in ensuring the quality of model output.

[0066] In existing large-scale reward modeling techniques, traditional policy network optimization methods focus solely on rating accuracy, leading to an imbalance between rating accuracy and interpretability in the inference chain, resulting in "correct but unreliable" ratings. Experiments show that traditional models often exhibit "correct ratings but weak explanations" in complex tasks. For example, rating ethical responses may meet expert standards, but the reasoning process may fail to mention key ethical principles, leading to insufficient user trust in the rating results. This deficiency can pose serious application risks, especially in scenarios with extremely high interpretability requirements, such as financial risk control and medical decision-making.

[0067] To address the aforementioned problems, this application provides a policy network optimization method. The policy network optimization method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0068] Reference Figure 1 , Figure 1 This is a flowchart illustrating a policy network optimization method provided in an embodiment of this application, as shown below. Figure 1 As shown in the figure, the policy network optimization method provided in this application embodiment may include the following steps, which are described in detail below.

[0069] S101: Obtain sample information, which includes query text, model answer, and expert annotation information;

[0070] In this application, the sample information may be the application information of large models in diverse tasks such as mathematical reasoning, ethical judgment, and emotional dialogue. This application does not impose any restrictions on this.

[0071] In this application, the expert annotation information is generated manually, and needs to be generated by combining task characteristics and professional knowledge. The entire process relies on the professional judgment of experts.

[0072] First, experts will define the evaluation dimensions based on the task type (such as mathematical reasoning or ethical judgment). These dimensions can be pre-defined general dimensions or, as in this case, dynamically generated specific dimensions for different task types. This application does not impose any limitations on this. For example, ethical judgment tasks have three specific evaluation dimensions: social fairness, predictability of consequences, and compliance with ethical principles. Mathematical reasoning tasks have two specific evaluation dimensions: completeness of logical steps and accuracy of formula application.

[0073] Secondly, for the model's answers, experts will analyze the performance of each evaluation dimension one by one (such as whether the "completeness of logical steps" in the mathematical reasoning task meets the standard, and whether the "social fairness" in the ethical judgment task is satisfied), and manually give the scores for each evaluation dimension and the overall score.

[0074] Finally, based on a professional evidence system, the model's responses were analyzed from various evaluation dimensions, and the overall quality of the model's responses was analyzed to obtain expert annotation information.

[0075] In one possible implementation, the expert annotation information includes expert annotation scores and expert annotation analysis. The expert annotation scores include the evaluation dimension-level scores and overall scores of the experts on the model's answer. The expert annotation analysis includes the scoring and analysis process of the experts on the model's answer from various evaluation dimensions based on a professional evidence system, as well as the analysis results of the experts on the overall quality of the model's answer.

[0076] To facilitate understanding, this application provides a sample information example using an ethical judgment task, as follows:

[0077] The query text is: An e-commerce platform plans to analyze the "place of origin" information in users' resumes to prioritize recommending "local residents" for offline internship positions. Determine whether this plan complies with ethical requirements.

[0078] The model responded that the solution complies with ethical requirements. The reasons are: local residents are more familiar with local transportation and culture, enabling them to quickly adapt to internships and improve recruitment efficiency; furthermore, the platform only uses user-submitted resume information, thus avoiding privacy breaches.

[0079] Suppose there are three evaluation dimensions for this ethical judgment task: social fairness, consequence foresight, and ethical compliance. The weight of the social fairness evaluation dimension is 0.35, the weight of the consequence foresight evaluation dimension is 0.3, and the weight of the ethical compliance evaluation dimension is 0.35, with a total weight of 1. Then the expert-annotated scores can be as follows: the social fairness evaluation dimension score is 20 points, with a weighted score of 20 × 0.35 = 7 points; the consequence foresight evaluation dimension score is 35 points, with a weighted score of 35 × 0.3 = 10.5 points; and the ethical compliance evaluation dimension score is 40 points, with a weighted score of 40 × 0.35 = 14 points. The total score is 7 + 10.5 + 14 = 31.5 points, rounded to 32 points.

[0080] The expert analysis indicates that, following the principle of "multi-source evidence fusion reasoning," the scoring analysis cites ethical guidelines and laws and regulations as evidence. In the social fairness assessment dimension, Article 3 of the Employment Promotion Law stipulates that "workers shall enjoy the right to equal employment and independent job selection in accordance with the law, and workers shall not be discriminated against in employment because of differences in ethnicity, race, gender, religious belief, etc." Place of origin is an "indirect discriminatory feature," but the model's response ignores the risk of "regional discrimination." "Prioritizing local residents" essentially deprives non-local users of equal competitive opportunities, violating the core ethical principle of "employment fairness." Therefore, this assessment dimension only scores 20 points. In the consequence foresight assessment dimension, the model only mentions "recruitment efficiency" and fails to foresee two major risks: first, legal risk, violating Article 28 of the Personal Information Protection Law, which states that "the processing of sensitive personal information requires separate consent." Although place of origin is not core sensitive information, its use in "discriminatory screening" constitutes a violation; second, brand risk, potentially triggering public controversy over "regional discrimination" and damaging the platform's social image. The consequence analysis is one-sided and lacks consideration for multiple stakeholders (non-local users, regulatory agencies). This assessment dimension scores 35 points. In the ethical compliance assessment dimension, GDPR Article 5.1.b stipulates that "the processing of personal data shall be limited to the extent necessary to achieve the purpose of processing." "Place of origin information" is not directly related to "competence in the internship position" and falls under the category of "processing data beyond the necessary scope." Although the model mentions "no privacy breach" and meets the basic compliance requirements, it does not meet the requirement of "legality of the purpose of data use." Therefore, this assessment dimension scores 40 points.

[0081] Overall, the model's response suffers from three major problems: "lack of fairness, insufficient risk assessment, and biased understanding of the criteria." The proposed solution does not meet ethical requirements and needs to be optimized to "screen based on job competency (such as communication skills and professional skills) and remove the place of origin association rule."

[0082] S102: Generate the reasoning chain of the model's answer based on the query text;

[0083] In this application, the reasoning chain is automatically generated, requiring the combination of relevant models and knowledge bases, without the need for manual intervention.

[0084] First, the set of evaluation dimensions is determined based on the query text;

[0085] Secondly, the evaluation dimension set, model responses, and corresponding professional knowledge bases (such as mathematical formula libraries and ethical guidelines libraries) are input into the inference chain generation model. The inference chain generation model first uses RAG technology to retrieve supporting evidence (such as theorems in mathematical problems and guidelines in ethical judgments) that match each evaluation dimension from the professional knowledge base; then it calls text generation models such as T5, and combines the model responses with the retrieved evidence to generate a detailed analysis report for each dimension (including positive matching and negative conflict content); finally, the scores for each dimension are calculated, and the "dimensional name - analysis report - dimension score" is integrated into a complete inference chain.

[0086] In one possible implementation, the reasoning chain contains information on multiple evaluation dimensions. Each evaluation dimension includes an analysis report and a score, generated based on evidence matching that dimension. The analysis report is a structured report containing analysis of the positive and negative aspects of the evidence. For example, in a mathematical reasoning task, the reasoning chain might include phrases like "according to the quadratic formula on page 57 of the People's Education Press junior high school mathematics textbook" and "the discriminant Δ = 9 > 0, therefore there are two unequal real roots." In an ethical judgment task, the reasoning chain might include phrases like "violation of Article 28 of the Personal Information Protection Law."

[0087] For ease of understanding, this application uses an ethical judgment task as an example. Assume there are three evaluation dimensions for this task: social fairness, predictability of consequences, and compliance with ethical principles. The weight of the social fairness evaluation dimension is 0.35, the weight of the predictability of consequences evaluation dimension is 0.3, and the weight of the compliance with ethical principles evaluation dimension is 0.35, for a total weight of 1. An example of the reasoning chain can be shown below:

[0088] Regarding the social fairness assessment dimension, Article 3 of the Employment Promotion Law and the Guidelines for the Identification of Employment Discrimination were retrieved from the Labor and Employment Ethics Code Database and the Laws and Regulations Database. The analysis of the model under evaluation answered the question of not realizing that "prioritizing local household registration" constitutes regional discrimination and deprives non-local household registration users of equal competitive opportunities. Combined with semantic relevance, the dimension score was calculated to be 20 points and the weighted score was 7 points.

[0089] Regarding the consequence foresight assessment dimension, Article 28 of the Personal Information Protection Law and cases of regional discrimination against e-commerce platforms were retrieved from the data compliance law database and the enterprise risk case database. It was pointed out that the model failed to foresee legal compliance risks and brand public opinion risks, and did not consider the rights and interests of multiple stakeholders. The dimension score was 35 points and the weighted score was 10.5 points.

[0090] Regarding the ethical compliance assessment dimension, GDPR Article 5.1.b and Article 6 of the Personal Information Protection Law were retrieved from the international data ethics guidelines database and the data compliance law database. Although the analysis model meets the data security requirements, the "place of origin information" is irrelevant to job competency and the data processing exceeds the necessary scope, resulting in a dimension score of 40 points and a weighted score of 14 points.

[0091] S103: Based on the inference chain and the expert annotation information, perform deep reinforcement learning on the policy network to obtain the target policy network. The reward function used in the deep reinforcement learning includes at least a basic reward and a cognitive reward. The basic reward is used to indicate the degree of matching between the overall prediction score of the policy network on the model's answer quality based on the inference chain and the expert annotation information. The cognitive reward is used to indicate the degree of matching between the inference chain and the expert annotation information in terms of the accuracy of evidence citation and the comprehensiveness of evaluation dimension analysis.

[0092] In this application, as one possible implementation, the basic reward is used to indicate the degree of matching between the overall predicted score of the policy network for the quality of the model's response based on the inference chain and the overall score in the expert annotation information, and the cognitive reward is used to indicate the degree of matching between the inference chain and the expert annotation analysis in the expert annotation information in terms of the accuracy of evidence citation and the comprehensiveness of evaluation dimension analysis.

[0093] In this application, the core role of the reasoning chain is to provide "interpretable scoring criteria" for the policy network, including evidence analysis reports and scores for each evaluation dimension.

[0094] In this application, one of the core functions of the policy network is to perform comprehensive reasoning based on the inference chain, analyze the reasonableness of the scores and the strength of evidence support for each evaluation dimension in the inference chain, and then comprehensively judge the quality of the model's response, ultimately outputting a "predicted overall score" that reflects the quality of the model's response. For example, in a mathematical reasoning task, the inference chain might include "logical step completeness score 0.8", "formula application accuracy score 0.9", and corresponding evidence (such as "correct verification by citing the Pythagorean theorem"). The policy network analyzes the reasonableness and correlation of these dimension scores, and then predicts an "overall score of 0.85". This "0.85" is the overall predicted score of the policy network based on the inference chain for the quality of the model's response. The calculation logic of the basic reward is to compare the difference between this overall predicted score and the overall score in the expert annotation information. The smaller the difference, the higher the basic reward, thereby guiding the policy network to learn to conform to the expert's evaluation criteria and generate a more accurate overall score.

[0095] Another core function of the policy network is to match the reasoning chain with the expert annotation analysis in the expert annotation information. The calculation logic of cognitive reward is to compare the difference between the reasoning chain and the expert annotation analysis in the expert annotation information in terms of the accuracy of evidence citation and the comprehensiveness of evaluation dimension analysis. The smaller the difference, the higher the cognitive reward, thereby improving the interpretability of the reasoning chain.

[0096] It should be noted that cognitive reward is essentially the alignment of "accuracy of evidence citation and comprehensiveness of evaluation dimension analysis" in the reasoning chain with "professional evidence system and evaluation dimension logic" in expert annotation analysis. For example, expert annotation analysis in mathematical reasoning tasks will emphasize that "the accuracy of formula application needs to be verified by coefficient substitution". Cognitive reward will evaluate whether the reasoning chain contains "evidence verification of coefficient substitution process". This requires the policy network to understand "the correlation logic between evidence and dimension score" rather than simple text matching.

[0097] It's also important to note that when comparing the comprehensiveness of cognitive reward's evaluation dimensions with the expert annotation analysis in the context of regression analysis and expert annotation information, the evaluation dimensions are dynamically assessed based on the task type. For example, in an ethical judgment task, the expert annotation analysis might focus on the "social fairness" dimension to analyze the "risk of regional discrimination." Cognitive reward needs to assess whether the model's reasoning chain "generates risk judgment logic consistent with the expert annotation analysis for the dynamic dimension of 'social fairness.'" If the task switches to mathematical reasoning, cognitive reward needs to assess whether the reasoning chain's "step missing judgment" is consistent with the expert annotation analysis, focusing on the "completeness of logical steps." This "dynamic evaluation of dimensions based on task type and progress" is something that traditional fixed-dimensional reward modeling cannot achieve.

[0098] In summary, in this application, the basic reward is used to evaluate the scoring accuracy of the inference chain, and the cognitive reward is used to evaluate the interpretability of the inference chain. In one possible implementation, a proximal policy optimization (PPO) algorithm can be used to perform deep reinforcement learning on the policy network. By learning user feedback and environmental interaction, the output policy is continuously adjusted, dynamically optimizing the weight allocation of the final score and evaluation dimensions, thereby achieving a synergistic improvement in accuracy and interpretability.

[0099] Traditional policy network optimization methods focus solely on scoring accuracy, leading to an imbalance between scoring accuracy and interpretability in the inference chain. In contrast, the proposed solution employs a reward function that includes at least basic and cognitive rewards for deep reinforcement learning of the policy network. This multi-objective collaborative reinforcement learning simultaneously improves scoring accuracy and interpretability in the inference chain, optimizing inference chain generation. It provides a new solution for achieving more intelligent and transparent model evaluation, demonstrating broad application prospects and significant practical value in areas such as intelligent question answering, decision support, and content moderation.

[0100] It is important to emphasize that, in this application, the role of cognitive reward is to "guide the model to actively generate logically coherent and well-supported reasoning chains while improving scoring accuracy." For example, in financial risk control scenarios, the model must not only provide a "risk score" (basic reward constraint), but also explicitly explain the "regulatory provisions and user behavior characteristics on which the risk score is based" (cognitive reward constraint) through the reasoning chain. This "multi-objective synergy" design represents a fundamental breakthrough from the traditional "single-objective optimization" of reward modeling, and its innovation lies in "reconstructing the objective system of the reward function." The innovation of cognitive reward essentially solves the core pain point of "lack of interpretability" in traditional reward models.

[0101] In one possible implementation, the degree of matching between the reasoning chain and the expert annotation information in terms of the accuracy of evidence citation and the comprehensiveness of the evaluation dimension analysis is determined as follows:

[0102] Using the expert annotation information as a reference, the inference chain and the expert annotation information are compared at the text level, and the overlap evaluation index between the inference chain and the expert annotation information in a preset dimension is calculated as the matching degree.

[0103] The preset dimensions include any one or more of the following: key information coverage, logical structure similarity, and evidence citation matching degree.

[0104] It should be noted that, in this application, the degree of matching between the inference chain and the expert annotation information in terms of the accuracy of evidence citation and the comprehensiveness of the evaluation dimension analysis can be calculated using evaluation metrics in the field of natural language generation (NLG) (such as ROUGE-L mentioned in the document). The core is to quantify the degree of overlap between the two in terms of content relevance, logical completeness and evidence consistency.

[0105] The specific calculation logic is as follows: Using expert-annotated analysis as a benchmark (including the expert's analytical perspective on each evaluation dimension, key cited evidence, and logical derivation process), the reasoning chain (generated by the multi-source evidence fusion reasoning module, including evidence retrieval results and analysis reports for each dimension) is compared with the expert-annotated analysis at the textual level. Through indicators such as ROUGE-L, the degree of overlap between the reasoning chain and the expert-annotated analysis is measured in dimensions such as "key information coverage" (e.g., whether it mentions the core evaluation points in the expert-annotated analysis), "logical structure similarity" (e.g., whether it is consistent with the dimensional analysis order and argumentation logic of the expert-annotated analysis), and "evidence citation matching degree" (e.g., whether it cites content from the professional knowledge base recognized in the expert-annotated analysis, such as mathematical theorems and ethical guidelines). A higher degree of overlap results in a higher cognitive reward score, thus ensuring that the interpretability of the reasoning chain conforms to the analytical standards recognized by the experts.

[0106] Given that existing large-scale model reward modeling techniques rely on pre-defined, uniform evaluation dimensions (such as fluency and relevance) to generate the model's reasoning chain, they cannot dynamically adapt to task types. This means they cannot dynamically generate specific evaluation dimensions based on task type, leading to missing or redundant evaluation dimensions in professional domain assessments. For example, in mathematical reasoning tasks, the lack of specific evaluation dimensions such as "completeness of logical steps" and "accuracy of formula application" results in scoring biases for complex derivations, failing to effectively assess the rigor of logical derivation. In ethical judgment scenarios, the absence of specific evaluation dimensions such as "social fairness" and "foreseeability of consequences" makes it difficult to capture the value connotation and potential risks of the answer, lacking consideration of social impact.

[0107] To address this issue, this application proposes a novel implementation of generating reasoning chains, as detailed below:

[0108] In one possible implementation, the inference chain for generating the model's answer based on the query text includes:

[0109] S201: Based on the task type corresponding to the query text, generate a set of evaluation dimensions specific to the target task type corresponding to the query text;

[0110] In this application, the set of evaluation dimensions includes information on multiple evaluation dimensions, each of which includes a name, a detailed description, and an initial weight. It should be noted that the initial weights of each evaluation dimension must sum to 1 to ensure a quantitative balance in the importance of each evaluation dimension.

[0111] For ease of understanding, assume that a set of evaluation dimensions D = {d1, d2, ..., dn} and an initial weight vector specific to a certain target task type are generated. Weights satisfy Where di represents the i-th evaluation dimension in the set. This corresponds to the initial weights of the evaluation dimension di. Formula This indicates that the sum of these initial weights is 1, which ensures that the total proportion of the initial weights of each evaluation dimension in the overall evaluation system is reasonable.

[0112] For example, for the task type of "ethical judgment", assessment dimensions such as "social fairness" and "consequence foresight" can be generated and assigned corresponding initial weights, with the sum of each initial weight being 1.

[0113] Unlike the generalized model of traditional fixed evaluation dimensions, this application generates a set of evaluation dimensions specific to the task type corresponding to the query text, based on the task type corresponding to the query text. This breaks through the generalization limitations of traditional fixed evaluation dimensions, realizes task-adaptive evaluation dimensions, and enables precise alignment between evaluation dimensions and task characteristics.

[0114] S202: Generate the reasoning chain for the model's answer based on the set of evaluation dimensions specific to the target task type.

[0115] In this application, the model's response can be analyzed in depth and detail based on each evaluation dimension in the set of evaluation dimensions specific to the target task type, generating an inference chain for the model's response. After generating the set of evaluation dimensions specific to the task type corresponding to the query text, the evaluation dimensions in the generated inference chain of the model's response, based on the set of evaluation dimensions specific to the target task type, are precisely aligned with the task characteristics. This solves the problem of missing or redundant evaluation dimensions in the inference chain during professional domain evaluation, significantly improving the completeness and relevance of professional domain evaluation.

[0116] In one possible implementation, generating a set of evaluation dimensions specific to the target task type corresponding to the query text, based on the task type corresponding to the query text, includes:

[0117] S301: Perform task semantic understanding on the query text to determine the target task type corresponding to the query text;

[0118] In this application, the query text can be semantically understood using natural language processing technology. In one possible implementation, a pre-trained language model (such as BERT-Large) can be used to perform deep semantic encoding on the query text first, and then the task classification module can be used to determine the specific type of the task to which the query text corresponds, such as mathematical reasoning, ethical judgment, emotional support, or common sense question answering.

[0119] S302: Initialize the evaluation dimensions for the target task type and generate a set of evaluation dimensions specific to the target task type;

[0120] In one possible implementation, a large model with powerful generation capabilities (such as GPT-4) can be invoked to initialize the evaluation dimensions for the target task type based on its characteristics and requirements, thereby generating a set of evaluation dimensions that matches it.

[0121] For ease of understanding, the pseudocode for task semantic parsing and evaluation dimension initialization is provided in this application, as shown in Table 1 below:

[0122] Table 1 Task Semantic Parsing and Evaluation Dimension Initialization

[0123]

[0124] Here, embedding is a vector representation obtained by semantically encoding the input query text using the BERT_Large model, used to capture deep semantic features of the text; task_type is the task type determined after processing the embedding using the task classification head (such as "mathematical reasoning", "ethical judgment", etc.); prompt is a prompt word constructed based on task_type, explicitly informing GPT-4 to generate the evaluation dimensions and initial weights (the weights sum to 1) corresponding to this task type; dimension_weight_set is the final result generated and returned by GPT-4, which is a set containing the specific evaluation dimension names, descriptions, and corresponding initial weights.

[0125] Taking the "ethical judgment" task type as an example again, the prompt examples are shown in Table 2 below:

[0126] Table 2 Examples of Ethical Judgment Task Prompts

[0127]

[0128] To facilitate understanding, this application provides an example of generating an evaluation dimension set, using an ethical judgment task as an example, as follows:

[0129] For the ethical judgment (user data compliance) task of "an e-commerce platform plans to analyze the 'place of origin' information in user resumes to prioritize recommending local users to participate in offline internship recruitment, and determine whether this plan complies with ethical requirements", the task text is first semantically encoded using the BERT-Large model. After the Task_Classifier determines the task type, a prompt is constructed and GPT-4 is called to generate an evaluation dimension and initial weight set including "social fairness (weight 0.35, assessing whether there is discriminatory screening and ensuring equal competitive opportunities)", "consequence predictability (weight 0.3, assessing potential adverse consequences and consideration of the rights and interests of multiple stakeholders)", and "ethical compliance (weight 0.35, assessing whether the data processing complies with laws, regulations and ethical guidelines)".

[0130] Considering that existing large-scale model reward modeling techniques rely on implicit decisions of the model's internal parameters during the generation of the inference chain, without integrating explicit evidence support from external knowledge graphs (such as mathematical theorem libraries, ethical code libraries, and sentiment dictionaries), for example, in medical question-answering assessments, professional norms (such as the Hippocratic Oath) cannot be cited as ethical scoring criteria, or explicit verification of formulas such as the Pythagorean theorem and the definition of derivatives is lacking in mathematical question scoring, the inference chain becomes ambiguous, lacks traceability, and is difficult for users to understand the scoring logic. Therefore, in this application, a multi-source evidence fusion reasoning approach can be used to generate the inference chain of the model's answer. Specifically, as follows:

[0131] In one possible implementation, generating the inference chain for the model's answer based on the set of evaluation dimensions specific to the target task type includes:

[0132] S401: For each evaluation dimension, retrieve target evidence from the professional knowledge base corresponding to the target task type; based on the target evidence and the model answer, generate an analysis report for the evaluation dimension; based on the target evidence and the information of the evaluation dimension, generate a score for the evaluation dimension.

[0133] For each assessment dimension, relevant supporting evidence can be obtained from the corresponding professional knowledge base, such as mathematical formula base, ethical code base or sentiment dictionary, through specific search techniques (such as RAG search) as target evidence.

[0134] In one possible implementation, a text generation model (such as the T5 model) can be used to comprehensively process the target evidence and the model's response to generate an analysis report for that evaluation dimension. The analysis report covers the degree to which the model's response matches the positive evidence in that evaluation dimension, as well as any potential conflicts with the negative evidence.

[0135] In one possible implementation, the semantic relevance of the target evidence to the specific description of the evaluation dimension can be measured, and the score of the evaluation dimension can be obtained by combining the weight of the evaluation dimension.

[0136] The scoring formulas for the evaluation dimensions are as follows:

[0137]

[0138] Among them, E i Let d be the vector of the target evidence. i To evaluate the vector that specifically describes the dimension, w i To evaluate the current weights of the dimension.

[0139] S402: Based on the analysis reports and scores of each of the evaluation dimensions, generate the reasoning chain for the model's answer.

[0140] For ease of understanding, the pseudocode for multi-source evidence fusion reasoning is provided in this application, as shown in Table 3 below:

[0141] Table 3 Pseudocode for Multi-Source Evidence Fusion Reasoning

[0142]

[0143]

[0144] The core logic involves retrieving relevant evidence from a knowledge base for each generated evaluation dimension, combining it with model responses to generate a dimension analysis report, and calculating the final score to form a complete reasoning chain.

[0145] Specifically, `dimensions` is the aforementioned set of evaluation dimensions, including dimension names, specific descriptions, and initial weights; `response` is the model's answer to be evaluated; and `knowledge_base` is a knowledge base matching the task type (such as a mathematical formula library or an ethical guidelines library). Evidence related to each dimension is retrieved, and an analysis report (`analysis`) is generated based on the evidence and the answer. Then, a score for that dimension is calculated using semantic similarity (considering the dimension weights). Finally, the analysis results for all dimensions are integrated into `reasoning_chain`.

[0146] This application designs a multi-source evidence fusion reasoning mechanism, which retrieves supporting evidence from the corresponding professional knowledge base, generates a detailed dimensional analysis report by combining the model's response, and calculates dimensional scores through semantic relevance, forming a traceable and evidence-based reasoning chain. This significantly improves the transparency and credibility of the scoring process, fills the interpretability gap of traditional single-modal black-box scoring, and enhances the interpretability and credibility of the reasoning chain. It makes the scoring basis traceable (e.g., "violation of Article 7 of the General Data Protection Regulation"). Validated by indicators such as NLG and ROUGE-L, the interpretability score of the reasoning chain is significantly improved, transforming the scoring basis from implicit decision-making based on internal model parameters to explicit support from external evidence. This solves the trust problem in interpretability-sensitive fields such as medicine and finance.

[0147] In one possible implementation, the reasoning chain for generating the model's answer based on the analysis reports and scores of each of the evaluation dimensions can be generated by combining the analysis reports and scores of each of the evaluation dimensions.

[0148] Existing large-scale reward modeling techniques rely on experience-based weighting for evaluation dimensions, lacking a dynamic adjustment mechanism driven by user feedback. In practical applications, different users exhibit significant differences in their focus on "logical rigor" and "emotional plausibility." For instance, users in the education field prioritize the correctness of reasoning steps, while users in psychological counseling scenarios emphasize the subtlety of emotional resonance. Traditional techniques struggle to accommodate the individualized differences in users' focus on dimensions such as logical rigor and emotional plausibility, leading to a disconnect between scoring standards and user expectations.

[0149] To address this issue, this application provides another method for generating the model's inference chain based on the analysis reports and scores of each of the aforementioned evaluation dimensions. This method can use interactive weight calibration to ensure that the scores in the inference chain meet the user's actual needs. The core of interactive weight calibration is to dynamically adjust the weights of each evaluation dimension through user feedback on the model's inference process, making the evaluation criteria of the reward model more aligned with user needs. Specifically:

[0150] In one possible implementation, generating the reasoning chain for the model's answer based on the analysis reports and scores of each of the evaluation dimensions includes:

[0151] S501: Obtain user feedback on the scores of each of the aforementioned evaluation dimensions;

[0152] In this application, examples of user feedback on the scores of each of the evaluation dimensions include "This score places too much emphasis on emotional support and ignores logical rationality" and "Logical rigor should be given a higher weight."

[0153] S502: Based on the user's feedback on the scores of each of the evaluation dimensions, the initial weights of each of the evaluation dimensions are calibrated to obtain the calibrated weights of each of the evaluation dimensions;

[0154] In this application, user feedback on scores for each of the evaluation dimensions can be transformed into specific weight adjustment signals, such as increasing or decreasing the weight value of a certain dimension. During the adjustment process, algorithms such as Bayesian optimization are employed to progressively optimize the importance allocation of each dimension while ensuring the total weight sum is 1. Simultaneously, to prevent weight adjustments from deviating excessively from the initial reasonable range, a regularization mechanism is introduced to constrain weight changes, ensuring the model maintains the stability of the evaluation system while learning user preferences.

[0155] In one possible implementation, the user's feedback on the scores of each of the evaluation dimensions can be converted into a weight adjustment signal ΔW, and the weight vector W can be updated using a Bayesian optimization algorithm. t+1 =W t +α·ΔW, where α is the learning rate, is dynamically optimized through reinforcement learning, and a weight regularization term is introduced to prevent overfitting. The weight regularization term is:

[0156]

[0157] For ease of understanding, the pseudocode for interactive weight calibration is provided in this application, as shown in Table 4 below:

[0158] Table 4. Pseudocode for Interactive Weight Calibration

[0159]

[0160]

[0161] Its core logic is to dynamically adjust the weights based on user feedback on the evaluation dimensions, making the model evaluation criteria more aligned with user needs. Specifically, W is the current weight vector, representing the importance of each evaluation dimension; feedback is the user's feedback on the rating (e.g., "the weight for logical rigor should be higher"), which is converted into a specific adjustment direction delta_W using the ParseFeedback function. Subsequently, the weights are iteratively updated based on the learning rate α, and normalization is used to ensure the sum of the weights is 1. To prevent over-adjustment, a regularization term reg_term is introduced, utilizing the initial weights W... 0 The weight variation range is constrained, and the final output is the calibrated weight vector W'. The entire process automates the transition from user feedback to weight adjustment, balancing personalization and stability through Bayesian optimization and regularization.

[0162] For ease of understanding, a typical user feedback processing example is provided in this application, as follows:

[0163] The user's input was: "This rating places too much emphasis on emotional support and ignores logical rationality."

[0164] The analysis is as follows: Increase the weight of the "logical rationality" dimension by 0.1 and decrease the weight of "emotional support" by 0.1.

[0165] Existing technologies rely on manual experience to set static weights, making it difficult to respond to personalized assessment needs. This application innovatively introduces a Bayesian optimization-driven interactive weight adjustment strategy, which transforms user natural language feedback (such as "increase the weight of the emotional support dimension") into quantitative adjustment signals. While maintaining weight normalization constraints, the regularization mechanism avoids excessive deviation from the initial reasonable range, effectively bridging the gap between assessment standards and user preferences. This allows the model's assessment standards to continuously evolve with user preferences, improving the acceptance and practicality of the scoring results.

[0166] S503: Based on the weights of each of the calibrated evaluation dimensions, adjust the scores of each of the evaluation dimensions to obtain the calibrated scores of each of the evaluation dimensions;

[0167] For details on the calculation method, please refer to the aforementioned content, which will not be repeated here.

[0168] S504: Based on the analysis reports of each of the evaluation dimensions and the calibrated scores of each of the evaluation dimensions, generate the reasoning chain for the model's answer.

[0169] In this application, the analysis reports of each of the evaluation dimensions and the calibrated scores of each of the evaluation dimensions can be combined to generate the reasoning chain of the model's answer.

[0170] Based on the above interactive weight calibration, in one possible implementation, the reward function used in the deep reinforcement learning also includes:

[0171] Regularization penalty, which indicates the degree of deviation of the calibrated weights of each of the evaluation dimensions from the initial weights of each of the evaluation dimensions.

[0172] For ease of understanding, its total reward function formula can be expressed as follows:

[0173] R=γ1R base +γ2R cog +γ3(1-L reg )

[0174] Where γ1, γ2, and γ3 are hyperparameters, determined through a grid search, R base Based on the reward, R cog For cognitive reward, L reg For regularization penalties.

[0175] For ease of understanding, the pseudocode for the core logic of policy network optimization is provided in this application, as shown in Table 5 below:

[0176] Table 5 Strategy Network Optimization

[0177]

[0178]

[0179] Specifically, π is the policy network to be optimized. The input (query, response, expert_label, expert_reasoning) is processed by TaskAnalysis to generate a set of evaluation dimensions, where expert_label is the expert-annotated score, expert_reasoning is the expert-annotated analysis, EvidenceReasoning constructs the inference chain, and predicted_score is the model's output score. The reward function consists of three parts: base_reward measures the matching degree between the score and the expert annotation, cog_reward evaluates the interpretability of the inference chain through ROUGE-L, and reg_penalty constrains the degree to which the weights deviate from their initial values. γ1, γ2, and γ3 are hyperparameters used to balance the importance of each reward item. The total reward drives the network parameter update through the PPO algorithm, realizing end-to-end training from task analysis to policy optimization, and finally outputting the optimized policy network π', which generates a more interpretable inference process while ensuring the accuracy of the scoring.

[0180] The above describes a policy network optimization method provided by the embodiments of this application. The following describes the apparatus for performing the above policy network optimization method.

[0181] Please see Figure 2 , Figure 2 This is a schematic diagram of a policy network optimization device provided in an embodiment of this application. Figure 2 As shown, the strategy network optimization device includes:

[0182] Acquisition unit 11 is used to acquire sample information, which includes query text, model answer and expert annotation information;

[0183] The inference chain generation unit 12 is used to generate the inference chain of the model's answer based on the query text;

[0184] The deep reinforcement learning unit 13 is used to perform deep reinforcement learning on the policy network based on the inference chain and the expert annotation information to obtain a target policy network. The reward function used in the deep reinforcement learning includes at least a basic reward and a cognitive reward. The basic reward is used to indicate the degree of matching between the overall prediction score of the policy network on the model's answer quality based on the inference chain and the expert annotation information. The cognitive reward is used to indicate the degree of matching between the inference chain and the expert annotation information in terms of the accuracy of evidence citation and the comprehensiveness of evaluation dimension analysis.

[0185] In one possible implementation, the inference chain generation unit includes:

[0186] The evaluation dimension set generation unit is used to generate an evaluation dimension set specific to the target task type corresponding to the query text, based on the task type corresponding to the query text.

[0187] The inference chain generation subunit is used to generate the inference chain of the model's answer based on the set of evaluation dimensions specific to the target task type.

[0188] In one possible implementation, the evaluation dimension set generation unit is specifically used for:

[0189] Perform task semantic understanding on the query text to determine the target task type corresponding to the query text;

[0190] The evaluation dimensions for the target task type are initialized to generate a set of evaluation dimensions specific to the target task type.

[0191] In one possible implementation, the inference chain generating sub-unit is specifically used for:

[0192] For each evaluation dimension, target evidence is retrieved from the professional knowledge base corresponding to the target task type; based on the target evidence and the model's response, an analysis report for the evaluation dimension is generated; based on the target evidence and the information of the evaluation dimension, a score for the evaluation dimension is generated.

[0193] Based on the analysis reports and scores of each of the evaluation dimensions, the reasoning chain for the model's answer is generated.

[0194] In one possible implementation, the inference chain generating sub-unit is specifically used for:

[0195] Obtain user feedback on their scores for each of the evaluation dimensions;

[0196] Based on the user's feedback on the scores of each of the evaluation dimensions, the initial weights of each of the evaluation dimensions are calibrated to obtain the calibrated weights of each of the evaluation dimensions.

[0197] Based on the calibrated weights of each of the evaluation dimensions, the scores of each of the evaluation dimensions are adjusted to obtain the calibrated scores of each of the evaluation dimensions.

[0198] Based on the analysis reports of each of the evaluation dimensions and the calibrated scores of each of the evaluation dimensions, the reasoning chain for the model's answer is generated.

[0199] In one possible implementation, the reward function used in the deep reinforcement learning also includes:

[0200] Regularization penalty, which indicates the degree of deviation of the calibrated weights of each of the evaluation dimensions from the initial weights of each of the evaluation dimensions.

[0201] In one possible implementation, the deep reinforcement learning unit is specifically used for:

[0202] Using the expert annotation information as a reference, the inference chain and the expert annotation information are compared at the text level, and the overlap evaluation index between the inference chain and the expert annotation information in a preset dimension is calculated as the matching degree.

[0203] The preset dimensions include any one or more of the following: key information coverage, logical structure similarity, and evidence citation matching degree.

[0204] Each unit in the aforementioned strategy network optimization device can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can invoke and execute the operations corresponding to each unit.

[0205] This application also provides an electronic device in its embodiments. (See reference...) Figure 3 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0206] like Figure 3As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0207] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0208] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the strategy network optimization methods provided in this application.

[0209] This application also provides a computer-readable storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the strategy network optimization methods provided in this application.

[0210] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0211] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0212] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0213] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for policy network optimization, the method comprising: The method comprises the following steps: Obtaining sample information, wherein the sample information comprises query text, model answers and expert annotation information; Generating an inference chain of the model answers based on the query text; Performing deep reinforcement learning on a strategy network based on the inference chain and the expert annotation information to obtain a target strategy network, wherein a reward function used in the deep reinforcement learning comprises at least a basic reward and a cognitive reward, the basic reward is used to indicate a matching degree between an overall prediction score of the model answers quality based on the inference chain and the expert annotation information, and the cognitive reward is used to indicate a matching degree between the inference chain and the expert annotation information in terms of evidence reference accuracy and evaluation dimension analysis comprehensiveness.

2. The method of claim 1, wherein, The method of generating the inference chain of the model answers based on the query text comprises the following steps: Generating an evaluation dimension set exclusive to a target task type corresponding to the query text based on the task type of the query text; Generating the inference chain of the model answers based on the evaluation dimension set exclusive to the target task type.

3. The method of claim 2, wherein, The method of generating the evaluation dimension set exclusive to the target task type corresponding to the query text based on the task type of the query text comprises the following steps: Performing task semantic understanding on the query text to determine the target task type corresponding to the query text; Performing evaluation dimension initialization on the target task type to generate the evaluation dimension set exclusive to the target task type.

4. The method of claim 2, wherein, The method of generating the inference chain of the model answers based on the evaluation dimension set exclusive to the target task type comprises the following steps: For each evaluation dimension, retrieving a target evidence from a professional knowledge base corresponding to the target task type, generating an analysis report of the evaluation dimension based on the target evidence and the model answers, and generating a score of the evaluation dimension based on information of the target evidence and the evaluation dimension; Generating the inference chain of the model answers based on the analysis report and the score of each evaluation dimension.

5. The method of claim 4, wherein, The method of generating the inference chain of the model answers based on the analysis report and the score of each evaluation dimension comprises the following steps: Obtaining feedback of a user on the score of each evaluation dimension; Calibrating an initial weight of each evaluation dimension based on the feedback of the user on the score of each evaluation dimension to obtain a calibrated weight of each evaluation dimension; Adjusting the score of each evaluation dimension based on the calibrated weight of each evaluation dimension to obtain a calibrated score of each evaluation dimension; Generating the inference chain of the model answers based on the analysis report of each evaluation dimension and the calibrated score of each evaluation dimension.

6. The method of claim 5, wherein, The reward function used in the deep reinforcement learning further comprises: A regularization penalty, which is used to indicate a deviation degree of the calibrated weight of each evaluation dimension from an initial weight of each evaluation dimension.

7. The method of claim 1, wherein, The matching degree of the inference chain and the expert annotation information in terms of evidence reference accuracy and evaluation dimension analysis comprehensiveness is determined in the following manner: With the expert annotation information as a reference benchmark, the reasoning chain is compared with the expert annotation information at a text level, and an overlap degree evaluation index of the reasoning chain and the expert annotation information in a preset dimension is calculated as the matching degree; The preset dimension includes any one or more of key information coverage, logical structure similarity, and evidence reference matching degree.

8. A computer program product, characterised in that, The computer-readable instructions, when executed on an electronic device, cause the electronic device to implement the policy network optimization method of any one of claims 1 to 7.

9. An electronic device, comprising: The memory is configured to store computer programs. The processor is configured to execute the computer programs to enable the electronic device to implement the policy network optimization method of any one of claims 1 to 7. The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the policy network optimization method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, ​

Images