Method and device for intelligent question and answer

Abstract

The invention provides a method and equipment for intelligent question answering, and relates to the technical field of computers and Internet. The method comprises the steps of obtaining a cue word template; obtaining a question and answer task and adopted response data and rejected response data corresponding to the question and answer task; determining a first judgment expansion sequence corresponding to the adopted response data and a second judgment expansion sequence corresponding to the rejected response data through a reward model in combination with the cue word template and the question and answer task; determining a system reward signal based on the first judgment expansion sequence and the second judgment expansion sequence; the system reward signal is used for executing reinforcement learning strategy gradient optimization to drive reward model parameter updating; wherein the reward model after parameter updating is used for performing automatic quality evaluation on question and answer data generated in the training process of the intelligent question and answer model, so as to perform parameter tuning on the intelligent question and answer model according to an evaluation result. According to the embodiment of the invention, the efficiency of the intelligent question-answering system can be improved at low cost.

Description

Technical Field

[0001] This disclosure relates to the field of computer and internet technology, and in particular to a method and electronic device for intelligent question answering. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] The training efficiency of existing intelligent question answering systems is highly dependent on the quality of the reward function. However, the mainstream technical approach faces a dual dilemma: while pairwise comparison-based models reduce human intervention, their output reward signals lack granularity, leading to ambiguous optimization directions and slow convergence; while point-based scoring models based on human annotation can provide absolute scores, their high annotation costs become a bottleneck for model iteration.

[0004] Therefore, the industry urgently needs a low-cost, efficient technical solution that can automatically generate fine-grained, highly adaptive reward signals. Summary of the Invention

[0005] The purpose of this disclosure is to provide a method, apparatus, electronic device, computer-readable storage medium, and computer program product for intelligent question answering, which can generate fine-grained and highly adaptive reward signals at low cost in order to improve the performance of intelligent question answering systems.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] This disclosure provides a method for intelligent question answering, comprising: obtaining a prompt word template, the prompt word template including a set of basic evaluation dimensions for response quality and instance dynamic evaluation meta-instructions; obtaining a question-answering task and pairwise preference data corresponding to the question-answering task, the pairwise preference data including accepted response data and rejected response data; based on the pairwise preference relationship between the accepted response data and rejected response data, through a preference-aware mechanism in a reward model, combined with the prompt word template and the question-answering task, determining a first judgment expansion sequence corresponding to the accepted response data and a second judgment expansion sequence corresponding to the rejected response data; the judgment expansion sequence including pointwise quality evaluation values; determining a system reward signal based on the first judgment expansion sequence and the second judgment expansion sequence; the system reward signal being used to perform gradient optimization of a reinforcement learning strategy to drive parameter updates of the reward model; wherein the parameter-updated reward model is used to perform automated quality evaluation of question-answer data pairs generated during the training of the intelligent question answering model, so as to fine-tune the parameters of the intelligent question answering model according to the evaluation results so that the intelligent question answering model is suitable for intelligent question answering tasks.

[0008] In some embodiments, based on the pairwise preference relationship between the accepted and rejected response data, and through the preference-aware mechanism in the reward model, combined with the prompt word template and the question-and-answer task, a first judgment expansion sequence corresponding to the accepted response data and a second judgment expansion sequence corresponding to the rejected response data are determined. This includes: the reward model performing n independent quality scores on the accepted response data based on the prompt word template and the question-and-answer task to obtain the first judgment expansion sequence; where n is an integer greater than 0; and the reward model performing n independent quality scores on the rejected response data based on the prompt word template and the question-and-answer task to obtain the second judgment expansion sequence.

[0009] In some embodiments, when the reward model performs n independent quality scores on the same response data, it employs random sampling or diversity sampling strategies to generate different judgment unfolding sequences.

[0010] In some embodiments, determining a system reward signal based on the first judgment expansion sequence and the second judgment expansion sequence includes: determining the average value of the point-based quality assessment values ​​in each of the first judgment expansion sequences to obtain an average score for accepted data; determining the average value of the point-based quality assessment values ​​in each of the second judgment expansion sequences to obtain an average score for rejected data; and determining the system reward signal based on the average score for accepted data and the average score for rejected data.

[0011] In some embodiments, determining the system reward signal based on the average score of accepted data and the average score of rejected data includes employing at least one of the following reward signal generation mechanisms: generating a reward signal based on the relative score difference between a single quality score and the average score of the comparison data, wherein: for accepted response data, a positive reward is allocated when the single score is higher than the average score of rejected data; for rejected response data, a positive reward is allocated when the single score is lower than the average score of accepted data; generating a reward signal based on the average score difference between the average score of accepted data and the average score of rejected data, wherein: a positive reward is allocated when the average score difference is greater than zero.

[0012] In some embodiments, the reward model performs n independent quality scores on the adopted response data based on the prompt word template and the question-and-answer task. The execution process of each independent quality score includes: the reward model generating personalized evaluation criteria based on the instance dynamic evaluation meta-instruction, the question-and-answer task, and the adopted response data, wherein the personalized evaluation criteria generated each time are different; the reward model integrating the basic evaluation dimension set of response quality and the personalized evaluation criteria to perform a quality score on the adopted response data, and obtain the score result of that independent quality score.

[0013] In some embodiments, the reward model generates personalized evaluation criteria based on the instance dynamic evaluation meta-instruction, the question-and-answer task, and the adopted response data, including: dynamically generating a personalized evaluation dimension set based on the instance dynamic evaluation meta-instruction, the question-and-answer task, and the adopted response data; dynamically configuring the evaluation granularity and weight in the personalized evaluation dimension set based on the type, complexity, and response data characteristics of the question-and-answer task; and generating the personalized evaluation criteria for the adopted response data based on the configured personalized evaluation dimension set.

[0014] In some embodiments, the reward model is optimized by using the system reward signal to obtain an optimized reward model, including: obtaining the scoring results of each independent quality score for the adopted response data; determining the accepted reward value for each independent score for the adopted response data based on the scoring results of each independent quality score; sorting the accepted reward values; and optimizing the parameters of the reward model based on the accepted reward values ​​in descending order.

[0015] In some embodiments, based on the prompt word template and the question-and-answer task, a first judgment unfolding sequence for the adopted response data is generated through a reward model, including: filling the question-and-answer task and the adopted response data into the prompt word template to obtain a target prompt word; and inputting the target prompt word into the reward model so that the reward model performs a quality score on the adopted response data based on the target prompt word.

[0016] This disclosure provides an electronic device comprising: a memory and a processor; the memory for storing computer program instructions; and the processor for calling the computer program instructions stored in the memory to implement the method for intelligent question answering as described above.

[0017] This disclosure provides a computer-readable storage medium having computer program instructions stored thereon to implement the method for intelligent question answering as described in any of the preceding embodiments.

[0018] This disclosure provides a computer program product or computer program that includes computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the aforementioned method for intelligent question answering.

[0019] The methods, apparatus, electronic devices, computer-readable storage media, and computer program products for intelligent question answering provided in this disclosure utilize prompt word templates to drive a reward model to generate personalized evaluation criteria that are adaptive to the task, thereby achieving fine-grained quantitative scoring of answer quality. By comparing the scoring results of accepted and rejected answers, accurate reward signals are automatically generated, significantly improving the optimization efficiency and directional accuracy of the strategy model. Ultimately, a closed-loop system of collaborative optimization between the reward model and the intelligent question answering model is formed, significantly improving the response quality and training efficiency of the intelligent question answering system without requiring extensive manual annotation.

[0020] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] Figure 1A schematic diagram of a scenario for a method or apparatus for intelligent question answering that can be applied to embodiments of this disclosure is shown.

[0023] Figure 2 This is a schematic diagram illustrating a comparison of paired and point defects according to an exemplary embodiment.

[0024] Figure 3 This is a flowchart illustrating a method for intelligent question answering according to an exemplary embodiment.

[0025] Figure 4 This is a flowchart illustrating a method for intelligent question answering according to an exemplary embodiment.

[0026] Figure 5 This is a prompt word template shown according to an exemplary embodiment.

[0027] Figure 6 This is a flowchart illustrating a reward signal determination method according to an exemplary embodiment.

[0028] Figure 7 This is a parameter tuning method illustrated according to an exemplary embodiment.

[0029] Figure 8 This is a diagram illustrating an adaptive reward model training framework according to an exemplary embodiment.

[0030] Figure 9 This is a block diagram illustrating an apparatus for intelligent question answering according to an exemplary embodiment.

[0031] Figure 10 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0032] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0033] Those skilled in the art will recognize that embodiments of this disclosure can be a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0034] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0035] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0036] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0037] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0038] In the description of this disclosure, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences; the terms "contains," "includes," and "has" are used to indicate an open-ended meaning of inclusion and refer to the existence of additional elements / components / etc. besides those listed.

[0039] To better understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0040] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0041] Figure 1 A schematic diagram of a scenario for a method or apparatus for intelligent question answering that can be applied to embodiments of this disclosure is shown.

[0042] Please refer to Figure 1 The diagram illustrates an implementation environment provided by an exemplary embodiment of this disclosure.

[0043] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0044] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, virtual reality devices, smart home devices, etc.

[0045] Server 105 can be a server that provides various services, such as a backend management server that supports the devices operated by users using terminal devices 101, 102, and 103. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal devices.

[0046] A server can be a standalone physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This disclosure does not impose any restrictions on this.

[0047] Server 105 may, for example, acquire a prompt word template, which includes a set of basic evaluation dimensions for response quality and instance dynamic evaluation meta-instructions; acquire a question-and-answer task and corresponding pairwise preference data, which includes accepted response data and rejected response data; based on the pairwise preference relationship between the accepted and rejected response data, through a preference-aware mechanism in the reward model, combined with the prompt word template and the question-and-answer task, determine a first judgment expansion sequence corresponding to the accepted response data and a second judgment expansion sequence corresponding to the rejected response data; the judgment expansion sequence includes pointwise quality evaluation values; based on the first and second judgment expansion sequences, determine a system reward signal; the system reward signal is used to perform gradient optimization of the reinforcement learning strategy to drive the update of the reward model parameters; wherein the parameter-updated reward model is used to automatically evaluate the quality of question-and-answer data pairs generated by the intelligent question-and-answer model during training, so as to fine-tune the parameters of the intelligent question-and-answer model according to the evaluation results so that the intelligent question-and-answer model is suitable for intelligent question-and-answer tasks.

[0048] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Server 105 can be a single physical server or a combination of multiple servers. Depending on actual needs, it can have any number of terminal devices, networks, and servers.

[0049] Under the above system architecture, this disclosure provides a method for intelligent question answering, which can be executed by any electronic device with computing power.

[0050] Reinforcement learning from human feedback (RLHF): Its core objective is to adjust the behavior of an existing artificial intelligence model (such as a large language model) so that its output is more in line with human value judgments and preferences.

[0051] In some embodiments, reinforcement learning that aligns with human preferences can be fully understood through a classic "coach-athlete" analogy.

[0052] Athlete: That's the AI ​​model. It has already completed "pre-training" on massive amounts of text, just like an athlete who has strong muscles and basic motor skills, knows how to run and jump, but doesn't understand the specific rules of the game or winning strategies.

[0053] The ultimate goal is not simply to make athletes run faster or jump higher, but to make them win the competition—this "winning the competition" corresponds to "conforming to human preferences".

[0054] To achieve the above goals, reinforcement learning processes that align with human preferences typically involve three key stages.

[0055] Step 1: Supervision and fine-tuning – learning the basics (imitating champions).

[0056] Method: Hire a champion coach (human annotator) to demonstrate standard, graceful technical movements (providing high-quality question-and-answer pair data). The athlete (base model) adjusts their movements by imitating these "standard answers".

[0057] Objective: To enable the model to initially grasp "how to answer questions correctly" and become a basic "assistant".

[0058] Step 2: Training the reward model – cultivating referees / coaches (learning taste).

[0059] Objective: To train an independent reward model that learns human evaluation criteria. Once trained, this reward model will be able to act like a coach, observing any action of an athlete (the model-generated response) and assigning a score (reward) based on human preferences.

[0060] Limitations: There are countless kinds of questions in the world, and we cannot prepare "standard answers" for all of them. Moreover, it is not enough to simply be "correct"; we also need it to be "useful," "harmless," and "appropriate in speech."

[0061] Step 3: Reinforcement Learning and Fine-tuning – Practical Training (Evolving under the guidance of a coach)

[0062] Procedure: Now let the athletes (models) enter the real training field.

[0063] It encountered a new problem (hint).

[0064] It generates an answer.

[0065] The reward model (coach) immediately scores the answer.

[0066] The model is updated based on this score using a reinforcement learning algorithm: if the score is high, the behavior of generating this answer is reinforced; if the score is low, the behavior is weakened.

[0067] Objective: Through thousands of such cycles, the model (athlete) is continuously shaped by the feedback from the reward model (coach), eventually becoming very good at generating answers that can get high scores—that is, answers that are highly consistent with human preferences.

[0068] Reward models (RM) play a crucial role in reinforcement learning human feedback (RLHF), aiming to align the behavior of large models with human preferences. However, traditional pairwise training methods only provide binary "good" or "bad" labels, which are disconnected from pointwise reasoning capabilities; while pointwise training relies on expensive and rigid absolute rating systems. To address these issues, this application innovatively proposes a preference-aware task-adaptive reward model (PaTaRM) training framework. This framework directly constructs pointwise training signals by utilizing relative preferences in pairwise data, without the need for additional pointwise labels; simultaneously, it dynamically generates rating criteria adapted to the task and instances. PaTaRM is not only efficient but also possesses broad generalization ability and interpretability, enabling the generation of effective pointwise training signals during reinforcement learning training without explicit labels, thereby significantly improving the performance of reward models.

[0069] Point-based reasoning ability: refers to the model's ability to independently and absolutely evaluate and reason about the quality of a single answer and assign it a specific and meaningful score.

[0070] Pairwise comparison: Given two different answers (A and B) to a question (hint), the model only needs to determine "Is A better than B?"

[0071] Specifically, in the Large Language Model (LLM) alignment process, the reward model (RM) is used to evaluate the quality of text generated by the policy model (such as an intelligent question-answering system), providing supervision signals for subsequent reinforcement learning (RLHF) alignment of human preferences. Industry-leading solutions fall into two main categories.

[0072] (a) Traditional BT (Bradley-Terry Model) reward model.

[0073] Technical principle: Based on human preference data, pairs of comparison samples are constructed, and a reward model is trained to learn the relative preference relationship, which can output a score for a response.

[0074] Implementation: Remove the original output layer of the LLM, add a reward head, input two candidate responses during training, output relative preference scores, optimize using Bradley-Terry loss, and train the data using the preferences.

[0075] Output format: Given a text, output the reward score directly.

[0076] (ii) Generative Reward Models (GRMs).

[0077] Technical principle: Utilize the generative capabilities of large language models to output structured scoring feedback instead of a single numerical value.

[0078] Implementation: Transform reward modeling into a text generation task to generate detailed feedback containing the reasons for the rating.

[0079] Output format: Given a text, generate a structured text containing the scoring criteria and specific scores.

[0080] Within the GRM framework, models can be further categorized into two types based on the input and output formats.

[0081] 1. Pointwise reward model: Input a response, and this model can assign an absolute quality score to each response; 2. Pairwise reward model: Two candidate responses are input into the prompt, and the model outputs which response is better.

[0082] Figure 2 This is a schematic diagram illustrating a comparison of paired and point defects according to an exemplary embodiment.

[0083] refer to Figure 2 The existing technology has the following limitations.

[0084] 1. Limitations of BT Reward Models: While scalar reward models (RM) are effective for aligning basic preferences, they also have significant limitations: they cannot fully utilize the generative and inference capabilities of LLMs. They often capture superficial correlations rather than genuine human preferences. Furthermore, they are prone to overfitting and are sensitive to changes in distribution.

[0085] 2. Limitations of GRM Pair Training Methods: While traditional pair training methods can learn relative preference relationships through binary contrast, their inherent relative comparison characteristics mean they can only provide a comparison of "which is better," rather than giving specific absolute quality scores. This limitation is particularly prominent in practical deployment scenarios, as many real-world applications require explicit numerical scores for decision-making and ranking, rather than simple relative comparisons.

[0086] 3. Limitations of the GRM Point-Based Training Method: While point-based training can provide absolute quality assessment, it faces significant practical challenges. First, this method requires extensive manual annotation to assign absolute scores to each response, which not only consumes enormous human and time resources but is also difficult to maintain consistently in large-scale applications. Second, existing point-based training heavily relies on predefined static scoring criteria. These rigid criteria lack adaptability to different tasks and scenarios, making it difficult to handle diverse scoring requirements. More seriously, different annotators have subjective differences in their understanding and judgment criteria for absolute scores, leading to poor annotation consistency and consequently affecting the stability and reliability of model training.

[0087] In related technologies, generative reward models face significant challenges in integrating pairwise training and pointwise training methods. The fundamental differences between the two training paradigms make it difficult for existing technologies to effectively bridge this gap. The lack of a mechanism for dynamically adjusting scoring criteria prevents the adaptive generation of scoring rules based on specific task characteristics and response content, resulting in insufficient adaptability in complex and ever-changing real-world application scenarios. Furthermore, existing technologies offer limited performance improvements, especially when dealing with complex tasks requiring fine-grained scoring and multi-dimensional considerations, often failing to achieve ideal results.

[0088] Figure 3 This is a flowchart illustrating a method for intelligent question answering according to an exemplary embodiment. The method provided in this disclosure can be executed by any electronic device with computing power, for example, the method can be executed by the above-described... Figure 1 The execution can be performed by a server or terminal device in the embodiments, or it can be performed by both a server and a terminal device. In the following embodiments, the server is used as the execution subject for illustration, but this disclosure is not limited to this.

[0089] Reference Figure 3 The method for intelligent question answering provided in this disclosure may include the following steps.

[0090] The aforementioned intelligent question-answering method can be an intelligent question-answering method in human-computer interaction systems, such as human-computer dialogue interaction systems and customer service interaction systems.

[0091] Step S302: Obtain the prompt word template, which includes a set of basic assessment dimensions for response quality and instance dynamic assessment meta-instructions.

[0092] The basic evaluation dimension set for response quality is a predefined set of core quality dimensions applicable to the vast majority of question-and-answer scenarios. It serves as the foundational and unchanging evaluation framework for the reward model.

[0093] The basic assessment dimensions for response quality can be a general, fundamental quality checklist. It is a set of predefined, universally applicable assessment criteria that provides a stable and consistent basis for judging the reward model (RM).

[0094] In some embodiments, the basic evaluation dimension set of response quality can be a general quality evaluation standard, which can be further divided into multiple sub-standards according to the question-and-answer task type. Each sub-standard corresponds to a task type. For example, for task types such as mathematical tasks, chat tasks, and instruction-following tasks, there can be a corresponding sub-standard. The sub-standard corresponding to mathematical tasks may be more related to the correctness of logical reasoning, while chat tasks may be more concerned with the usefulness and completeness of the answer.

[0095] The instance dynamic evaluation meta-instruction is a strategy instruction built into the prompt word template. Its function is to guide the reward model to dynamically generate a set of personalized evaluation criteria for the currently evaluated question-answer task-response data pair. This criterion serves as a refinement and expansion of the basic evaluation dimension set of response quality, and is used to achieve accurate and contextualized quality evaluation of the instance.

[0096] The instance dynamic evaluation meta-instruction tells the reward model "how to dynamically generate a set of evaluation criteria based on the current specific question-and-answer instance (i.e. the object being evaluated)".

[0097] In some embodiments, the reward model can combine instance dynamic evaluation meta-instructions to dynamically generate a set of personalized evaluation criteria for question-answering task-response data pairs.

[0098] Personalized evaluation criteria can be a set of temporary, highly customized evaluation rules that are dynamically generated by the reward model in a single independent evaluation, based on the guidance of the instance-based dynamic evaluation meta-instruction, combined with the specific question-and-answer task and the evaluated response data.

[0099] Figure 5 This is a prompt word template shown according to an exemplary embodiment.

[0100] refer to Figure 5 The above prompt word template can be at least one of modules 501 to 506.

[0101] Specifically, section 501 is used to assign the question-and-answer task; section 502 is used to set the basic evaluation dimension set for response quality, which can be placed directly in the prompt word template or filled in when used, and this application does not impose any restrictions on this; section 503 is used to fill in the question-and-answer task and the data of accepted and / or rejected responses corresponding to the question-and-answer task; section 504 is used to describe how to determine the final score based on the basic evaluation dimension set for response quality and the instantiated evaluation criteria; and sections 505 and 506 are used for the quality evaluation output format.

[0102] refer to Figure 5 As can be seen, the scoring criteria designed in this application consist of two parts: a set of global evaluation criteria consistent with the task (a set of basic evaluation dimensions for response quality) and a set of criteria specific to each instance (instantiated evaluation criteria). The reward model can dynamically construct instantiated evaluation criteria for each prompt.

[0103] In some embodiments, global evaluation criteria capture general requirements such as correctness, relevance, and safety, ensuring consistency across the entire dataset. Instance-specific criteria are generated based on the specific context of each prompt and candidate answer, enabling fine-grained reasoning and tailored scoring.

[0104] Step S302: Obtain the question-and-answer task and the corresponding pairwise preference data. The pairwise preference data includes accepted response data and rejected response data.

[0105] Paired preference data refers to a set of response data corresponding to the same question-and-answer task, containing a set of responses with relative quality. This set consists of accepted response data and rejected response data, where accepted response data represents outputs that are relatively more in line with quality requirements, and rejected response data represents outputs that are relatively less in line with quality requirements. The two form a comparative sample through preference relationships.

[0106] The question-answering task can be any specific question or instruction that you want the model (such as the policy model in this application) to complete.

[0107] "Question-answering task" is a broad concept here; it is not just about "asking and answering," but can refer to any task that a policy model is expected to accomplish (including text generation tasks). It usually appears in the form of a "hint" or "instruction."

[0108] Example.

[0109] Knowledge-based question: "Explain what photosynthesis is?"

[0110] Creative task: "Write a five-character quatrain about the moon."

[0111] Reasoning task: "If Xiaoming is taller than Xiaohong, and Xiaohong is taller than Xiaohua, then who is the tallest?"

[0112] The instruction follows the task: "Translate the following sentence into English: The weather is really nice today."

[0113] Chat-style task: "I've been feeling a lot of work stress lately, can you give me some advice?"

[0114] The above-mentioned question-and-answer task may include not only text content, but also any one or more of multimedia information such as voice, images, and videos. This application does not impose any restrictions on this.

[0115] Accepted response data: For a given question-and-answer task, among multiple candidate answers, the one that is labeled (e.g., by human labelers) as better and more desirable. Characteristics: Usually more accurate, more helpful, less harmful, and more in line with human values ​​and instructions.

[0116] Rejected responses: For the same question-and-answer task, the response that is judged (e.g., by a human annotator) as worse or unacceptable in a comparison. Characteristics: May contain factual errors, logical inconsistencies, irrelevant answers, or simply be of lower quality than accepted responses.

[0117] The aforementioned accepted and / or rejected response data may include not only text content, but also any one or more multimedia information such as voice, images, and videos; this application makes no restriction on this.

[0118] In some embodiments, it is necessary to obtain the accepted response data and rejected response data corresponding to the question-and-answer task in pairs.

[0119] Step S304: Based on the pairwise preference relationship between the accepted response data and the rejected response data, the first judgment expansion sequence corresponding to the accepted response data and the second judgment expansion sequence corresponding to the rejected response data are determined through the preference perception mechanism in the reward model, combined with the prompt word template and the question-and-answer task; the judgment expansion sequence includes point-based quality assessment values.

[0120] Point-based quality assessment values ​​refer to a specific, quantified quality score output by the reward model in a single independent evaluation for a given response (whether accepted or rejected), based on a prompt word template and a question-and-answer task.

[0121] In some embodiments, a first decision expansion sequence for accepted response data and a second decision expansion sequence for rejected response data can be generated using a reward model based on prompt word templates and question-and-answer tasks.

[0122] A decision unfolded sequence refers to the complete data sequence obtained by a reward model after performing n independent quality assessments on a single response (whether accepted or rejected), encompassing all assessment processes and results. The core component of this sequence is n point-based quality assessment values, where n is an integer greater than or equal to 1.

[0123] Step S306: Based on the first judgment unfolding sequence and the second judgment unfolding sequence, determine the system reward signal; the system reward signal is used to perform reinforcement learning policy gradient optimization to drive the update of reward model parameters; wherein the reward model with updated parameters is used to perform automated quality evaluation on the question-answer data pairs generated by the intelligent question answering model during training, so as to fine-tune the parameters of the intelligent question answering model according to the evaluation results so that the intelligent question answering model is suitable for intelligent question answering tasks.

[0124] In some embodiments, if the point-based quality assessment value in the first judgment expansion sequence is greater than the point-based quality assessment value in the second judgment expansion sequence, then the system reward signal is configured as a positive reward signal (e.g., greater than 0); otherwise, a negative reward signal (e.g., less than or equal to 0) is configured.

[0125] In some embodiments, if the average value of the point-based quality assessment values ​​in the first judgment unfolding sequence is greater than the average value of the point-based quality assessment values ​​in the second judgment unfolding sequence, then a reward signal is configured to be greater than 0; otherwise, a negative reward signal is configured.

[0126] This disclosure provides a reinforcement learning-based intelligent question answering model optimization method. By introducing a prompt word template containing a basic evaluation dimension set and dynamic evaluation meta-instructions, the reward model is guided to generate judgment sequences containing multi-review evaluations for each adoption / rejection response in pairwise preference data. By comparing the statistical features between the sequences, a system reward signal is generated to drive the update of the reward model parameters, and finally a closed loop for intelligent question answering training that can achieve automated quality evaluation and continuous optimization is constructed.

[0127] Figure 4 This is a flowchart illustrating a method for intelligent question answering according to an exemplary embodiment. The method provided in this disclosure can be executed by any electronic device with computing power, for example, the method can be executed by the above-described... Figure 1 The execution can be performed by a server or terminal device in the embodiments, or it can be performed by both a server and a terminal device. In the following embodiments, the server is used as the execution subject for illustration, but this disclosure is not limited to this.

[0128] Reference Figure 4 The method for intelligent question answering provided in this disclosure may include the following steps.

[0129] The aforementioned intelligent question-answering method can be an intelligent question-answering method in human-computer interaction systems, such as human-computer dialogue interaction systems and customer service interaction systems.

[0130] Step S402: Obtain the prompt word template, which includes a set of basic assessment dimensions for response quality and instance dynamic assessment meta-instructions.

[0131] Step S404: Obtain the question-and-answer task and the accepted and rejected response data corresponding to the question-and-answer task.

[0132] Step S406: Based on the prompt word template and the question-and-answer task, a first judgment expansion sequence for the accepted response data and a second judgment expansion sequence for the rejected response data are generated through the reward model.

[0133] In some embodiments, based on the prompt word template and the question-and-answer task, a first judgment unfolding sequence for the adopted response data is generated through a reward model, including: filling the prompt word template (e.g., the question-and-answer task and the adopted response data) with the prompt word template. Figure 5 From the bolded portion of the prompt word template shown, obtain the target prompt word; input the target prompt word into the reward model so that the reward model can score the quality of the adopted response data based on the target prompt word.

[0134] In some embodiments, after receiving the target prompt, the reward model will dynamically evaluate the meta-instruction based on the instance, and combine the adopted response data and / or rejected response data to generate an instanced evaluation criterion for the adopted response data or rejected response data. One of the response tasks - adopted response data / rejected response data can generate an instanced evaluation criterion.

[0135] In some embodiments, the reward model can combine a general response instruction scoring standard and an instantiated evaluation standard generated by the reward model to score adopted response data and rejected response data separately.

[0136] In some embodiments, based on the prompt word template and the question-and-answer task, a reward model is used to generate a first judgment expansion sequence for the accepted response data and a second judgment expansion sequence for the rejected response data. This includes: the reward model performs n independent quality scores on the accepted response data based on the prompt word template and the question-and-answer task to obtain the first judgment expansion sequence; where n is an integer greater than 0; and the reward model performs n independent quality scores on the rejected response data based on the prompt word template and the question-and-answer task to obtain the second judgment expansion sequence.

[0137] In some embodiments, the quality scores of accepted responses can be performed independently n times, meaning that each quality score can be independent of the others. The scoring results for rejected responses are the same.

[0138] In some embodiments, the reward model performs n independent quality scores on the adopted response data based on prompt word templates and question-and-answer tasks. The execution process of each independent quality score includes: the reward model dynamically evaluates the meta-instructions, question-and-answer tasks, and adopted response data based on the instance, and generates personalized evaluation criteria, wherein the personalized evaluation criteria generated each time are different; the reward model integrates the basic evaluation dimension set of response quality and the personalized evaluation criteria to perform a quality score on the adopted response data, and obtains the score result of that independent quality score.

[0139] Personalized evaluation criteria can be a set of temporary, highly customized evaluation rules that are dynamically generated by the reward model in a single independent evaluation, based on the guidance of the instance-based dynamic evaluation meta-instruction, combined with the specific question-and-answer task and the evaluated response data.

[0140] In some embodiments, given a cue x and candidate responses (selected response yc or rejected response yr), the reward model generates n decision unfolding sequences. (The sequence of n judgments for the adopted response data) and (An unfolded sequence of n judgments for rejected response data), where i is a positive integer, j is a positive integer, and n is a positive integer. The core of this mechanism lies in generating different perspectives through multiple samplings, thereby obtaining a more stable and comprehensive quality score.

[0141] Step S408: Determine the system reward signal based on the first judgment expansion sequence and the second judgment expansion sequence; when the point quality assessment value in the first judgment expansion sequence is greater than the point quality assessment value in the second judgment expansion sequence, the system reward signal is a positive reinforcement signal; when the point quality assessment value in the first judgment expansion sequence is less than or equal to the point quality assessment value in the second judgment expansion sequence, the system reward signal is a negative correction signal.

[0142] Step S410: The reward model is optimized by using the system reward signal to obtain an optimized reward model. The optimized reward model is used to automatically evaluate the quality of the question-and-answer data pairs generated by the intelligent question-and-answer model during training, so as to optimize the parameters of the intelligent question-and-answer model according to the evaluation results so that the intelligent question-and-answer model is suitable for intelligent question-and-answer tasks.

[0143] In some embodiments, the optimized reward model can be used to automatically evaluate the quality of question-answer data pairs generated during the training of the intelligent question-answering model (such as a policy model), so as to fine-tune the parameters of the intelligent question-answering model based on the evaluation results so that the intelligent question-answering model is suitable for intelligent question-answering tasks.

[0144] The above method can convert pairwise preference data into pointwise training signals without explicit labels.

[0145] This disclosure presents an innovative reward model training method that achieves low-cost, high-efficiency optimization of intelligent question-answering models. Its core effect lies in utilizing readily available pairwise preference data (accepted and rejected responses) and dynamically generated instantiated scoring criteria to enable the reward model to automatically generate absolute quality scores for answers. The system then generates reward signals by comparing the scores of good and bad answers. This mechanism eliminates the need for expensive and subjective manual absolute scoring, significantly reducing annotation costs and substantially improving the discriminative power, stability, and interpretability of the reward model. Ultimately, it provides reliable guidance for the accurate and efficient alignment of downstream intelligent question-answering models.

[0146] Figure 6 This is a flowchart illustrating a reward signal determination method according to an exemplary embodiment.

[0147] refer to Figure 6 The above-mentioned reward signal determination method may include the following steps.

[0148] Step S602: Determine the average value of the point-based quality assessment values ​​in each first judgment unfolding sequence to obtain the average score of the accepted data.

[0149] Step S604: Determine the average value of the point-based quality assessment values ​​in each second judgment expansion sequence to obtain the average score of the rejection data.

[0150] In some embodiments, for each selected response and the response of rejection This can generate n decision expansion sequences. Each expansion sequence is scored using an adaptive scoring criterion to obtain a point-based quality assessment value. The i-th unfolded sequence generates a score. ,for The j-th expanded sequence generates a score. Then calculate the average score for each response: , Where i is a positive integer, j is a positive integer, and n is a positive integer. This averaging process effectively reduces the randomness of a single score and improves the reliability of the scoring.

[0151] Step S606: Determine the system reward signal based on the average score of accepted data and the average score of rejected data.

[0152] In some embodiments, determining the system reward signal based on the average score of the accepted data and the average score of the rejected data includes employing at least one of the following reward signal generation mechanisms.

[0153] 1. A reward signal is generated based on the relative score difference between a single quality score and the average score of the comparison data. Specifically: for accepted response data, a positive reward is assigned when the single score is higher than the average score of the rejected data; for rejected response data, a positive reward (e.g., greater than 0) is assigned when the single score is lower than the average score of the accepted data, otherwise a negative reward (e.g., less than or equal to 0) is assigned. "Comparison data" refers to another set of data in the pairwise preference data. When evaluating a single score of accepted response data, the comparison data refers to rejected response data. When evaluating a single score of rejected response data, the comparison data refers to accepted response data.

[0154] 2. A reward signal is generated based on the average difference between the average score of the accepted data and the average score of the rejected data, wherein: when the average difference is greater than zero, a positive reward (e.g., greater than 0) is allocated; otherwise, a negative reward (e.g., less than or equal to 0) is allocated. In some embodiments, the first judgment expansion sequence may include a first first judgment expansion sequence obtained through the i-th quality score, and the system reward signal may include a first accepted reward signal corresponding to the first first judgment expansion sequence, where i is an integer greater than or equal to 0 and less than or equal to n.

[0155] Below, this application will use the first acceptance reward signal corresponding to the first first judgment expansion sequence as an example to explain how to obtain the reward signal corresponding to each first judgment expansion sequence one by one.

[0156] In some embodiments, determining the system reward signal based on the average score of accepted data and the average score of rejected data may include: determining the difference between the point-based quality assessment value in the first first judgment unfolding sequence and the average score of rejected data as a first difference; if the first difference is less than or equal to 0, then setting the first accepted reward signal to 0; if the first difference is greater than 0, then setting the first accepted reward signal to a constant greater than 0 or setting it to a function value positively correlated with the first difference.

[0157] Below, this application will use the first acceptance reward signal corresponding to the first first judgment expansion sequence as an example to explain how to obtain the reward signal corresponding to each first judgment expansion sequence one by one.

[0158] In some embodiments, the second judgment expansion sequence includes a first and second judgment expansion sequence obtained through the j-th quality score, and the system reward signal includes a first rejection reward signal corresponding to the first and second judgment expansion sequences, where j is an integer greater than or equal to 0 and less than or equal to n.

[0159] In some embodiments, determining the system reward signal based on the average score of accepted data and the average score of rejected data includes: determining the difference between the point-based quality assessment value in the first and second judgment expansion sequences and the average score of accepted data as a second difference; if the second difference is greater than or equal to 0, then setting the second acceptance reward signal to 0; if the second difference is less than 0, then setting the second acceptance reward signal to a constant greater than 0 or to a function positively correlated with the absolute value of the second difference.

[0160] Based on the above methods, this application proposes a preference-aware reward allocation mechanism: for each unfolded sequence, a reward is allocated based on its relative rating. The specific implementation formula is as follows: , ,in and Indicates the marginal score. It is an indicator function. It can be any general function (such as an exponential function) that is a constant or a scoring margin.

[0161] In some embodiments, the system reward signal includes a general reward signal. Determining the system reward signal based on the average score of accepted data and the average score of rejected data may include: determining a third difference between the average score of accepted data and the average score of rejected data; if the third difference is less than or equal to 0, setting the general reward signal to 0; if the third difference is greater than 0, setting the general reward signal to a constant greater than 0 or to a function value positively correlated with the third difference.

[0162] The goal of the above method is to ensure that the margin between the average score assigned to the accepted responses and the average score assigned to the rejected responses is positive. .

[0163] In some embodiments, after obtaining the aforementioned reward, the generative reward model can be trained end-to-end using policy gradient methods (such as GRPO, Reinforce++, or DAPO) to ensure that its output always reflects human preferences captured by the relative rating signal, without providing an absolute true score for each response.

[0164] The reward signal determination method proposed in this disclosure calculates the average score of accepted and rejected responses as a benchmark and dynamically allocates reward signals based on the difference between a single score and the opposing average score. This achieves three core technical effects: First, through multiple sampling and averaging processes, the stability and reliability of the scoring signal are significantly improved, effectively suppressing random fluctuations. Second, a preference-aware mechanism is introduced, transforming pairwise comparisons of preference relationships into refined point-based rewards, ensuring that the model can clearly distinguish between high-quality and low-quality responses. Third, an end-to-end optimization objective is constructed, enabling the reward model to autonomously learn and internalize human judgment standards through relative preference signals without requiring absolute true scores, thereby significantly improving the alignment efficiency and generation quality of downstream intelligent question-answering models.

[0165] Figure 7 This is a parameter tuning method illustrated according to an exemplary embodiment.

[0166] refer to Figure 7 The above parameter tuning method may include the following steps.

[0167] Step S702: Based on the scoring results of each independent quality score of the adopted response data.

[0168] Step S704: Determine the accepted reward value for each independent score of the adopted response data based on the score results of each independent quality score.

[0169] Step S706: Sort the received reward values.

[0170] Step S708: Optimize the parameters of the reward model based on the received reward value, in descending order.

[0171] This disclosure introduces a directional parameter tuning mechanism based on reward value ranking, prioritizing the learning of high-quality rating samples and effectively focusing on the direction of model optimization. This significantly accelerates training convergence while reducing noise interference. This innovative method not only improves the accuracy and robustness of the reward model's discrimination ability, but also enhances the stability of its preference perception, providing key support for the accurate alignment and performance breakthrough of downstream intelligent question answering models.

[0172] This application innovatively proposes a preference-aware task-adaptive reward model (PaTaRM) training framework. The following will explain and illustrate the above-mentioned adaptive reward model framework with specific embodiments.

[0173] Figure 8 This is a diagram illustrating an adaptive reward model training framework according to an exemplary embodiment.

[0174] Figure 8 This paper demonstrates the overall process of the reward model PataRM, which bridges the gap between pairwise and point-to-point GRMs through a preference-aware reward (PAR) mechanism and task-adaptive scoring criterion generation. The system utilizes relative preference signals from pairwise data to construct robust pointwise training signals, while dynamic scoring criterion adaptation flexibly generates scoring standards, satisfying both global task consistency and instance-specific inference requirements.

[0175] Figure 8 The framework shown has an upper section displaying the task-adaptive scoring criterion generation for inference, while the lower section describes the point-based training procedure, which includes dynamic scoring criterion adaptation and a preference-aware reward (PAR) mechanism.

[0176] In some embodiments, this application also provides a method applicable to the above-described training framework for an adaptive reward model for preference-aware tasks, the specific steps of which are shown below.

[0177] Step 1: Data preparation and preprocessing.

[0178] 1.1 Implement data cleaning and quality control processes to ensure the consistency and reliability of preference pairs.

[0179] 1.2 Establish a unified data format standard, separate preference pairs, and require a set of preference pairs to form two data entries, which should be placed adjacent to each other in the data file.

[0180] 1.3 Place the processed data into the prompt template, as shown in the prompt template below. Figure 5 As shown.

[0181] Step 2: Model initialization and basic capability pre-training.

[0182] 2.1 Implement the Supervised Fine-Tuning (SFT) phase to lay the foundation for subsequent reinforcement learning training based on the preference-aware reward mechanism. This ensures that the model maintains its original language generation quality while acquiring preference modeling capabilities.

[0183] Step 3: PAR-based reinforcement learning.

[0184] 3.1 Implementation principle.

[0185] Generative decision unpacking mechanism: The reward model is designed as a generative reward model. Given a cue x and candidate responses (selected response yc or rejected response yr), the reward model generates n decision unpacking sequences. (The sequence of n judgments for the adopted response data) and (An unfolded sequence of n judgments for rejected response data), where i is a positive integer, j is a positive integer, and n is a positive integer. The core of this mechanism lies in generating different perspectives through multiple samplings, thereby obtaining a more stable and comprehensive quality score.

[0186] Rating extraction mechanism: For each selection response and the response of rejection This can generate n judgment expansion sequences. Each expansion sequence is scored using an adaptive scoring criterion. The i-th unfolded sequence generates a score. ,for The j-th expanded sequence generates a score. Then calculate the average score for each response: , This averaging process effectively reduces the randomness of a single score and improves the reliability of the rating.

[0187] Optimization Objective Design: The reward model is directly optimized through reinforcement learning, using a preference-aware reward mechanism as the training signal. Specifically, the objective is to ensure a positive margin between the average score assigned to accepted responses and the average score assigned to rejected responses. This design enables generative reward models to be trained end-to-end using policy gradient methods such as GRPO, Reinforce++, or DAPO, ensuring that their outputs always reflect human preferences captured by relative rating signals, without requiring an absolute true score for each response.

[0188] Preference-aware reward allocation mechanism: For each unfolded sequence, a reward is allocated based on its relative rating. The specific implementation formula is as follows: , ,in and Indicates the marginal score. It is an indicator function. It can be any general function, whether it is a constant or a scoring margin.

[0189] Step 4: Training and Convergence Determination.

[0190] 4.1 When the model's performance converges on the validation set or reaches a preset threshold, training is terminated, and the final model is obtained.

[0191] Step 5: Model Deployment and Inference Application.

[0192] 5.1 Deploying PaTaRM for inference enables scoring of response quality.

[0193] 5.2 Integrating PaTaRM into downstream RLHF experiments can provide reward signals to guide policy model updates.

[0194] The above method comprises three steps: Step 1: Establishing a dynamic scoring criterion generation mechanism that combines global task consistency criteria and instance-specific criteria. Based on specific task characteristics and response content, it dynamically generates 1-3 additional criteria to supplement dimensions not covered by the main scoring criteria. Step 2: Utilizing the inherent reasoning capabilities of the large language model, it automatically balances the importance of different criteria without requiring manual setting of weight parameters. The granularity and focus of the criteria are adjusted in real time based on the complexity of the prompt words, task type, and response characteristics. Step 3: During the reinforcement learning training process of the reward model, pairwise comparison data is converted into point-based training signals to ensure... ^c > The optimization objective of ^r is to ensure that the model output always reflects human preferences.

[0195] The above methods have at least some beneficial technical effects.

[0196] 1. Technical Advantages of Task-Adaptive Dynamic Criterion Generation: Compared to traditional static predefined criteria, the task-adaptive dynamic criterion generation mechanism significantly improves the system's adaptability. It can dynamically adjust criteria according to the specific needs of different tasks and scenarios, thereby improving the accuracy and relevance of the results. This mechanism eliminates the need for manually pre-defining complex scoring criteria, effectively reducing annotation costs and time consumption, and lowering the barrier to system deployment. Simultaneously, by establishing global criteria to ensure consistency in scoring across datasets, and combining instance-specific criteria to provide fine-grained personalized scoring, it achieves an organic unity of standardization and personalization. Furthermore, the dynamically generated scoring criteria provide clear basis and explanation for each scoring decision, significantly enhancing the interpretability and transparency of the reward model's decision-making process.

[0197] 2. Technical Performance of the PaTaRM Framework: The preference-aware reward mechanism demonstrates outstanding performance improvement. Based on a preference-aware and task-adaptive point-based reward training framework, this mechanism exhibits strong performance enhancement capabilities in downstream RLHF tasks, proving its value in practical applications. This mechanism completely eliminates the dependence on explicit point labels, generating effective absolute quality signals solely from relative preference data. It demonstrates excellent robustness and adaptability in complex tasks and diverse scenarios, particularly when handling fine-grained quality differences.

[0198] From an overall technical perspective, the PataRM framework reduces annotation costs compared to traditional methods, while maintaining the original performance level and achieving a significant performance improvement, demonstrating the practical value of technological innovation. In terms of usability, PataRM is fully compatible with the standard RLHF process and can be seamlessly integrated into existing reinforcement learning frameworks, providing great convenience and flexibility for practical deployment and application.

[0199] In summary, the above methods: 1. Significantly reduce annotation costs and achieve efficient point-based signal distillation: This scheme innovatively utilizes paired preference data and automatically distills high-quality point-based training signals through an efficient preference-aware mechanism, without requiring any explicit absolute score labels. This innovation breaks through the high dependence of traditional point-based training on manual annotation, greatly reducing data preparation and annotation costs, and providing a theoretical and engineering foundation for the rapid iteration and deployment of reward models in large-scale real-world scenarios; 2. Task-adaptive dynamic scoring criterion generation enhances model generalization and scenario adaptability: This framework introduces a task-adaptive scoring criterion generation module, which can dynamically construct multi-dimensional scoring criteria that satisfy both global consistency and personalized needs based on specific task types and instance contexts. Compared to static predefined standards, dynamic scoring mechanisms significantly enhance the adaptability of reward models to diverse scenarios and complex tasks, effectively improving the model's generalization ability and practical application value. 3. Enhancing the interpretability and optimization performance of reward models, bridging pairwise and pointwise training paradigms: Through a preference-aware reward mechanism, PataRM can transform relative preference signals in pairwise data into interpretable pointwise quality assessments, compatible with and integrating pairwise comparison and pointwise scoring training paradigms. This method not only improves the optimization performance of reward models in reinforcement learning but also provides clear scoring criteria and decision explanations for model output, significantly enhancing the system's transparency and credibility.

[0200] This embodiment innovatively proposes PaTaRM, a training framework for preference-aware task-adaptive reward models. This framework directly constructs point-based training signals using relative preferences from pairwise data, eliminating the need for additional point-based labels; simultaneously, it dynamically generates scoring criteria adapted to the task and instances. PaTaRM is not only efficient but also possesses broad generalization ability and interpretability, enabling the generation of effective point-based training signals during reinforcement learning training without explicit labels, thereby significantly improving the performance of reward models.

[0201] It should be particularly noted that the steps in the various embodiments of the above-described method for intelligent question answering can be overlapped, substituted, added, or deleted from each other. Therefore, these reasonable permutations and combinations of the method for intelligent question answering should also fall within the protection scope of this disclosure, and the protection scope of this disclosure should not be limited to the described embodiments.

[0202] Based on the same inventive concept, this disclosure also provides a device for intelligent question answering, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of the method embodiments described above, the implementation of this device embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be repeated.

[0203] Figure 9 This is a block diagram illustrating an apparatus for intelligent question answering according to an exemplary embodiment. (Refer to...) Figure 9 The device 900 for intelligent question answering provided in this embodiment may include: a template acquisition module 901, a response data acquisition module 902, a scoring module 903, and a reward determination module 904.

[0204] The system includes several modules: a template acquisition module for acquiring prompt word templates, which include a set of basic evaluation dimensions for response quality and instance dynamic evaluation meta-instructions; a response data acquisition module for acquiring question-and-answer tasks and corresponding pairwise preference data, which includes accepted response data and rejected response data; a scoring module for determining a first judgment expansion sequence corresponding to the accepted response data and a second judgment expansion sequence corresponding to the rejected response data, based on the pairwise preference relationship between the accepted and rejected response data, through a preference-aware mechanism in the reward model, combined with the prompt word templates and the question-and-answer task; the judgment expansion sequences include point-based quality evaluation values; a reward determination module for determining a system reward signal based on the first and second judgment expansion sequences; the system reward signal is used to perform gradient optimization of the reinforcement learning strategy to drive the update of the reward model parameters; and the updated reward model is used to automatically evaluate the quality of question-and-answer data pairs generated during the training of the intelligent question-and-answer model, so as to fine-tune the parameters of the intelligent question-and-answer model according to the evaluation results to make the intelligent question-and-answer model suitable for intelligent question-and-answer tasks.

[0205] It should be noted that the template acquisition module, response data acquisition module, scoring module, reward determination module, and parameter tuning module mentioned above correspond to S302 to S308 in the method embodiment. The examples and application scenarios implemented by the above modules and their corresponding steps are the same, but are not limited to the content disclosed in the above method embodiment. It should be noted that the above modules, as part of the apparatus, can be executed in a computer system such as a set of computer-executable instructions.

[0206] In some embodiments, based on the pairwise preference relationship between the accepted and rejected response data, and through the preference-aware mechanism in the reward model, combined with the prompt word template and the question-and-answer task, a first judgment expansion sequence corresponding to the accepted response data and a second judgment expansion sequence corresponding to the rejected response data are determined. This includes: the reward model performing n independent quality scores on the accepted response data based on the prompt word template and the question-and-answer task to obtain the first judgment expansion sequence; where n is an integer greater than 0; and the reward model performing n independent quality scores on the rejected response data based on the prompt word template and the question-and-answer task to obtain the second judgment expansion sequence.

[0207] In some embodiments, when the reward model performs n independent quality scores on the same response data, it employs random sampling or diversity sampling strategies to generate different judgment unfolding sequences.

[0208] In some embodiments, determining a system reward signal based on the first judgment expansion sequence and the second judgment expansion sequence includes: determining the average value of the point-based quality assessment values ​​in each of the first judgment expansion sequences to obtain an average score for accepted data; determining the average value of the point-based quality assessment values ​​in each of the second judgment expansion sequences to obtain an average score for rejected data; and determining the system reward signal based on the average score for accepted data and the average score for rejected data.

[0209] In some embodiments, determining the system reward signal based on the average score of the accepted data and the average score of the rejected data includes employing at least one of the following reward signal generation mechanisms: generating a reward signal based on the relative score difference between a single quality score and the average score of the comparison data, wherein: for accepted response data, a positive reward is allocated when the single score is higher than the average score of the rejected data; for rejected response data, a positive reward is allocated when the single score is lower than the average score of the accepted data; generating a reward signal based on the average score difference between the average score of the accepted data and the average score of the rejected data, wherein: a positive reward is allocated when the average score difference is greater than zero.

[0210] In some embodiments, the reward model performs n independent quality scores on the adopted response data based on the prompt word template and the question-and-answer task. The execution process of each independent quality score includes: the reward model generating personalized evaluation criteria based on the instance dynamic evaluation meta-instruction, the question-and-answer task, and the adopted response data, wherein the personalized evaluation criteria generated each time are different; the reward model integrating the basic evaluation dimension set of response quality and the personalized evaluation criteria to perform a quality score on the adopted response data, and obtain the score result of that independent quality score.

[0211] In some embodiments, the reward model generates personalized evaluation criteria based on the instance dynamic evaluation meta-instruction, the question-and-answer task, and the adopted response data, including: dynamically generating a personalized evaluation dimension set based on the instance dynamic evaluation meta-instruction, the question-and-answer task, and the adopted response data; dynamically configuring the evaluation granularity and weight in the personalized evaluation dimension set based on the type, complexity, and response data characteristics of the question-and-answer task; and generating the personalized evaluation criteria for the adopted response data based on the configured personalized evaluation dimension set.

[0212] In some embodiments, the reward model is optimized by using the system reward signal to obtain an optimized reward model, including: obtaining the scoring results of each independent quality score for the adopted response data; determining the accepted reward value for each independent score for the adopted response data based on the scoring results of each independent quality score; sorting the accepted reward values; and optimizing the parameters of the reward model based on the accepted reward values ​​in descending order.

[0213] In some embodiments, based on the prompt word template and the question-and-answer task, a first judgment unfolding sequence for the adopted response data is generated through a reward model, including: filling the question-and-answer task and the adopted response data into the prompt word template to obtain a target prompt word; and inputting the target prompt word into the reward model so that the reward model performs a quality score on the adopted response data based on the target prompt word.

[0214] Since the functions of the device 900 have been described in detail in their respective method embodiments, they will not be repeated here.

[0215] The modules and / or sub-modules and / or units described in the embodiments of this disclosure can be implemented in software or hardware. The described modules and / or sub-modules and / or units can also be located in a processor. The names of these modules and / or sub-modules and / or units do not, in some cases, constitute a limitation on the module and / or sub-module and / or unit itself.

[0216] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a portion of a module or program segment containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer program instructions.

[0217] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0218] Figure 10 A schematic diagram of an electronic device suitable for implementing embodiments of the present disclosure is shown. It should be noted that... Figure 10 The illustrated electronic device 1000 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0219] like Figure 10 As shown, the electronic device 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage section 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device 1000. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0220] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed.

[0221] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing computer program instructions for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit (CPU) 1001, it performs the functions defined above in the system of this disclosure.

[0222] It should be noted that the computer-readable storage medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable computer program instructions. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Computer program instructions contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0223] In another aspect, this disclosure also provides a computer-readable storage medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The aforementioned computer-readable storage medium carries one or more programs, which, when executed by a device, enable the device to perform the following functions: acquiring a prompt word template, the prompt word template including a set of basic evaluation dimensions for response quality and instance dynamic evaluation meta-instructions; acquiring a question-and-answer task and corresponding accepted and rejected response data; based on the prompt word template and the question-and-answer task, generating a first judgment expansion sequence for accepted response data and a second judgment expansion sequence for rejected response data through a reward model; determining a system reward signal based on the first and second judgment expansion sequences; when the first judgment expansion sequence is greater than the second judgment expansion sequence, the system reward signal is a positive reinforcement signal; when the first judgment expansion sequence is less than or equal to the second judgment expansion sequence, the system reward signal is a negative correction signal; and using the system reward signal to perform parameter tuning on the reward model to obtain an optimized reward model; wherein the optimized reward model is used to perform automated quality evaluation on question-and-answer data pairs generated during the training of the intelligent question-and-answer model, so as to perform parameter tuning on the intelligent question-and-answer model based on the evaluation results so that the intelligent question-and-answer model is suitable for intelligent question-and-answer tasks.

[0224] According to one aspect of this disclosure, a computer program product or computer program is provided, comprising computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the methods provided in various optional implementations of the above embodiments.

[0225] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive) and includes several computer program instructions to cause an electronic device (such as a server or terminal device) to execute the method according to the embodiments of this disclosure.

[0226] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0227] It should be understood that this disclosure is not limited to the detailed structures, drawing arrangements or implementations shown herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method for intelligent question answering, characterized in that, include: Obtain the prompt word template, which includes a set of basic assessment dimensions for response quality and instance dynamic assessment meta-instructions; Obtain the question-and-answer task and the corresponding pairwise preference data, wherein the pairwise preference data includes accepted response data and rejected response data; Based on the pairwise preference relationship between the accepted and rejected response data, the preference perception mechanism in the reward model, combined with the prompt word template and the question-and-answer task, determines a first judgment expansion sequence corresponding to the accepted response data and a second judgment expansion sequence corresponding to the rejected response data; the judgment expansion sequence includes point-based quality assessment values. Based on the first judgment expansion sequence and the second judgment expansion sequence, a system reward signal is determined; the system reward signal is used to perform reinforcement learning policy gradient optimization to drive the update of the reward model parameters; The updated reward model is used to automatically evaluate the quality of question-and-answer data pairs generated during the training of the intelligent question-and-answer model, so as to fine-tune the parameters of the intelligent question-and-answer model based on the evaluation results, making the intelligent question-and-answer model suitable for intelligent question-and-answer tasks.

2. The method according to claim 1, characterized in that, Based on the pairwise preference relationship between the accepted and rejected response data, and through the preference-aware mechanism in the reward model, combined with the prompt word template and the question-answering task, a first judgment expansion sequence corresponding to the accepted response data and a second judgment expansion sequence corresponding to the rejected response data are determined, including: The reward model performs n independent quality scores on the adopted response data based on the prompt word template and the question-and-answer task to obtain the first judgment expansion sequence; n is an integer greater than 0; The reward model performs n independent quality scores on the rejected response data based on the prompt word template and the question-and-answer task to obtain the second judgment expansion sequence.

3. The method according to claim 2, characterized in that, When the reward model performs n independent quality scores on the same response data, it employs random sampling or diversity sampling strategies to generate different judgment expansion sequences.

4. The method according to claim 2, characterized in that, Based on the first judgment expansion sequence and the second judgment expansion sequence, the system reward signal is determined, including: Determine the average value of the point-based quality assessment values ​​in each first judgment unfolding sequence to obtain the average score of the accepted data; Determine the average value of the point-based quality assessment values ​​in each second judgment expansion sequence to obtain the average score of the rejection data; The system reward signal is determined based on the average score of the accepted data and the average score of the rejected data.

5. The method according to claim 4, characterized in that, Determining the system reward signal based on the average score of the accepted data and the average score of the rejected data includes employing at least one of the following reward signal generation mechanisms: A reward signal is generated based on the relative score difference between a single quality score and the average score of the comparison data, wherein: for accepted response data, a positive reward is allocated when the single score is higher than the average score of the rejected data; for rejected response data, a positive reward is allocated when the single score is lower than the average score of the accepted data. A reward signal is generated based on the average difference between the average score of the accepted data and the average score of the rejected data, wherein: when the average difference is greater than zero, a positive reward is allocated.

6. The method according to claim 2, characterized in that, The reward model performs n independent quality scores on the adopted response data based on the prompt word template and the question-and-answer task, wherein the execution process of each independent quality score includes: The reward model generates personalized evaluation criteria based on the instance dynamic evaluation meta-instruction, the question-and-answer task, and the adopted response data, wherein the personalized evaluation criteria generated each time are different. The reward model integrates the basic evaluation dimension set of response quality and the personalized evaluation criteria to score the quality of the adopted response data, thereby obtaining the score result of this independent quality score.

7. The method according to claim 6, characterized in that, The reward model generates personalized evaluation criteria based on the instance dynamic evaluation meta-instruction, the question-and-answer task, and the adopted response data, including: Based on the instance, the dynamic evaluation meta-instruction, the question-and-answer task, and the adopted response data, a personalized evaluation dimension set is dynamically generated. Based on the type, complexity, and response data characteristics of the question-and-answer task, the evaluation granularity and weights of the personalized evaluation dimension set are dynamically configured. The personalized evaluation criteria for the adopted response data are generated based on the configured personalized evaluation dimension set.

8. The method according to claim 2, characterized in that, The reward model is optimized by using the system reward signal to perform parameter tuning on the reward model, including: Obtain the score results for each independent quality score of the adopted response data; The reward value for each independent quality score is determined based on the score results of each independent score for the adopted response data. Sort the received reward values; The reward model is optimized by adjusting the parameters based on the received reward value, in descending order.

9. The method according to claim 1, characterized in that, Based on the prompt word template and the question-and-answer task, a first judgment sequence for the adopted response data is generated through a reward model, including: The question-and-answer task and the accepted response data are filled into the prompt word template to obtain the target prompt word; The target prompt is input into the reward model so that the reward model can score the quality of the adopted response data based on the target prompt.

10. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement the method for intelligent question answering as described in any one of claims 1-9.

Images