Planning guided adaptive verifiable multilingual question answering method and system

Abstract

The application provides a kind of planning guided adaptive verifiable multilingual question and answer method and system, belong to natural language processing field, including: S1: constructing input data set and task set, for the supervised fine-tuning of base model, obtain the fine-tuned base model with preliminary multilingual generation ability and structured evaluation ability;S2: the fine-tuned base model is used as strategy model, constructs structured list module, structured generative criticism modeling module and adaptive multilingual reward optimization module, for each answer and its thinking process is structured evaluation and reward calculation, generates the reward and advantage function for driving strategy update, so as to the iterative optimization of strategy model.The method of the application significantly improves the generation quality of large language model in low-resource, multilingual environment, and promotes the global application landing of trusted artificial intelligence technology.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing, and specifically relates to a planning-guided adaptive verifiable multilingual question-answering method and system. Background Technology

[0002] In recent years, large language models based on the Transformer architecture have achieved revolutionary progress in the field of natural language processing. Models such as the GPT series, LLaMA series, and Qwen series have demonstrated near-human performance in numerous language tasks, including English. This success has prompted the research and industry communities to focus on extending the capabilities of large language models (LLMs) to multilingual scenarios, aiming to build truly global artificial intelligence applications.

[0003] However, large language models (LLMs) exhibit significant imbalances in the distribution of multilingual capabilities. Their performance is largely influenced by the quality and size of the training corpus. For resource-rich languages, such as English and Chinese, the models demonstrate superior performance; however, for many globally resource-scarce languages, the performance of the models drops significantly due to the scarcity of high-quality corpora.

[0004] Existing technologies include:

[0005] 1. A general alignment scheme based on RLHF (Reinforcement Learning from Human Feedback):

[0006] This is the mainstream technical approach for aligning large language models with human intent. Its core comprises three stages: First, using supervised learning, a pre-trained base model is fine-tuned on high-quality instruction data to obtain an initial model capable of understanding and following human instructions. Second, multiple responses to the same prompt are sampled from this initial model. Human annotators then perform pairwise preference comparisons on these responses and train a reward model that quantifies human preferences based on this preference dataset, enabling it to predict responses that better align with human values. Third, the trained reward model is used as a proxy reward signal, and reinforcement learning algorithms are applied to optimize the initial model's strategy. By maximizing the cumulative reward provided by the reward model, the model parameters are iteratively adjusted to ensure that the model output is not only accurate but also safer, harmless, and better aligned with complex and abstract human intents. This results in a universal alignment framework that does not rely on manually designed reward functions and can be widely applied to various tasks.

[0007] 2. Mindset hints:

[0008] Thought chain cues, as a cutting-edge cuing engineering technique, hinge on guiding large language models to reveal their internal reasoning processes through specific instruction design. Specifically, when designing cues using this method, the model is explicitly instructed to "think step by step," thereby prompting it to generate clear intermediate reasoning steps similar to human problem-solving. This mechanism not only makes the model's thought process more transparent but has also proven to significantly improve the accuracy of answers in complex tasks such as arithmetic, common sense, and symbolic reasoning, while enhancing the comprehensibility and interpretability of the model's output.

[0009] 3. Enhanced search generation:

[0010] To effectively address the issue of factual "illusions" that may arise during content generation in large language models, a retrieval-enhanced generation method has been proposed. The core idea of ​​this method is to automatically retrieve information fragments highly relevant to the current question from one or more external knowledge sources (such as Wikipedia, specialized databases, etc.) before the model performs the generation task. Then, this authoritative information obtained from the retrieval is used as additional context and input into the generation model along with the original question. This approach provides the model with reliable factual support, enabling it to generate answers based on solid evidence, thereby significantly improving the accuracy and reliability of the output content.

[0011] 4. Multi-task and multilingual pre-training:

[0012] The core objective of this approach is to enable models to autonomously learn and construct universal representations across different languages ​​and tasks from complex linguistic data through large-scale pre-training on massive, multilingual mixed corpora. This method allows models to capture underlying, task-independent linguistic patterns and world knowledge, thereby significantly improving their generalization ability in cross-linguistic transfer and diverse downstream tasks.

[0013] However, existing technologies have the following drawbacks:

[0014] 1. The black box nature of reward mechanisms and evaluation bias:

[0015] In existing technologies, reward models based on RLHF typically simplify complex generation quality assessment into a single scalar score, resulting in a lack of interpretability in the optimization process and making it difficult to accurately locate and correct specific errors in the generated content. Furthermore, existing systems often rely on training with high-resource language data, and their evaluation criteria are prone to systematic biases in low-resource language tasks, failing to adapt to different language habits and cultural backgrounds, thus misleading the direction of model optimization.

[0016] 2. The unstructured and unverifiable nature of the reasoning process:

[0017] While existing thought chain methods can stimulate model reasoning capabilities, the intermediate steps they generate are in the form of free text, lacking a unified structure, making it difficult to achieve automatic verification and error correction. The reasoning paths are also prone to redundancy or logical errors. Furthermore, these methods are highly dependent on prompt engineering, have weak cross-language and cross-task generalization capabilities, and lack stability, making them unsuitable as a robust core technology.

[0018] 3. Insufficient knowledge integration and logical planning abilities:

[0019] Although retrieval-enhanced generation techniques introduce external knowledge sources, the models still lack the ability to deeply integrate information and systematically plan. Existing methods tend to mechanically copy or simply piece together retrieval results, failing to organize and elaborate on the core of the problem from multiple dimensions in a structured way. This results in generated content that is logically loose, lacks coherence, and is prone to missing key analytical perspectives.

[0020] 4. Representation discrimination problem in multilingual joint training:

[0021] While current multilingual pre-training models integrate data from multiple languages, the model parameters tend to optimize the representation of high-resource languages ​​due to their advantages in data scale and information density, while the representation learning of low-resource languages ​​is suppressed, resulting in the model significantly lagging behind in low-resource language tasks. Summary of the Invention

[0022] To address the aforementioned technical problems, this invention provides a planning-guided adaptive verifiable multilingual question-answering method, comprising the following steps:

[0023] Step S1: Construct the input dataset and task set for supervised fine-tuning of the base model to obtain a fine-tuned base model with preliminary multilingual generation and structured evaluation capabilities;

[0024] Step S2: Using the fine-tuned base model as the strategy model, a structured checklist module is constructed to decompose the complex task into multi-dimensional logical paths and pre-constraints, guiding the strategy model to follow preset reasoning nodes when generating answers and thinking processes. Simultaneously, a structured generative criticism modeling module is constructed to trace the source, verify logical consistency, and locate errors for each planning node in the thinking process, generating verifiable structured criticism feedback. Finally, an adaptive multilingual reward optimization module is used to calculate the reward value and advantage function by combining the checklist achievement degree and criticism verification results, and the strategy model is iteratively optimized in a closed loop using a reinforcement learning algorithm.

[0025] Beneficial effects:

[0026] 1. Solving the problems of black box reward and low-resource language bias in RLHF: This invention breaks through the limitation of traditional reward models that only output a single scalar, and constructs a fine-grained reward mechanism with interpretability to alleviate the evaluation bias caused by high-resource language data dominance.

[0027] 2. Constructing a structured planning checklist mechanism for multilingual tasks: This invention establishes a unified strategy candidate pool that spans multiple languages ​​and tasks, guiding the model to automatically select and follow a structured outline before answering, systematically planning the answer path, and improving logical rigor and content completeness.

[0028] 3. Design a verifiable reward function with language consistency and task self-identification capabilities: This invention introduces a language consistency reward to ensure that the model uses the same language to think and answer after identifying the language of the question; it combines the accuracy of task identification and the quality of checkpoint completion to achieve multi-dimensional weighted evaluation.

[0029] 4. Achieving self-verification and multi-dimensional reward optimization mechanism for the model: In the reinforcement learning process, this invention guides the model to self-evaluate the selected strategy and generated content, enhancing the credibility and controllability of the answer and forming a closed-loop reasoning mechanism of "planning-execution-verification". Attached Figure Description

[0030] Figure 1 This is a schematic diagram of a planning-guided adaptive verifiable multilingual question-answering method according to the present invention.

[0031] Figure 2 This is a schematic diagram of the overall framework of the method of the present invention;

[0032] Figure 3 This is a block diagram of a planning-guided adaptive verifiable multilingual question-answering system according to the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0034] Example 1:

[0035] like Figure 1 As shown in the figure, an adaptive verifiable multilingual question-answering method guided by planning provided by an embodiment of the present invention includes the following steps:

[0036] Step S1: Construct the input dataset and task set for supervised fine-tuning of the base model to obtain a fine-tuned base model with preliminary multilingual generation and structured evaluation capabilities;

[0037] Step S2: Using the fine-tuned base model as the strategy model, a structured checklist module is constructed to decompose the complex task into multi-dimensional logical paths and pre-constraints, guiding the strategy model to follow preset reasoning nodes when generating answers and thinking processes. Simultaneously, a structured generative criticism modeling module is constructed to trace the source, verify logical consistency, and locate errors for each planning node in the thinking process, generating verifiable structured criticism feedback. Finally, an adaptive multilingual reward optimization module is used to calculate the reward value and advantage function by combining the checklist achievement degree and criticism verification results, and the strategy model is iteratively optimized in a closed loop using a reinforcement learning algorithm.

[0038] Traditional methods typically start with a question and generate tokens unidirectionally from left to right. This process is highly opaque, lacking structured planning and self-verification mechanisms. In contrast, the planning-guided method of this invention takes the human cognitive pattern of "conceptualization before response" as its starting point. Starting from the actual task requirements and credible response standards of low-resource languages, it reverse-engineers the structured dimensions and verification standards required for the response, achieving high-fidelity, high-credibility answer generation with minimal labeled data. Specifically, the 'planning' in this invention refers to a set of dimensions pre-defined by the structured checklist module, covering the key points of the task logic; 'guidance' refers to the process through the SFT stage... <think>The standardized label format and variance-aware reward mechanism in the RL phase ensure that the model's sampling distribution always converges to these pre-defined logical nodes, thus solving the problem of loose logic in the large model generation process. This invention integrates cutting-edge concepts of reinforcement learning and interpretable artificial intelligence to construct a complete question-answering method that integrates policy planning, content generation, multi-dimensional verification, and adaptive optimization, providing a new technical paradigm for trustworthy artificial intelligence applications in low-resource, multilingual environments. The complete training and inference process of this invention is as follows: Figure 2 As shown, it includes two closely connected stages: the Supervised Fine-tuning Stage (SFT Stage) and the Reinforcement Learning Stage (RL Stage).

[0039] In one embodiment, step S1 above—constructing an input dataset and a task set for supervised fine-tuning of the base model to obtain a fine-tuned base model with preliminary multilingual generation and structured evaluation capabilities—specifically includes:

[0040] Step S11: Construct the dataset, including the instruction fine-tuning dataset and the multilingual dataset;

[0041] Step S12: Construct a task set including: natural language reasoning, text creation, text completion, heuristic text generation, summarization, dialogue, and open-ended question answering.

[0042] The goal of this step is to endow the model with basic multilingual generation capabilities and structured evaluation capabilities. For example... Figure 2 As shown, the input includes instruction tuning data (Instruction Tuning: Question) and multilingual support data (Multilingual Support), and specifically covers (Subjective Tasks) to enhance the model's versatility, including: natural language inference, text creation, text completion, heuristic text generation, summarization, dialogue, and open-ended question answering.

[0043] After supervised fine-tuning training, an initialized base model is obtained. This model has initially acquired the ability to think according to a structured checklist, generate answers, and evaluate them. The output format typically includes `` <think> ...< / think> `and` <answer> ...< / answer> `tag`

[0044] In one embodiment, step S2 above involves using the fine-tuned base model as the strategy model. A structured checklist module is constructed to decompose complex tasks into multi-dimensional logical paths and pre-constraints, guiding the strategy model to follow preset inference nodes during answer generation and the thinking process. Simultaneously, a structured generative criticism modeling module is constructed to trace the source, verify logical consistency, and locate errors for each planning node in the thinking process, generating verifiable structured criticism feedback. Finally, an adaptive multilingual reward optimization module is used to calculate the reward value and advantage function by combining checklist achievement and criticism verification results. A reinforcement learning algorithm is then used to perform closed-loop iterative optimization of the strategy model, specifically including:

[0045] Step S21: The structured list module is used to define and store task-specific answer lists and general reasoning process lists. Each list defines multiple evaluation dimensions and their corresponding weights. Each list is stored in the form of evaluation tuples {dimension, weight}, and the total weight is 1. Specifically, it includes:

[0046] Step S211: Define the evaluation dimensions of the task-specific answer list: semantic validity, task completion, information fidelity, and format compliance, as well as the corresponding weights for each dimension.

[0047] The task-specific answer list is specifically designed to evaluate whether the final answer generated by the strategy model meets the functional objectives required for a specific task type. For each type of task to be processed (e.g., text summarization, open-ended question answering, instruction following, text creation, etc.), three to eight core evaluation dimensions are extracted through analysis and summarization by domain experts. Each dimension corresponds to a key and identifiable aspect of answer quality.

[0048] Typical evaluation dimensions for task-specific answer lists include, but are not limited to:

[0049] Semantic validity is used to assess the internal logical consistency of the answer content itself, ensuring that there are no self-contradictions.

[0050] Task completion rate is used to evaluate whether the answer fully and accurately responds to all explicit and implicit requirements in the query;

[0051] Information fidelity, mainly for tasks such as summarizing and rewriting, assesses whether the generated content is strictly faithful to the source text information and does not contain any unfounded fabrication or distortion.

[0052] And format compliance, used to assess whether the presentation format of the answer (such as a list, table, JSON structure, or specific style) strictly follows the format specifications given by the user. The specific number of these dimensions, their precise definitions, and their associated weight values ​​can all be flexibly configured and expanded according to the actual needs of different task categories.

[0053] Step S212: Define the evaluation dimensions of the general reasoning process checklist: task recognition reward, logical rigor, reasoning correctness, error recognition, language consistency and redundancy, and the corresponding weights for each dimension.

[0054] A universal reasoning process checklist is specifically designed to evaluate the quality of the reasoning trajectory or thought process chain presented by a strategy model before generating the final answer, thereby enabling supervision of the model's internal thinking process. Based on the analysis of higher-order human cognitive processes and reliable reasoning patterns, and combined with a diagrammatic structure, a set of universal thinking quality assessment dimensions applicable to all tasks is defined.

[0055] The general reasoning process checklist includes the following six core evaluation dimensions:

[0056] (1) Task recognition reward: Evaluate whether the model accurately identifies and understands the user's query intent and the type of task during the thinking process;

[0057] (2) Logical rigor: Pay attention to whether the connection between reasoning steps is smooth and natural, and whether the preceding steps can effectively support the subsequent conclusions, so as to ensure the coherence of the overall logical chain;

[0058] (3) Correctness of reasoning: Verify whether each specific factual assertion and logical inference link in the reasoning chain is correct and whether the methods and steps used are appropriate;

[0059] (4) Error identification: assess whether the reasoning process actively includes checks and explanations for potential uncertainties, boundary conditions, or its own errors;

[0060] (5) Language consistency: Assess whether the language used in the thinking process is consistent with the target output language to prevent language drift;

[0061] (6) Redundancy: Judge whether the expression of the thought chain is concise and efficient, and whether it avoids unnecessary repetition or irrelevant steps.

[0062] These six dimensions together constitute a refined and structured supervision standard for the internal reasoning process of the model.

[0063] Whether it's a task-specific answer list or a general reasoning process list, this invention uses the exact same mathematical and data format for definition and storage. Specifically, each list is defined as a set containing multiple pairs of "evaluation dimensions" paired with "weights".

[0064] The structured inventory module achieves high configurability through parametric design. The descriptive text for each dimension and its corresponding weight value are stored in machine-readable configuration files or database records. All dimension weights are normalized values, ensuring that the sum is always 1. Furthermore, this invention supports dynamic loading and hot updating of the inventory, allowing for real-time iteration and optimization of the evaluation dimensions and weights based on actual feedback without service interruption.

[0065] The structured inventory module is the core of this invention for defining and storing evaluation criteria. It provides a unified, interpretable, and language-independent quality metric benchmark for the entire reward modeling framework. The core function of this module is to predefine and maintain a hierarchical, configurable evaluation strategy library. This library decomposes the complex task of evaluating generated content quality into a series of measurable evaluation dimensions with varying degrees of importance. This module itself does not directly output computational signals; instead, it serves as the "fundamental specification" or "scoring criterion library" for the entire system, providing the sole basis and standard for evaluation and calculation for the downstream Structured Generative Criticism Modeling (SGCM) and Adaptive Multilingual Reward Optimization (AMRO) modules. This has the following beneficial effects:

[0066] (1) The evaluation criteria have been thoroughly structured and made explicit: the quality requirements of generated content, which were originally based on intuition and were implicit and vague, have been clearly decomposed and defined as a series of specific dimensions that can be clearly described and objectively measured. This has transformed the evaluation process from a "subjective artistic judgment" to an objective engineering process that can be "checked item by item according to the standard", laying the foundation for automated and highly consistent evaluation.

[0067] (2) It ensures the fundamental consistency of cross-language assessment: Since all assessment dimensions in the checklist are defined based on the inherent nature of the task and the universal laws of human logical thinking, they are decoupled from specific language symbols and expression habits. Therefore, the same checklist can be applied equally to assessments in any language without modification, eliminating the "reward drift" problem caused by differences in assessment standards from the source. This is the key technology for achieving truly reliable multilingual model alignment.

[0068] (3) It provides high configurability and scalability: By defining dimensions and setting weights as flexibly adjustable parameters, this module allows users to dynamically adjust the focus of the evaluation system according to different product goals, application scenarios, cultural backgrounds, or security specifications. At the same time, when new task types need to be supported, they can be easily integrated simply by designing a new task-specific list for that task, which greatly improves the adaptability of this invention.

[0069] (4) Separation of process and result supervision: By establishing an independent and general reasoning process checklist, quality specifications for the model's "internal thinking process" are clearly proposed. This enables the training process to simultaneously supervise and optimize the rigor of the model's reasoning and the correctness of the answer output, which plays a crucial role in cultivating the model's deep reasoning ability and improving the reliability and interpretability of its decisions.

[0070] (5) It provides a clear interaction contract for downstream modules: The list pattern output by this module, as a stable and clear machine-readable specification, clarifies "which aspects to evaluate" for the SGCM module and also indicates "how to comprehensively weigh different aspects" for the AMRO module. It is the cornerstone for the efficient and accurate collaborative work of various modules within the method of this invention, ensuring the modularity, low coupling and good maintainability of the system architecture.

[0071] Step S22: The structured generative criticism modeling module receives the query-inference-answer triples generated by the model, independently scores each evaluation dimension based on the list, and outputs structured criticism results, specifically including:

[0072] Step S221: Receive the tuple output by the policy model. The tuple contains three parts: user query, inference trajectory generated by the model, and final answer; determine the task category to which the user query belongs, and obtain the task-specific answer list and general inference process list corresponding to the task category from the structured list module.

[0073] The input to this module is a complete output tuple from the policy model, which contains three parts: the user query, the inference trajectory generated by the model (usually text identified by specific labels), and the final answer generated by the model.

[0074] List Binding: Based on the semantics of the user query or through a separate task identification module, the task category to which the user belongs is determined. Subsequently, the task-specific answer list corresponding to that task category is retrieved from the structured list module, along with the general reasoning process list. These two lists together define the complete set of dimensions to be examined in this assessment.

[0075] Step S222: Perform dimensional analysis on the task-specific answer list and the general reasoning process list to generate structured criticism, including: the dimension name, score, and reasoning text for each evaluation dimension.

[0076] In the structured criticism generation stage, dimensionality parsing is first performed, treating each evaluation dimension in the bound list as an independent evaluation subtask. The total number of dimensions in the list is a configurable parameter, typically set between 5 and 15 to represent different levels of evaluation granularity. Subsequently, for each evaluation dimension, the module sequentially performs evidence extraction and score prediction operations. Specifically, this involves scanning and locating specific text fragments related to the current dimension as evidence in the input query, reasoning trajectory, and final answer tuple. Based on this evidence and following a predefined controlled template, a concise textual reason that can be directly linked to and traced back to the specific part of the input is generated. Simultaneously, based on the same evidence, a soft score between 0 (completely disagree) and 1 (completely agree) is output to quantify the degree of agreement. Finally, through the criticism assembly step, the module integrates the evaluation dimension name, reason text, and score of all dimensions in dimensional order to generate the final structured criticism, which includes the following two parts:

[0077] (1) Evaluation: including scores and evidence in dimensions such as task recognition reward, logical rigor, reasoning correctness, error recognition, language consistency and redundancy.

[0078] (2) Response evaluation: Includes scores and evidence for several checkpoints (Checkpoint 1 to Checkpoint n) for a specific task.

[0079] The Structured Generative Criticism Modeling (SGCM) module is the core evaluation engine of this invention. The module's capabilities stem from the training process in step S1 (supervised fine-tuning phase), which introduces inference trajectories and corresponding structured criticism samples, giving the base model a functional paradigm for self-evaluation. Therefore, this module is not an external component independent of the policy model, but rather a specific working mode possessed by the base model after supervised fine-tuning.

[0080] During the inference phase, SGCM receives the query-inference-answer triplet output by the policy model. Following an authoritative structured checklist pattern, it performs independent analysis on each evaluation dimension through dimensional parsing. Subsequently, based on the input evidence fragments and adhering to a predefined controlled template, it generates evaluation pairs for each dimension, including "quantified scores" and "evidence extraction." Finally, the module performs a critique assembly step, integrating the names, scores, and reasoning texts corresponding to all dimensions to generate a fixed-format and interpretable structured critique, providing verifiable signal input for the adaptive multilingual reward optimization module. The SGCM module offers the following benefits:

[0081] (1) The evaluation process is made completely transparent and explainable: By decomposing the overall evaluation into independent dimensions and mandating that each score be based on specific reasons based on input evidence, the evaluation decision-making process of the model becomes completely transparent and traceable. Users and developers can clearly understand the reasons behind each score, which greatly enhances the credibility of the system.

[0082] (2) It generates highly robust and consistent evaluation signals: the fixed output structure (dimensional evaluation pairs) and the strict requirement of evidence binding fundamentally eliminate the randomness, ambiguity and subjectivity inherent in traditional free text criticism. Even in low-resource language scenarios, as long as the input is certain, the generated critical output can remain highly stable, thus providing a reliable and consistent signal basis for downstream reinforcement learning training.

[0083] (3) Provides rich and structured intermediate supervision signals: The criticism output by this module is not a single summary score, but a set of detailed scores and reasons covering multiple quality dimensions. This provides richer and more instructive gradient information for the iterative optimization of the model than scalar rewards, enabling the model to make targeted improvements in different aspects such as logic, factuality, and completeness, thereby improving the overall performance more efficiently.

[0084] (4) Support for consistent alignment across languages ​​and tasks: Since all evaluation behaviors strictly follow the same language-independent structured checklist pattern, the SGCM module executes the same essentially identical "scoring criteria" across different languages ​​and tasks. This mechanism fundamentally ensures the cross-language consistency and cross-task adaptability of the reward model evaluation, effectively solving the "reward drift" problem commonly found in existing technologies.

[0085] Step S23: The adaptive multilingual reward optimization module is used to fuse structured criticism results with contextual information, calculate multiple rewards, and weight them to synthesize a final reward signal to drive policy model optimization. Specifically, it includes:

[0086] Step S231: Based on structured criticism, contextual information, and user query text, calculate the reward, including parallel calculation of the following four sub-rewards: a. Language consistency reward: determining whether the generated content is consistent with the target language; b. Task recognition reward: determining whether the model correctly identifies the task type; c. Structured reward for the reasoning process: weighted summation of scores for each dimension of the reasoning process, multiplied by a weighting factor based on the variance of the scores; d. Structured reward for the final answer: weighted summation of scores for each dimension of the answer, multiplied by the same weighting factor; finally, the above four sub-rewards are weighted and merged, and the structural invalidity penalty term is subtracted to form the final reward; the structural invalidity penalty term is used to apply deterministic deductions to outputs with format errors, missing reasoning trajectories, or violations of structural requirements.

[0087] a. Language consistency reward Determine if the generated content is consistent with the target language;

[0088] ;

[0089] in, The output of the strategy model (including the reasoning process and the final answer);

[0090] f(): Language recognition function, which uses the integrated prediction results of one or more language recognition models or machine translation interfaces;

[0091] The expected output language inferred from the query;

[0092] : Indicator function, equal to 1 when the condition in parentheses is true, otherwise equal to 0.

[0093] The language consistency reward aims to encourage the model to generate and respond strictly in the language expected by the query. This reward is calculated using an indicator function: the function takes a value of 1 (i.e., a reward is given) when the linguistic features of the generated content are completely consistent with the expected target language; otherwise, it takes a value of 0. This effectively suppresses the phenomenon of unintentional or intentional degradation or drift towards high-resource languages ​​(such as English) in low-resource language scenarios.

[0094] b. Task recognition reward : Determine whether the model correctly identifies the task type;

[0095] ;

[0096] in, : Real and normal task tags;

[0097] The task label predicted by the classifier.

[0098] The task identification reward aims to incentivize the policy model to correctly identify the user's true intent and task category during the internal inference phase. This reward is also calculated using an indicator function: the function returns 1 when the model's predicted task type matches the actual reference task label, and 0 otherwise. This ensures that a correct and targeted evaluation checklist can be used for subsequent self-evaluation.

[0099] c. Structured rewards for the reasoning process The scores for each dimension of the reasoning process are weighted and summed, and then multiplied by a weighting factor based on the variance of the scores.

[0100] ;

[0101] in, : Reward points for reasoning process;

[0102] The standard deviation of the sub-scores of reasoning is used to represent the degree of dispersion (or consistency) of each dimension of the reasoning process. The higher the value, the greater the noise in the reasoning process or the more inconsistent the logic.

[0103] : Variance-perceived weighting function, which uses the standard deviation of thought sub-scores. Let be the independent variable, and let the output function value be the weight adjustment factor. For a monotonically decreasing function, when When it increases, The output value decreases;

[0104] Total number of dimensions of reasoning and thinking;

[0105] : No. The original scores for each dimension of reasoning;

[0106] : No. The corresponding weight coefficients for each dimension of reasoning.

[0107] The structured reward for the reasoning process aims to aggregate the fine-grained evaluations of the model's thought process (reasoning trajectory) across various dimensions by the SGCM module into a single reward signal. This calculation introduces a continuous downweighting function mechanism. Specifically, the calculation involves first multiplying the scores of each dimension in the reasoning process checklist by their corresponding weights and summing the results. Then, this weighted sum is multiplied by a factor based on the variance of the thought process scores. The weighting factor is used to reduce the reward weight of a sample when the score variance is large (meaning high evaluation uncertainty).

[0108] d. Structured reward for the final answer The scores for each dimension of the answer are weighted and summed, then multiplied by the same weighting factor.

[0109] ;

[0110] in, Bonus points for the final answer after variance smoothing;

[0111] The final answer is in the... Original scores on each evaluation dimension;

[0112] The final answer is in the... The corresponding weight coefficients for each evaluation dimension;

[0113] The total number of evaluation dimensions.

[0114] The final answer structured reward aims to aggregate the fine-grained evaluations of the model's final answer across various dimensions from the SGCM module. It's worth noting that the smoothing of this reward also relies on the stability of the thought process. The specific calculation logic is as follows: All dimension scores belonging to the "task-specific answer list" and their preset weights are selected from the SGCM output and summed using a weighted average. Subsequently, the base answer reward is adjusted using the same continuously weighted function coefficients based on the variance of the thought process as in the inference reward calculation. This coupled design ensures that if the model's thought process is chaotic (high variance), even if the answer happens to be correct, the final reward will be suppressed.

[0115] Finally, the four sub-rewards are weighted and merged, and the structural invalidity penalty is subtracted to form the final reward. The structural invalidity penalty is used to apply deterministic deductions to outputs with format errors, missing reasoning paths, or violations of structural requirements.

[0116] ;

[0117] in, These are the preset weighting coefficients for each sub-reward. Invalid Structure Penalty: This is a deterministic deduction that occurs when the model output contains structural errors (e.g., missing values). <think>or <answer>This penalty is applied directly to issues such as broken tags, formatting errors, and failure to comply with structured requirements.

[0118] Step S232: Length-aware loss normalization is used to normalize the reward according to the total number of lexical units in the generated samples, in order to eliminate the bias introduced by different generated sample lengths on the optimization process.

[0119] This invention employs length-aware loss normalization to eliminate the bias introduced by different generated sample lengths on the optimization process (e.g., preventing the model from tending to generate extremely long text to accumulate rewards). Specifically, it performs a length normalization operation by calculating the total number of tokens in all generated samples in the current batch and dividing the aggregated loss by this total number (or taking its reciprocal as the normalization factor). This step ensures that model training focuses on the quality of individual content rather than simply the sequence length.

[0120] Variance stabilization is employed to calculate the mean and standard deviation of the statistical rewards for the current batch or historical window, thereby maintaining the distinguishability and stability of the reward signal.

[0121] This invention employs variance stabilization and non-zero variance preservation to ensure that the reward signal during reinforcement learning remains discriminative and its distribution is stable, introducing a dynamic variance stabilization mechanism. The system uses a segmented statistical strategy based on the number of samples currently being processed:

[0122] When the sample size is small (e.g., less than or equal to 64), the mean and standard deviation of the current batch of rewards are calculated directly.

[0123] When the sample size is large (e.g., greater than 64), the mean and standard deviation are calculated using the statistical data within the historical window.

[0124] Meanwhile, built-in logic ensures that the reward variance is always maintained non-zero, preventing gradient vanishing or the model from being unable to distinguish between good and bad policies due to the homogenization of reward signals.

[0125] Based on the mean and standard deviation of the rewards, the rewards for each sample are standardized to calculate the standardized advantage value, which is then input into the loss function of the group relative policy optimization to drive the parameter update of the policy model.

[0126] As the final step before output, this invention standardizes the reward of each sample based on the aforementioned mean and standard deviation to calculate the odds value. The specific calculation logic is as follows: subtract the mean from the initial reward value of the sample, and then divide by the standard deviation. The resulting standardized odds values ​​(A1, A2, …, A…) are then calculated. G ), will be directly input into the loss function of Group Relative Policy Optimization (GRPO) ( In this context, the parameters of the driving strategy model are updated.

[0127] The Adaptive Multilingual Reward Optimization (AMRO) module relies heavily on a set of configurable parameters to adapt to different scenarios:

[0128] Sub-reward weighting coefficient: Used to control the relative strength of the four sub-rewards (language, task, reasoning, and answer) in the final synthesis.

[0129] Variance-aware weighting function parameters: used to adjust the system's sensitivity to scoring inconsistencies, such as adjusting parameters in a continuous weighting function.

[0130] Statistical window size: A threshold (e.g., 64) used to define the "sample size" in variance stabilization calculations.

[0131] The Adaptive Multilingual Reward Optimization Module (AMRO) is the core of the reward signal synthesis and stabilization in this invention, corresponding to... Figure 2 The reward model part of the algorithm integrates the multi-dimensional, fine-grained structured criticism generated by the upstream SGCM module with key system-level control signals (including language attributes and task categories), transforming and outputting a single, stable, and comprehensive reward value suitable for reinforcement learning algorithm optimization. Its core design idea is to integrate the complex multi-objective alignment problem into a scalar objective that can be efficiently optimized through weighted aggregation, continuous weight reduction, and stability adjustment techniques, achieving the following beneficial effects:

[0132] (1) A robust integration of complex multi-objective rewards is achieved: by decoupling the global alignment objective into four sub-objectives and calculating them separately, and then by adjusting the coupling based on thinking variance, dual constraints on reasoning and results are achieved.

[0133] (2) Eliminate length bias and training oscillation: introduce length-aware loss normalization to effectively prevent the model from generating long and redundant text; introduce variance stabilization mechanism to ensure that the training process can still maintain stable gradient updates when the reward of low-resource language samples fluctuates greatly, thus avoiding model collapse.

[0134] (3) Provides standardized optimization objectives: The final output advantage value has been strictly statistically standardized, enabling this module to seamlessly connect with various mainstream policy gradient optimization algorithms, thereby improving the system's versatility and training efficiency.

[0135] The reinforcement learning phase utilizes the Group Relative Policy Optimization (GRPO) algorithm to achieve model self-iteration and improvement. In this phase, the initialized base model serves as the policy model, generating multiple candidate outputs for the same input problem. Simultaneously, to prevent model collapse during optimization, a reference model and a base model are introduced, and the update magnitude of the policy model is limited by calculating the KL constraint. Each answer and its corresponding thought process undergoes structured evaluation and reward calculation by the structured checklist module, the structured generative criticism modeling module, and the adaptive multilingual reward optimization module, ultimately generating the reward and advantage functions to drive policy updates.

[0136] For example, follow Figure 2 The architecture and data flow shown demonstrate the entire process from question input to reward generation for a trained model.

[0137] 1. Tasks and Inputs:

[0138] Task type: Open Domain Question Answering (OEQA).

[0139] Search (Tibetan):

[0140] " ”;

[0141] (The Chinese meaning is: "What is the capital of the Tibet Autonomous Region? Please briefly describe two important periods in its history.")

[0142] Expected output language: Tibetan.

[0143] 2. Initialization Phase (corresponding to "Fine-tuning Phase: SFT"):

[0144] (1) Training data: Multilingual supervised fine-tuning data (containing a large number of triplet samples similar to "query-thinking-answer") and standard multilingual SFT corpus were used for mixed training.

[0145] (2) Specialized training: Conduct structured criticism generation training so that the model learns to generate evaluations based on the checklist.

[0146] (3) Model output: A policy model is obtained through training. The model already possesses the ability to understand Tibetan queries and generate Tibetan responses. According to ` <think> ...< / think> <answer> ...< / answer> The ability to generate structured output in a formatted manner. Potential for preliminary self-assessment based on internal checklist standards.

[0147] 3. Online Inference and Optimization Phase (corresponding to "Reinforcement Learning: RL"):

[0148] When a user submits the above Tibetan query, the system enters the reinforcement learning inference loop. The inference model currently used for the service is the model initialized by SFT (or the version optimized in the previous round of RL).

[0149] 3.1 Strategy Model Generates Candidate Answers:

[0150] The inference model samples the same query multiple times to generate a set (e.g.) (Number) candidate responses. Each response includes the thought process and the final answer, for example:

[0151] : <think> (Thinking: 1. The task is OEQA, which requires identifying the capital city and briefly describing two historical periods; 2. Logical deduction: confirm that the capital city is Lhasa; 3. Verification of reasoning correctness: recall and select the Tubo period and the Ganden Phodrang period as two important periods; 4. Error check: confirm that there is no confusion of historical dates...)< / think> .

[0152] <answer> (Answer: The capital of the Tibet Autonomous Region is Lhasa. Two important historical periods are: the Tubo Dynasty (approximately 7th-9th centuries), which was the first unified regime in the history of the Tibet Autonomous Region; and the Ganden Phodrang period (from the mid-17th century), which marked the establishment of the Gelugpa theocratic system.)< / answer> .

[0153] : May contain variations with different historical periods, varying levels of detail, or accidental factual errors.

[0154] 3.2 Assessment using the structured checklist approach (corresponding to "structured checklist approach"):

[0155] The system calls a predefined list to evaluate each candidate response.

[0156] Task-Specific Answer Critiques: Loads preset checkpoints for OEQA tasks, such as:

[0157] (1) Checkpoint 1 (weight) =0.4): Assess whether "Lhasa" is the correct capital city.

[0158] (2) Checkpoint 2 (weight) =0.4): Assess whether the requirements of "where" and "two periods" are fully covered.

[0159] (3) Checkpoint 3 (weight) =0.2): Evaluate whether the Tibetan expression format conforms to the question and answer norms.

[0160] General Reasoning Process List (ThinkCritiques): Load the following standard dimensions:

[0161] (1) Task Recognition Reward (weight) ): Evaluate whether this is a question-and-answer task.

[0162] (2) Logical Soundness (weight) ): Evaluate whether the logic of the thinking process is rigorous.

[0163] (3) Correct Reasoning (weight) ): Assess whether inferences drawn from historical facts are accurate.

[0164] (4) Error Identification (weight) ): Assess whether a self-check has been performed.

[0165] (5) Language Consistency (weight) ): Assess whether the thinking language is consistent.

[0166] (6) Redundancy (weight) ): Assess whether the thinking process is redundant.

[0167] 3.3 Evaluation of Structured Generative Criticism Modeling (SGCM) Implementation:

[0168] The SGCM module processes each candidate response according to the above list. Perform the assessment. For example, generate a score for each dimension. and It also provides structured critique of the evidence. For example, for the "Correct Reasoning" dimension, SGCM would locate "choosing the Tubo period..." and give it a high score and reasons.

[0169] 3.4 Adaptive Multilingual Reward Optimization (AMRO) Calculation of Rewards (corresponding to the reward model flow on the right side of the diagram):

[0170] The AMRO module receives criticism information, computes sub-rewards in parallel, and performs complex post-processing:

[0171] (1) Parallel sub-reward calculation:

[0172] Language consistency reward ( ): Detected For Tibetan, and the target Consistent, therefore = .

[0173] Task recognition reward ( The model correctly predicted: For OEQA, with the tag Consistent, therefore = .

[0174] Structured reward for reasoning process ( First, calculate the weighted sum. (For example, a score of 0.92). Calculate the standard deviation of the thinking process score. Assuming If the value is very small (considering stability), then the continuously decreasing weight function is... .final .

[0175] Final answer structured reward ( ): Calculate the weighted sum of the answers (For example, a score of 0.90). Key step: Also multiply by the weighting factor of the thought process. .final .

[0176] (2) Post-reward processing and advantage calculation:

[0177] Initial reward: The weighted sum of the four sub-rewards mentioned above is obtained. (For example, 0.925). Similarly, calculate... .

[0178] Length-aware loss normalization: Calculate the total number of tokens in the output of this group. The loss is normalized to prevent long text bias.

[0179] Variance stabilization: assuming a sample size of 100%. Calculate the group's reward directly. mean and standard deviation .

[0180] Advantages calculation: Calculate This standardized odds value will be directly used for GRPO loss calculation.

[0181] 3.5 Strategy Optimization and Updates:

[0182] Use the calculated advantage value Combined with KL constraints (constraints and reference models) (differences), by minimizing Loss to update policy model The parameters are optimized. Responses with higher rewards (and more stable thought processes) are reinforced, thereby enabling targeted optimization of the model on Tibetan question-answering tasks.

[0183] This embodiment fully demonstrates the workflow of the present invention on complex tasks involving low-resource languages ​​(Tibetan):

[0184] 1. Planning guidance: The model follows a structured generation paradigm of "think first, then answer", and the process is transparent and controllable.

[0185] 2. Multi-dimensional verification: Through diagrammatic thinking evaluation and question-and-answer evaluation, the factuality, logic and language consistency of the answers were comprehensively evaluated.

[0186] 3. Adaptive optimization: AMRO module introduction The weight reduction function and variance stabilization mechanism effectively solve the problems of large evaluation fluctuations and unstable training in low-resource scenarios, and successfully drive the efficient convergence of the model.

[0187] 4. Cross-language stability: The entire assessment system is based on a language-independent list, ensuring that the standards for assessing Tibetan are consistent with those for high-resource languages.

[0188] Example 2:

[0189] like Figure 3 As shown, this embodiment of the invention provides a planning-guided adaptive verifiable multilingual question-answering system, comprising the following modules:

[0190] The supervised fine-tuning module 31 is used to construct the input dataset and task set, and to perform supervised fine-tuning on the base model to obtain a fine-tuned base model with preliminary multilingual generation and structured evaluation capabilities.

[0191] The reinforcement learning module 32 uses the fine-tuned base model as the policy model. By constructing a structured checklist module, it decomposes complex tasks into multi-dimensional logical paths and pre-constraints to guide the policy model in following preset inference nodes when generating answers and thinking processes. At the same time, it constructs a structured generative criticism modeling module to trace the source, verify logical consistency, and locate errors for each planning node in the thinking process, generating verifiable structured criticism feedback. Finally, using an adaptive multilingual reward optimization module, it calculates the reward value and advantage function by combining the checklist achievement degree and the criticism verification results, and performs closed-loop iterative optimization of the policy model through reinforcement learning algorithms.

[0192] A planning-guided adaptive verifiable multilingual question answering device includes one or more electronic devices, wherein the one or more electronic devices are used to implement the planning-guided adaptive verifiable multilingual question answering method.

[0193] An electronic device includes: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a program-guided adaptive verifiable multilingual question-answering method.

[0194] A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement a planned, guided, adaptive, verifiable, multilingual question-answering method.

[0195] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.< / answer> < / think> < / think>

Claims

1. A planning-guided adaptive verifiable multilingual question-answering method, characterized in that, include: Step S1: Construct the input dataset and task set for supervised fine-tuning of the base model to obtain a fine-tuned base model with preliminary multilingual generation and structured evaluation capabilities; Step S2: Using the fine-tuned base model as the strategy model, a structured checklist module is constructed to decompose the complex task into multi-dimensional logical paths and pre-constraints, guiding the strategy model to follow preset reasoning nodes when generating answers and thinking processes. Simultaneously, a structured generative criticism modeling module is constructed to trace the source, verify logical consistency, and locate errors for each planning node in the thinking process, generating verifiable structured criticism feedback. Finally, an adaptive multilingual reward optimization module is used to calculate the reward value and advantage function by combining the checklist achievement degree and criticism verification results, and the strategy model is iteratively optimized in a closed loop using a reinforcement learning algorithm.

2. The planning-guided adaptive verifiable multilingual question-answering method according to claim 1, characterized in that, Step S1: Constructing the input dataset and task set for supervised fine-tuning of the base model to obtain a fine-tuned base model with preliminary multilingual generation and structured evaluation capabilities, specifically including: Step S11: Construct the dataset, including the instruction fine-tuning dataset and the multilingual dataset; Step S12: Construct a task set including: natural language reasoning, text creation, text completion, heuristic text generation, summarization, dialogue, and open-ended question answering.

3. The planning-guided adaptive verifiable multilingual question-answering method according to claim 2, characterized in that, The construction of the structured inventory module, the structured generative criticism modeling module, and the adaptive multilingual reward optimization module in step S2 specifically includes: Step S21: The structured list module is used to define and store task-specific answer lists and general reasoning process lists. Each list defines multiple evaluation dimensions and their corresponding weights. Each list is stored in the form of evaluation tuples {dimension, weight}, and the total weight is 1. Step S22: The structured generative criticism modeling module is used to receive the query-reasoning-answer triple generated by the model, score each evaluation dimension independently according to the list, and output structured criticism results; Step S23: The adaptive multilingual reward optimization module is used to fuse the structured criticism results with contextual information, calculate multiple rewards, and weight them to synthesize a final reward signal to drive the optimization of the policy model.

4. The planning-guided adaptive verifiable multilingual question-answering method according to claim 3, characterized in that, The task-specific answer list and the general reasoning process list in step S21 specifically include: Step S211: Define the evaluation dimensions of the task-specific answer list: semantic validity, task completion, information fidelity, and format compliance, as well as the corresponding weights for each dimension; Step S212: Define the evaluation dimensions of the general reasoning process checklist: task recognition reward, logical rigor, reasoning correctness, error recognition, language consistency and redundancy, and the corresponding weights for each dimension.

5. The planning-guided adaptive verifiable multilingual question-answering method according to claim 4, characterized in that, Step S22: The structured generative criticism modeling module receives the query-inference-answer triples generated by the model, independently scores each evaluation dimension according to the list, and outputs structured criticism results, specifically including: Step S221: Receive a tuple output from the policy model, the tuple containing three parts: user query, inference trajectory generated by the model, and final answer; determine the task category to which the user query belongs, and obtain a task-specific answer list and a general inference process list corresponding to the task category from the structured list module; Step S222: Perform dimensional analysis on the task-specific answer list and the general reasoning process list to generate structured criticism, including: the dimension name, score and reasoning text for each evaluation dimension.

6. The planning-guided adaptive verifiable multilingual question-answering method according to claim 3, characterized in that, Step S23: The adaptive multilingual reward optimization module is used to fuse the structured criticism results with contextual information, calculate multiple rewards, and weight them to synthesize a final reward signal to drive policy model optimization, specifically including: Step S231: Based on the structured criticism, contextual information, and user query text, calculate the reward, including parallel calculation of the following four sub-rewards: a. Language Consistency Reward: Determines whether the generated content is consistent with the target language; b. Task Recognition Reward: Determines whether the model correctly identifies the task type; c. Structured reward for the reasoning process: The scores of each dimension of the reasoning process are weighted and summed, and then multiplied by a weighting factor based on the variance of the scores; d. Structured reward for the final answer: The scores for each dimension of the answer are weighted and summed, and then multiplied by the same weighting factor; Finally, the four sub-rewards mentioned above are weighted and merged, and combined with the structural invalidity penalty term to form the final reward; Step S232: Length-aware loss normalization is used to normalize the reward according to the total number of tokens in the generated samples, so as to eliminate the bias introduced by different generated sample lengths to the optimization process; Variance stabilization is employed to calculate the mean and standard deviation of the statistical rewards for the current batch or historical window, thereby maintaining the distinguishability and stability of the reward signal. Based on the mean and standard deviation of the rewards, the rewards for each sample are standardized to calculate the standardized advantage value, which is then input into the loss function of the group relative policy optimization to drive the parameter update of the policy model.

7. A planning-guided adaptive verifiable multilingual question-answering system, characterized in that, Includes the following modules: The supervised fine-tuning module is used to construct the input dataset and task set, and to perform supervised fine-tuning on the base model to obtain a fine-tuned base model with preliminary multilingual generation and structured evaluation capabilities. The reinforcement learning module uses the fine-tuned base model as the policy model. It decomposes complex tasks into multi-dimensional logical paths and pre-constraints by constructing a structured checklist module, guiding the policy model to follow preset inference nodes during answer generation and thought processes. Simultaneously, a structured generative criticism modeling module is constructed to trace the source, verify logical consistency, and locate errors at each planning node in the thought process, generating verifiable structured criticism feedback. Finally, an adaptive multilingual reward optimization module calculates the reward value and advantage function by combining checklist achievement and criticism verification results, and performs closed-loop iterative optimization of the policy model using a reinforcement learning algorithm.

8. A planning-guided adaptive verifiable multilingual question-answering device, characterized in that, It includes one or more electronic devices, wherein the one or more electronic devices are used to implement the method of any one of claims 1 to 6.

9. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1 to 6.

Images