Large language model training method and system based on hybrid verification and medium

Abstract

The invention discloses a large language model training method and system based on hybrid verification and a medium, and relates to the technical field of quantum information and artificial intelligence, and the method comprises the steps: constructing a quantum scientific data set, and dividing the quantum scientific data set into a first subset and a second subset; utilizing the first subset to supervise and finely adjust the large language model, thereby obtaining an initial strategy model after the quantum knowledge is injected; performing reinforcement learning optimization on the strategy model based on the verification perception reward model and the second subset to obtain a target large language model with scientific reasoning ability; the verification perception reward model is used for evaluating a plurality of candidate answers generated by the strategy model and outputting a plurality of reward values; constructing an optimization target based on the plurality of reward values, and updating parameters of the strategy model in a mode of maximizing expected return; the model training method aims at solving the problems of training data scarcity, physical constraint deficiency and reward over-optimization and illusion caused by traditional human feedback reinforcement learning.

Description

Technical Field

[0001] This invention relates to the fields of quantum information and artificial intelligence, and in particular to a method, system, and medium for training large language models based on hybrid verification. Background Technology

[0002] In recent years, large language models (LMs) based on the Transformer architecture have made groundbreaking progress in natural language understanding and generation tasks. After pre-training on massive amounts of data and subsequent supervised fine-tuning (SFT) and reinforcement learning based on human feedback (RLHF), existing general-purpose large language models have achieved remarkable intent understanding and text generation capabilities.

[0003] However, when applying large language models to vertical scientific fields such as quantum mechanics and high-energy physics, existing technical solutions face severe challenges. Scientific reasoning tasks differ from everyday conversations, demanding that large language models possess extremely high logical rigor, symbolic computation accuracy, and strict adherence to physical laws. In this scenario, existing technologies exhibit the following significant shortcomings and deficiencies: 1. The extreme scarcity of high-quality process data in the field is the primary bottleneck restricting the development of scientific big language models.

[0004] Lack of fine-grained reasoning steps: Existing open-source datasets (such as GSM8K and MATH) mainly focus on elementary mathematics or general common sense, lacking specialized data for higher-order physics fields such as quantum mechanics. Existing physics data typically only contains the "question" and the "final answer," severely lacking rigorously validated, step-by-step intermediate reasoning processes.

[0005] Unstructured knowledge is difficult to utilize: Although there is a lot of physical knowledge in textbooks and papers, it exists in the form of unstructured natural language. Large language models cannot directly internalize rigorous operator operation rules or state evolution logic from it through self-supervised learning.

[0006] Manual annotation is too costly: Quantum mechanics and other fields have extremely high cognitive barriers, and relying on field experts to perform large-scale process-level annotation is not feasible in terms of time and economic cost.

[0007] 2. Existing reward models fail in specialized fields. Current mainstream RLHF training paradigms rely on reward models to provide optimization signals, but this is difficult to implement effectively in scientific fields. Sparsity of outcome supervision: Rewarding only the final answer can lead to an overly sparse signal. For complex quantum computing problems, even if the final answer happens to be correct, the intermediate processes may be riddled with errors, and these "pseudo-correct" results can mislead the optimization direction of the model.

[0008] Limitations of human preferences: Traditional reward models primarily fit subjective human preferences (such as the usefulness and safety of responses), lacking the ability to judge objective physical truth. Reward models themselves also lack knowledge of quantum mechanics, making it impossible to accurately identify errors in formula derivation during the reasoning process.

[0009] The disconnect between validation and training: Although existing technologies attempt to introduce external calculators or solvers as auxiliary tools, these tools are usually only used to assist in the reasoning stage and fail to convert their validation results into effective gradient signals for backpropagation to the large language model. As a result, the scientific reasoning ability of the large language model itself has not been fundamentally improved.

[0010] In summary, overcoming the scarcity of high-quality process data and constructing a training method that can automatically verify reasoning steps and transform domain knowledge into dense reward signals are key technical problems that urgently need to be solved in current research on large scientific language models. Summary of the Invention

[0011] Based on the technical problems existing in the background technology, this invention proposes a training method, system and medium for large language models based on hybrid verification. It aims to solve the problems of scarce training data, lack of physical constraints and reward over-optimization and illusion caused by traditional human feedback reinforcement learning (RLHF) in rigorous scientific fields such as quantum mechanics.

[0012] The proposed method for training large language models based on hybrid validation includes: Construct a quantum science dataset that includes quantum knowledge questions and answer labels and divide it into a first subset and a second subset; The large language model is fine-tuned under supervision using the first subset, thereby obtaining the initial policy model after quantum knowledge injection; A large language model with scientific reasoning capabilities is obtained by optimizing the policy model through reinforcement learning based on the verification-aware reward model and the second subset. The verification-aware reward model evaluates multiple candidate answers generated by the policy model based on quantum knowledge questions in the second subset and outputs multiple reward values ​​that integrate deterministic verification signals and multidimensional semantic evaluation signals from the scientific execution suite. An optimization objective is constructed based on these multiple reward values, and the parameters of the policy model are updated by maximizing the expected return to increase the probability of the policy model generating answers with high reward values. The scientific execution suite is a symbolic and numerical solver for performing mathematical operations and deterministic verification of physical constraints.

[0013] Furthermore, the construction process of the quantum science dataset is as follows: Expanding the topic breadth based on the seed evolution paradigm to obtain initial data; The initial data is processed using a task-adaptive strategy: for retrieval-intensive tasks, concise answer tags are generated; for complex reasoning tasks, answer tags and thought chains containing detailed derivation steps are generated. Perform dual hybrid validation on the processed data, specifically as follows: The first layer of automated hybrid validation: uses a scientific execution suite to perform deterministic validation of the physical consistency and mathematical correctness of the processed data, and uses an independent large language model to perform semantic evaluation of logic and format. The second layer of human-machine collaborative auditing: For data that has passed the first layer of verification, a tiered sampling manual review is performed based on the difficulty label; Data filtered through a dual hybrid verification mechanism will be used as a quantum science dataset.

[0014] Furthermore, the second-layer human-machine collaborative audit includes a quality feedback closed-loop mechanism, specifically: Set a batch rejection threshold; The verification error rate of the current batch of data stratified sampling is calculated. If the verification error rate exceeds the batch rejection threshold, the current batch of data is determined to be invalid. The error patterns of the failed data are analyzed, and the generation instructions in the seed evolution paradigm are corrected accordingly, triggering the regeneration process of the initial data.

[0015] Furthermore, the construction process of the verification perception reward model is as follows: Configure the scientific execution suite to perform deterministic validation on verifiable dimensions among multiple candidate answers generated by the strategy model and output validation indicators; A dual-head parallel prediction network is constructed based on a pre-trained Transformer encoder, including a shared encoder backbone network, a multi-dimensional scoring head, and a dynamic weight allocation head; The shared encoder backbone network generates contextual representations based on the input question and the corresponding candidate answers; The multidimensional scoring head maps the contextual representation to a semantic evaluation score, which is used to evaluate the semantic quality of the generated content. The dynamic weight allocation header concatenates the context representation with the verification indicator and outputs the dynamic weights for each evaluation dimension. Dynamic reward calibration mechanism: For each evaluation dimension, the fusion score is calculated using the semantic evaluation score and the confidence adjustment coefficient; The final reward value is calculated by weighting and aggregating the dynamic weights and fusion scores.

[0016] Furthermore, the fusion score The calculation formula is as follows: ; in, for The confidence level adjustment coefficient, For the first Semantic evaluation scores in each dimension For the first Verification indicators for each dimension; The confidence adjustment coefficient satisfy: when hour, Verify that the perceived reward model fully adopts deterministic rewards; when hour, ; when hour, The default value is close to 0.

[0017] Furthermore, the verification-aware reward model is initialized through Oracle-guided distillation training, specifically as follows: Using a heterogeneous large language model ensemble as judges, soft target scores and ideal sample importance weights are generated for sample data containing the results of scientific execution suites. Using the soft target score and the importance weight of the ideal sample as supervision signals, the verification perception reward model is trained through a multi-task loss function to fit the judge's evaluation distribution and align it with their discrimination preferences.

[0018] Furthermore, the division into a first subset and a second subset is specifically as follows: The quantum science dataset is directly divided into a first subset and a second subset; or, Construct an auxiliary dataset containing general domain instruction data; The first subset is constructed by proportionally mixing a predetermined number of data points from the quantum science dataset with auxiliary datasets. The second subset is constructed by proportionally mixing a predetermined number of data from the remaining quantum science dataset, the auxiliary dataset, and a predetermined number of high-difficulty long-chain inference samples from the first subset.

[0019] Furthermore, the reinforcement learning optimization of the policy model based on the verification-aware reward model and the second subset specifically includes: The advantage function is estimated based on the reward value output by the verification-aware reward model; wherein, the calculation of the reward value assigns a higher weighting coefficient to the deterministic verification signal from the scientific execution suite than to the multidimensional semantic evaluation signal; Construct a proximal policy optimization objective function that includes a truncation function, wherein the objective function is optimized with the advantage function as the optimization guide; Finally, the parameters of the policy model are updated by maximizing the objective function.

[0020] A computer system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described above.

[0021] A computer-readable storage medium having stored a plurality of programs for being invoked by a processor and executing the method described above.

[0022] The advantages of the hybrid verification-based training method, system, and medium for large language models provided by this invention are as follows: It aims to address the problems of scarce training data, lack of physical constraints, and reward over-optimization and seemingly plausible illusions caused by traditional human feedback reinforcement learning (RLHF) in rigorous scientific fields such as quantum mechanics, which are prevalent in existing large language models. Specifically, through a task-adaptive data construction strategy and dual hybrid verification (combining the Scientific Execution Suite (SES) and semantic evaluation), it effectively solves the problem of seemingly reasonable but actually erroneous training data. The Verification-Aware Reward Model (VRM) introduces multi-dimensional semantic evaluation signals (mathematical correctness, physical consistency, and instruction compliance) and deterministic verification signals from the Scientific Execution Suite (SES), ensuring rigor while compensating for the sparsity of deterministic feedback, achieving a dynamic balance between general preference optimization and rigorous scientific supervision. This significantly improves the stability of the reinforcement learning process and prevents the large language model from experiencing capability collapse during training. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the structure of the present invention. Detailed Implementation

[0024] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0025] like Figure 1 As shown, the large language model training method based on hybrid validation proposed in this invention includes: Step 1: Construct a quantum science dataset that includes quantum knowledge questions and answer labels, and divide it into a first subset and a second subset; Step 2: Use the first subset to perform supervised fine-tuning of the large language model, thereby obtaining the initial policy model after quantum knowledge injection; Step 3: Optimize the policy model using reinforcement learning based on the verification-aware reward model and the second subset to obtain a target large language model with scientific reasoning capabilities. The verification-aware reward model evaluates multiple candidate answers generated by the policy model based on the quantum knowledge questions in the second subset and outputs multiple reward values ​​that integrate deterministic verification signals and multidimensional semantic evaluation signals from the scientific execution suite. An optimization objective is constructed based on these reward values, and the parameters of the policy model are updated by maximizing the expected return to increase the probability of the policy model generating answers with high reward values. The scientific execution suite is a symbolic and numerical solver for mathematical operations and deterministic verification of physical constraints.

[0026] This embodiment aims to address the problems of scarce training data, lack of physical constraints, and reward over-optimization and illusion caused by traditional human feedback reinforcement learning (RLHF) in existing large language models in rigorous scientific fields such as quantum mechanics.

[0027] This embodiment provides a general language model alignment scheme for scientific fields. It is not only applicable to quantum mechanics but also aims to provide a general framework for scientific reasoning alignment. By incorporating rule-based verification feedback into a reinforcement learning loop, the method of this embodiment can be extended to other disciplines such as mathematics, physics, and chemistry, which have extremely high requirements for logical rigor and axiomatic constraints, providing a highly reliable technical foundation for AI for Science.

[0028] Preferably, step one involves constructing a quantum science dataset that includes quantum knowledge questions and answer labels, and dividing it into a first subset and a second subset, specifically as follows: (1) Quantum Science Dataset Based on Task Adaptation and Hybrid Validation (QuantumQA); To support rigorous reasoning training and evaluation under physical constraints, this embodiment constructs a large-scale, rigorously validated quantum science dataset (named QuantumQA, with approximately 92,749 samples) containing multiple task types. The dataset features a highly systematic and diverse question design, covering various formats including short answer questions, calculation problems, fill-in-the-blank questions, true / false questions, and multiple-choice questions.

[0029] The construction process of this quantum science dataset aims to overcome the illusion problem existing in existing data and simultaneously support supervised fine-tuning (SFT) and reinforcement learning (RL) processes. This embodiment employs a task-adaptive data generation mechanism, customizing response structures for tasks of varying complexity. The specific construction process of the quantum science dataset is as follows: (a1) Expand the topic breadth based on the seed evolution paradigm to obtain initial data; The breadth expansion based on the "Seed-Evolve" paradigm differs from traditional data crawling. This embodiment adopts the "Seed-Evolve" paradigm. First, authoritative quantum mechanics textbooks and classic literature are acquired as raw corpora. These textbooks are digitized using Optical Character Recognition (OCR) technology. Subsequently, a large language model (e.g., DeepSeek V3) is used to extract core theorems from the digitized text based on specific prompts. To eliminate redundancy, a pre-trained semantic model (e.g., a BERT model with an all-MiniLM-L6-v2 architecture) is introduced to calculate the semantic similarity of the extracted content, filtering out entries exceeding a preset threshold, thereby generating a unique and high-quality seed library. Based on this seed library, a large language model (e.g., Qwen3-Max) is used for systematic concept decomposition, refining each baseline seed into multiple fine-grained topics, forming a broad coverage from basic theory to physical implementation. Similarly, the pre-trained semantic model is used to semantically deduplicate the generated topics, constructing a high-quality topic pool.

[0030] (a2) The topic pool is processed using a task-adaptive strategy: a heterogeneous model ensemble strategy (e.g., integrating DeepSeek V3, Qwen3-Max, and ChatGPT-5) is used to generate diverse question-and-answer data. For retrieval-intensive and basic tasks, concise answer tags are generated; for complex reasoning tasks, answer tags and thought chains containing detailed derivation steps are generated. To address the varying cognitive needs of scientific tasks, this embodiment proposes a task-adaptive data construction strategy that dynamically adjusts the response structure of data according to the task type to suppress the illusion of training models (including the large language model in the supervised fine-tuning stage and the policy model in the reinforcement learning stage). This construction strategy effectively characterizes the comprehensive capabilities of the training model in symbolic computation, logical reasoning, and quantum mechanical formula derivation, avoiding the training model from merely remaining at the level of superficial conceptual memory or pattern matching.

[0031] The task is divided into 5 question types, specifically: For multiple-choice, true / false, and fill-in-the-blank questions, an end-to-end generation method is used to first generate questions related to the target topic, and then generate the corresponding standard answers, distractors (if it is a multiple-choice question), and corresponding difficulty tags. Among them, the distractors are generated based on preset common physics misconceptions to ensure the question's differentiation.

[0032] For short answer and calculation questions, a step-by-step generation method is adopted: first, questions related to the target topic are generated; then, a structured standard problem-solving path, i.e. the answer, is generated based on the question, and the corresponding difficulty label is marked.

[0033] To improve the inference quality of the training data, this embodiment introduces an adaptive reasoning path injection mechanism to dynamically determine the necessity of reasoning paths. The specific execution process is as follows (b1) to (b4): (b1) Constructing tuple data: The generated questions ( ), standard answer ( ) and difficulty tags ( ) to form triplet data ( ), input independent large language models (such as Qwen3-Max or ChatGPT-5).

[0034] (b2) Logical necessity determination: The instruction heterogeneous big language model determines the necessity based on the difficulty label and the question content: if the conversion from question to answer label involves logical deduction, multi-step calculation or deep concept analysis (usually corresponding to the "calculation problem" task), it is determined to be "reasoning necessary"; if the task only involves direct fact retrieval or definition query, it is determined to be "reasoning not necessary".

[0035] (b3) Conditionalized Reasoning Generation: In response to the determination of "reasoning necessity," the heterogeneous large language model (i.e., a collection of multiple large language models) generates a rigorous, reverse derivation process based on the question and answer labels, and encapsulates this derivation process in specific identifiers (e.g., <think>Within the tag (i.e., for complex reasoning tasks, generating a thought chain with detailed derivation steps); in response to the judgment result of "reasoning is not necessary", the heterogeneous big language model skips the generation of the derivation process based on the question and answer, generating an empty thought chain (i.e., for retrieval-intensive tasks, generating a concise answer tag without a thought process), thereby avoiding "reasoning hallucination" in simple factual questions.

[0036] (b4) Structured parsing and synthesis: Parsing the output of the heterogeneous large language model. If an identifier (e.g.) is detected... <think>If a valid reasoning path is contained within the original question-answer pair, it is extracted and formatted into an independent thought process component. This component is then merged with the original question-answer pair to form an augmented data entry that is ultimately used to train the model (including the large language model in the supervised micro-stage and the policy model in the reinforcement learning stage).

[0037] (a3) Perform dual hybrid validation on the processed data; In terms of knowledge coverage and quality control, the quantum science dataset systematically covers core high-frequency research areas such as the fundamentals of quantum mechanics, quantum computing, quantum optics, quantum information, and quantum many-body systems, maintaining a balanced distribution across categories. To ensure the scientific rigor of the data, this embodiment introduces a hybrid verification protocol.

[0038] The Hybrid Verification Protocol is designed to ensure the scientific rigor of the questions and answer labels generated by heterogeneous large language models. Specifically: (c1) First-level automated hybrid verification: Deterministic verification of physical consistency and mathematical correctness of processed data is performed using a scientific execution suite, and semantic evaluation of logic and format is performed using an independent large language model; The Scientific Execution Suite (SES, which includes symbolic and numerical solvers such as SymPy and QuTiP) constructed in this embodiment is used to perform deterministic verification of mathematical operations and physical constraints in the processed data.

[0039] The scientific execution suite in this embodiment is essentially a series of executable packages, such as SymPy (for precise symbolic algebraic operations and analytical derivations), QuTiP (for operator operations and dynamics simulations of quantum systems), SciPy (for complex numerical solutions and integration of ordinary differential equations), SageMath (for integrated symbolic-numerical computations), and Z3-solver (for rigorous verification of mathematical and physical constraints), thereby performing deterministic verification of mathematical operations and physical constraints in the processed data.

[0040] Semantic Criticism: Utilizing large language models (such as DeepSeek-V3) as semantic critics to conduct semantic consistency checks on the generated data.

[0041] (c2) Second-level human-machine collaborative audit: For data that has passed the first-level verification, perform stratified sampling and manual review based on difficulty labels; Human-in-the-Loop (HITL) Auditing and Dynamic Rejection: Building upon scientific execution suites and automated semantic evaluation verification, a human-in-the-loop (HITL) auditing process is introduced. Specifically, this involves setting dynamic rejection thresholds. (For example When the error rate of a batch of data exceeds the threshold, an iterative correction mechanism is triggered until the data quality reaches the gold standard of high reliability.

[0042] (c3) Data filtered through a dual hybrid verification mechanism will be used as a quantum science dataset; Data verified through scientific execution suites, semantic evaluation, and human-machine collaborative auditing will be used as quantum science datasets.

[0043] This embodiment establishes a task-adaptive, high-quality quantum science dataset and model iteration paradigm. To overcome the challenge of a lack of high-quality process supervision data in the scientific field, this embodiment combines a task-adaptive data construction strategy to dynamically adjust the inference structure for tasks of varying difficulty (e.g., concise answers for simple tasks and thought chain derivation for complex tasks). By incorporating a hybrid verification protocol of Human-Machine Teaming (HITL), a highly reliable quantum science dataset is constructed, achieving inference performance that reaches or even surpasses that of large-scale proprietary models (such as ChatGPT and DeepSeek) with a relatively small parameter scale.

[0044] The quantum science dataset is not only suitable for domain knowledge injection in the supervised fine-tuning (SFT) stage, but also provides a high-confidence evaluation and feedback basis for subsequent construction of verification-aware reward models (VRM) and implementation of reinforcement learning (RLVR) based on verification-aware reward models.

[0045] (a4) Construction of the first and second subsets; The quantum science dataset covers five task types: short answer questions, fill-in-the-blank questions, true / false questions, multiple choice questions, and computational questions emphasizing multi-step derivation. In the supervised fine-tuning and subsequent reinforcement learning stages, this embodiment provides two optional data configuration methods for constructing the training data. The first method involves using only the aforementioned quantum science dataset as training data, directly dividing it into a first subset, a second subset, and a test set. This method maximizes the depth of the training model's fit to specific domain knowledge.

[0046] The second approach is the preferred approach in this embodiment. Since training with pure data in a specific domain can easily lead to mode collapse, to maintain the general representational ability of the trained model, the training data in this approach not only includes the aforementioned quantum science dataset but also proportionally mixes in an auxiliary dataset containing some general domain instruction data. The beneficial effect of introducing general instruction data for mixed training is that it can effectively alleviate the catastrophic forgetting phenomenon that easily occurs when a large amount of vertical domain knowledge is injected into the model, thereby significantly improving the model's reasoning ability in the quantum science domain while firmly retaining its original basic language understanding, dialogue interaction, and general instruction compliance capabilities. Furthermore, in terms of data usage, the second approach in this embodiment adopts a hierarchical partitioning strategy: The first subset, consisting of approximately 70% (optional) of the quantum science dataset and an auxiliary dataset mixed with general domain instruction data, is used for supervised fine-tuning of the large language model; thereby establishing the basic instruction compliance and domain knowledge capabilities of the large language model.

[0047] The second subset consists of approximately 25% (optional) of the remaining quantum science dataset, mixed with approximately 5% (optional) of high-difficulty long-chain inference samples randomly drawn from the first subset, and then mixed with an auxiliary dataset of general domain instruction data. This hybrid strategy aims to stabilize the training distribution by introducing some known high-difficulty long-chain inference samples, while effectively mitigating the reward over-optimization problem by utilizing unseen data, thereby improving the generalization ability of the policy model in the reinforcement learning phase.

[0048] Test set: Approximately 5% of the quantum science dataset is reserved as a strictly isolated test benchmark to evaluate the generalization ability of the policy model.

[0049] In one embodiment, step two involves supervising and fine-tuning the large language model using the first subset to obtain the policy model after quantum knowledge injection; specifically: This step aims to align the output of the large language model to a structured and easily verifiable reasoning paradigm through supervised fine-tuning using a first subset. Traditional supervised fine-tuning (STF) often leads to over-reasoning in simple problems or missing steps in complex problems, which is detrimental to subsequent objective verification. This embodiment adopts strategies (d1) to (d2): (d1) During the supervised fine-tuning phase, the large language model is forced to learn two distinct output patterns: For retrieval-intensive and basic concept tasks, the supervised fine-tuning model outputs concise and compact answers to reduce semantic noise. For complex reasoning tasks, the supervised fine-tuning model outputs answers along with a thought chain (CoT) containing detailed derivation steps, explicitly generating formula selection, symbolic derivation, and numerical computation steps. This structured output aims to maximize the exposure of the reasoning process, enabling the Science Execution Suite (SES) during the reinforcement learning phase to accurately capture and validate intermediate steps.

[0050] (d2) The multi-task hybrid and cold-start strategy adopts a proportionally hybrid sampling strategy, mixing the first subset of short answer, multiple choice, fill-in-the-blank, and problem solving tasks for training. The beneficial effects of this hybrid training strategy are: by introducing diverse tasks for joint learning, it can effectively avoid overfitting of the trained model in a single task during the supervised fine-tuning stage, and significantly improve the generalization ability of the trained model across tasks and the robustness of the underlying logical reasoning.

[0051] Specifically, the first subset of this embodiment includes high-difficulty long-chain inference samples; by using high-difficulty long-chain inference samples as cold-start data, the large language model after supervised fine-tuning has the ability to follow physical axioms and the normativity of symbolic computation before entering the reinforcement learning stage, thereby reducing the exploration space in the early stage of reinforcement learning and avoiding convergence difficulties.

[0052] In one embodiment, a verification-aware reward model based on verification perception and adaptive fusion is used; specifically: This embodiment proposes a Verification-Aware Reward Model (VRM), which aims to combine deterministic scientific tool verification signals with the inherent semantic evaluation capabilities of the VRM to construct a unified reward mechanism with physical and mathematical constraints and instruction compliance. The VRM mainly consists of a Scientific Execution Suite (SES), a shared encoder backbone network, a multi-dimensional scoring head, and a Dynamic Weight Allocation Head (DWA Head).

[0053] (e1) Configure the scientific execution suite to perform deterministic verification on verifiable dimensions in the response of the verification-aware reward model and output verification indicators; First, the second subset of problems was addressed using the Scientific Execution Suite (SES). With strategy models for the problem Generated candidate answers Perform verification dimension testing. Evaluate the set of dimensions. Divided into verifiable dimensions (including mathematical correctness) Physical consistency ) and pure semantic dimension (e.g., instructions follow Inst).

[0054] The scientific execution suite performs deterministic validation for each dimension and outputs a ternary validation indicator. , For dimensional indexing: : This indicates that the constraints have been satisfied through execution verification (e.g., the formula derivation is correct and the conservation law holds). : This indicates that the execution was successful but a violation was detected (such as numerical error or unit mismatch). : Indicates that execution is unavailable or yields no reliable results (applicable to purely semantic dimensions) Or parsing failure scenarios).

[0055] The validation indicators of all the above dimensions constitute the validation feedback vector. It is input into the neural network as a deterministic external signal.

[0056] (e2) Dual-head parallel prediction network architecture; A dual-head parallel prediction network is constructed based on a pre-trained Transformer encoder, comprising a shared encoder backbone network, a multi-dimensional scoring head, and a dynamic weight allocation head; the shared encoder backbone network... Backbone network Receive input pair And extract a unified contextual representation. Based on this, two parallel dedicated output modules (both using a 3-layer MLP (Multilayer Perceptron) structure and configured with the GeLU activation function) are designed to perform predictions respectively: Multi-dimensional Scoring Head: Used to evaluate the semantic quality of the generated data. This scoring head represents the context. Mapped to semantic evaluation score This provides an intrinsic quality estimate for the model, ensuring that there is still a dense supervisory signal when tools are unavailable.

[0057] Dynamic Weight Allocation Head (DWA Head): As a gated network, it is used to dynamically estimate the dynamic weights of each dimension. This DWA head represents the context. External verification feedback vector The data is concatenated, fed into a 3-layer MLP structure, and outputs the dynamic weights of each evaluation dimension. This mechanism enables the validation-aware reward model to automatically increase the weight of the reliability dimension or decrease the weight of the validation failure dimension based on feedback from the Scientific Execution Suite (SES).

[0058] (e3) Dynamic reward calibration and signal fusion; To address the sparse reward problem, this embodiment introduces a dynamic reward calibration mechanism. For the... Calculate the dimension. Fusion score corresponding to each dimension : ; in, For the first Semantic evaluation scores in each dimension This is the confidence level adjustment coefficient. When... When (verification successful), The validation-perceived reward model fully adopts deterministic rewards, which refer to the validation results of the scientific execution suite, such as validation passing or failing; when (Without verification) The verification of the perceptual reward model relies solely on semantic scoring. ;when When verification fails, use a very small coefficient, i.e. This reduces the penalty for failing deterministic verification while retaining the scoring of the semantic dimension to maintain optimization stability.

[0059] Final reward value The following is obtained through reliability-weighted aggregation: ; in, For the first Dynamic weights for each dimension To validate the feedback vector, this formula ensures that the optimization process is dominated by the dimension with reliable validation signals, while also taking into account the smooth supervision of the semantic dimension.

[0060] (e4) Oracle-guided distillation training; To initialize and validate the perceptual reward model, this embodiment employs a "LLM-as-a-Judge" pipeline for supervised learning. A sample dataset containing tool execution trajectories is constructed. Sample dataset Including instructions Candidate Responses , verification feedback vector Using Oracles models (a heterogeneous collection of large language models, such as DeepSeek and ChatGPT) as judges, the sample dataset is evaluated. Generate soft target scores and ideal weight This is used to construct the VRM training set. Each data set in the VRM training set includes instructions. Candidate Responses , verification feedback vector Soft target scoring and ideal weight .instruction Candidate Responses , verification feedback vector Soft target score serves as input for validating the perceived reward model. and ideal weight As a supervisory label for verifying the training process of the perceptual reward model.

[0061] In this embodiment, the specific training process is as follows: (f1) Construct a VRM training set for training the validation perceived reward model.

[0062] Specifically, for the VRM training set: instruction : That is, scientific questions (e.g., "calculate the energy of a photon with a wavelength of 620 nm").

[0063] Candidate Response : Corresponding to this instruction A candidate answer (which may be correct or contain various errors).

[0064] Validation feedback vector Scientific Execution Suite (SES) for candidate responses The ternary multidimensional verification indicator output after deterministic verification. It provides objective facts about the verifiable parts of the answer.

[0065] Supervision label: Soft target score and ideal weight These are generated from a powerful set of Oracle models (i.e., a heterogeneous set of large language models, such as DeepSeek, ChatGPT, etc.) and serve as targets to be fitted to validate the perceptual reward model.

[0066] (f2) Forward propagation: passing instructions Candidate Responses , verification feedback vector Input validation perception reward model (shared encoder backbone network → scoring head and dynamic weight allocation head).

[0067] (f3) Obtain the prediction: Verify that the perceptual reward model outputs the semantic evaluation score of its prediction. and dynamic weights .

[0068] (f4) Calculate the loss: Calculate the loss using the multi-task loss function (including mean squared error, MSE, etc.). Rating Regression Loss: Comparing Semantic Evaluation Scores With soft target scoring The gap; Weighted prediction loss: compare the predicted dynamic weights with ideal weight The gap.

[0069] (f5) Backpropagation and optimization: Jointly optimize the above loss and update all parameters of the verification perceptual reward model so that its predictive ability continuously approaches that of the Oracle model.

[0070] In this embodiment, the system employs a multi-task loss function to jointly optimize score regression and weight prediction. During this process, the oracle model, serving as a computationally expensive evaluation benchmark, provides high-precision supervision signals or truth references for training the validation-aware reward model. The validation-aware reward model, through the aforementioned joint optimization process, learns and fits the evaluation logic and weight allocation mechanism of the oracle model. Through this mechanism, the validation-aware reward model effectively internalizes and represents the scientific evaluation criteria inherent in the oracle model, thus essentially constructing a low-computational-cost equivalent proxy for the oracle model. Furthermore, in the subsequent reinforcement learning stage, this method eliminates the frequent need to invoke the computationally expensive oracle model; the validation-aware reward model can independently complete the evaluation task using only its own parameters, thereby significantly reducing computational overhead and achieving efficient and accurate evaluation.

[0071] (f6) Final goal: After the perceptual reward model has been trained, in the reinforcement learning stage of step three, the perceptual reward model only needs to receive questions from the second subset. Candidate answers corresponding to the strategy model to be optimized It can then leverage its internalized knowledge to automatically draw upon its internal "experience" (without needing to invoke the external Oracle model) and comprehensively output a high-quality reward value. This guides the optimization of the policy model during the reinforcement learning phase.

[0072] In one embodiment, step three involves optimizing the policy model using reinforcement learning based on the verification-perceptual reward model and the second subset to obtain a target large language model with scientific reasoning capabilities; specifically: Following supervised fine-tuning (SFT) of the large language model, this embodiment further introduces an alignment process based on quantum verifiable reinforcement learning. The core innovation of this process lies in using the adaptive fusion reward signal provided by the aforementioned Verification-Aware Reward Model (VRM) to guide the optimization and updating of the policy model.

[0073] In this embodiment, although Proximal Policy Optimization (PPO) is used as a typical representative of reinforcement learning algorithms, those skilled in the art should understand that the verification-aware reward signal provided by VRM has universality and can also be applied to other mainstream reinforcement learning or preference alignment algorithms.

[0074] (g1) Policy Sampling and Reward Calculation: The supervised fine-tuned large language model is used as the initial policy model for reinforcement learning; in each iteration, the quantum knowledge problem in the second subset is used. Input into the policy model to generate corresponding candidate answers. Then, the Verification Perceived Reward Model (VRM) and the Scientific Execution Suite (SES) were invoked to test the sample pairs. Perform multi-dimensional evaluation to obtain adaptive reward signals.

[0075] Configure the Scientific Execution Suite (SES) for execution: First try to... The mathematical and physical content in the code calls SES to perform deterministic verification of the signal and generate a verification feedback vector. .

[0076] Mixed-signal computation: Parallel computation of semantic scores within the VRM and dynamic weights generated by the Dynamic Weight Assignment Head (DWA Head). .

[0077] Scalar reward generation: via formula Calculate the final scalar reward. This reward value is characterized by dynamically adjusting the supervision strength based on the execution results of the verification tool. Specifically, it assigns high-weighted deterministic positive feedback when the tool passes verification, and smoothly falls back to semantic feedback when the tool becomes unavailable.

[0078] (g2) Dominance function estimation based on dynamic reward; In generating reward values Then, this embodiment utilizes The advantage function is calculated to measure the gain effect of taking the current response action relative to the average level in a specific state, thereby guiding the policy model parameters to be updated in the direction of high reward. In order to effectively reduce the estimation variance in complex quantum logic reasoning and maintain the policy model's keen ability to capture the verification perceived reward model (VRM), this embodiment preferably uses generalized advantage estimation (GAE).

[0079] Since validation-perceived reward models typically evaluate the final result of the inference sequence, the resulting reward value... It exhibits sparsity. In this embodiment, a dense, real-time reward distribution mechanism is employed to distribute the total reward value... Combined with the KL divergence penalty term to prevent policy deviation, it is broken down into instantaneous reward values ​​at each time step. , For time steps.

[0080] Typically, for non-terminating time steps in the inference path ( Its instant reward value Set as: ; And for the termination time step ( Its instant reward value for: ; in, To control the hyperparameters of the penalty intensity, and These represent the old and new strategies in a given context. The current generation The probability of.

[0081] Based on the aforementioned instant reward value Calculate time steps Timing difference error (TD-error) : ; in, This is a Critic network used to estimate the long-term expected reward of the current inference state in real time, providing a stable baseline for policy model updates. This Critic network is applicable not only to proximal policy optimization (PPO) but also to other mainstream Actor-Critic frameworks such as A2C. Essentially, this Critic network models the reward values ​​provided by the perceived reward model (VRM) over time, enabling the policy model to identify and learn which intermediate inference steps are more valuable for ultimately arriving at correct quantum science conclusions.

[0082] This is a discount factor used to balance the weight of current rewards and future long-term returns.

[0083] Indicates the current time step In this embodiment, the context refers to the semantic and logical context sequence of the current step. Specifically, it encompasses the quantum knowledge problem of the original input. and up to the current time step The generated sub-response fragment. This context. As input to the value evaluation network, it is used to assess the potential correctness of the currently generated path in solving a specific quantum physics or quantum computing problem.

[0084] To measure the gain effect of taking the current response (i.e., multiple answers generated by the policy model) relative to the average level under a specific state, this embodiment calculates the time step. Advantage function : ; in, For time step deviation, For time steps The timing difference error, This is the smoothing coefficient.

[0085] After obtaining all the advantage function values ​​in the current batch, in order to improve the numerical stability of the policy model training and accelerate convergence, this embodiment performs normalization on the advantage function values: ; in, Indicates time step The normalized advantage value, This represents the mean of the current batch's advantage value. This represents the standard deviation of the current batch's dominance value. To prevent extremely small constants with a denominator of zero, specifically, the original dominance values ​​are linearly transformed by calculating the mean and standard deviation of the dominance function values ​​within the batch. This ensures that the processed dominance distribution satisfies the standard normality property with a mean of 0 and a standard deviation of 1, thereby eliminating the influence of dimensions and balancing the gradient update magnitude.

[0086] (g3) Policy optimization objective function and CLIP mechanism; In the reinforcement learning optimization process, to prevent policy collapse when the policy model explores challenging scientific reasoning paths, this embodiment constructs a total optimization objective function that includes a truncation mechanism and KL divergence constraints. : ; in, Represents the expectation operator. It is the truncated proxy objective of Proximal Policy Optimization (PPO), and its specific calculation formula is as follows: ; This indicates the difference between the old and new strategies in the context. The probability ratio of generating the same action. This is a truncation function. To truncate the hyperparameters (e.g., to a value of 0.2), this function limits the policy update magnitude to [value missing]. Within a certain range, avoid excessively large single parameter updates that could cause drastic fluctuations in model performance.

[0087] This embodiment provides an evaluation index and method for a large language model. Based on the structural characteristics, answer formats, and reasoning requirements of different question types in the quantum knowledge question-answering task, this embodiment designs a comprehensive evaluation system covering exact matching, classification accuracy, and semantic assessment. The main evaluation indexes and methods for each question type are shown in Table 1 below: Table 1 Evaluation Indicators and Methods

[0088] Table 1 shows the output structure and corresponding main evaluation metrics configured for different question types in this embodiment. Specifically, for objective question types with a high degree of structure (such as multiple choice, true / false, and fill-in-the-blank questions), the system uses exact verification: for fill-in-the-blank questions requiring a unique numerical or symbolic answer, the exact match (EM) metric is used; for true / false questions that output binary classification results and multiple choice questions with single or multiple selection options, the accuracy metric is used. For subjective question types with a high degree of openness, the system uses semantic and logical verification: for short answer questions that output open text, the BERTScore metric is used to evaluate their semantic consistency; while for complex calculation questions that include derivation steps and numerical solutions, a large language model (such as Qwen3-235B-A22B-Instruct-2507 and other models) is introduced to evaluate the deep logic and reasoning process.

[0089] This embodiment constructs a multi-dimensional, hybrid verification and evaluation mechanism. This mechanism can dynamically adapt the corresponding verification strategy (i.e., question type matching) based on the specific question type characteristics of the question-and-answer data. This fine-grained evaluation system, customized for multiple question types, effectively ensures the comprehensiveness and accuracy of data verification in complex domains (such as calculation problems and short-answer questions involving large-scale formula derivation and logical reasoning), providing reliable quality feedback signals for subsequent training and optimization of large-scale models.

[0090] Example 1 This embodiment proposes a training method for large language models based on verification-aware reinforcement learning for the scientific reasoning domain. This method aims to address the problems of scarcity of high-quality training data and poor physical consistency faced by existing large language models when dealing with strongly logically constrained tasks such as quantum mechanics. The technical approach generally includes four core stages: first, constructing a scientifically rigorous quantum science dataset; second, establishing a verification-aware reward model (VRM) that integrates deterministic verification tools and multidimensional semantic evaluation; subsequently, using a portion of the constructed quantum science dataset for supervised fine-tuning (SFT) to achieve policy initialization; and finally, training the model using reinforcement learning based on the verification-aware reward model, enabling the model to maintain general reasoning capabilities while strictly adhering to the physical constraints and axiomatic systems of the scientific domain.

[0091] (h1) Construction of quantum science dataset; In the data construction phase, this embodiment adopts the "seed-evolution" paradigm to expand the breadth of topics and introduces a task-adaptive strategy: for retrieval-intensive and basic tasks, concise answer labels are generated; for complex reasoning tasks, answer labels and thought chains containing detailed derivation steps are generated to suppress training model illusions.

[0092] To ensure the scientific validity of the data, this embodiment employs a dual hybrid verification approach. It utilizes a Scientific Execution Suite (SES) to invoke automated scripts and symbolic solvers (such as SymPy) for deterministic verification, combined with semantic evaluation and Human-Machine Collaborative Auditing (HITL). This process effectively eliminates erroneous data that is mathematically correct but violates physical axioms (such as normalization and boundary conditions), thus constructing a high-confidence quantum science dataset.

[0093] (h2) Verify the construction of the perceived reward model; In the verification-aware reward model construction phase, to address the problem that traditional preference models struggle to identify subtle logical fallacies in scientific reasoning tasks, this embodiment constructs a verification-aware reward model (VRM). This model integrates two types of supervision signals: first, deterministic verification signals based on code execution or symbolic verification obtained through the Scientific Execution Suite (SES) toolkit; and second, multi-dimensional semantic evaluation signals based on Large Language Model-as-a-Judge (LLM-as-a-Judge), specifically covering three dimensions: mathematical correctness, physical consistency, and instruction following. Through this multi-source feedback mechanism, the verification-aware reward model can provide fine-grained and objective quality assessments for the reasoning steps generated by the policy model during subsequent reinforcement learning training.

[0094] (h3) Policy initialization based on supervised fine-tuning; Before reinforcement learning is introduced, this embodiment utilizes a rigorously validated quantum science dataset constructed in the first stage for partitioning. Supervised fine-tuning (SFT) of the pre-trained general-purpose large language model is then performed based on the first subset obtained from this partitioning. In this stage, the objective is optimized through maximum likelihood estimation (MLE), enabling the large language model to learn the knowledge distribution and reasoning patterns of a specific domain. Specifically, the large language model will learn how to adaptively select concise answers or detailed thought processes (CoT) derivations based on task complexity, thereby gaining an initial grasp of quantum mechanics terminology, axiomatic constraints, and problem-solving paradigms. This process effectively reduces meaningless exploration during the cold start phase, providing a high-quality initial policy model for subsequent reinforcement learning.

[0095] (h4) Reinforcement learning training based on a validation-aware reward model; In the policy model optimization phase of reinforcement learning, this embodiment utilizes the aforementioned verification-aware reward model to train the large language model after supervised fine-tuning and initialization. To address the reward sparsity problem caused by the varying verification difficulty of different types of scientific problems (such as numerical computation and concept interpretation), this embodiment introduces a verification-aware reward model. In complex scientific reasoning tasks, relying solely on deterministic feedback from external tools such as the Science Execution Suite (SES) often requires the policy model to output a strict formal grammar (such as symbolic derivation or quantum circuit code). For lengthy intermediate concept reasoning, non-normalized expressions, or steps that cannot be directly compiled, the tool typically fails and returns zero reward, resulting in a lack of effective gradient guidance in the reinforcement learning (RL) phase and the "reward sparsity" problem.

[0096] The mechanism proposed in this embodiment dynamically adjusts the fusion weights of deterministic and semantic signals based on the deterministic verification signals output by the scientific execution suite, thereby providing dense reward coverage for the entire inference chain: for strongly logical and executable inference steps, feedback from the scientific execution suite is prioritized to ensure physical constraints and accuracy; for nodes where informal logical reasoning or tool execution fails, the weights of semantic evaluation are adaptively enhanced, utilizing the inherent physical intuition and logical judgment of the verification-aware reward model to fill the gaps in the supervision signals. This strategy effectively stabilizes the training process, avoids optimization collapse caused by the lack of local rewards in the policy model, and significantly improves the inference accuracy and robustness of the policy model in complex scientific fields.

[0097] In summary, the technical approach of this embodiment, through a closed-loop process of "data → fine-tuning → reward → reinforcement learning," enables the construction of a quantum knowledge language model to be systematic, scientific, and scalable.

[0098] (h5) Presentation of experimental results; To verify the effectiveness and advancement of the method in this embodiment, two mainstream large-scale models, Qwen3-8B and Meta-Llama-3-8B-Instruct, were selected as the base models. Their performance was compared and evaluated under the base model state, the supervised fine-tuning of the large language model based on the training data constructed in this embodiment (i.e., the large language model after SFT), and the further introduction of the reinforcement learning method (RLVR) proposed in this embodiment. The experimental results are shown in Table 2. It can be observed that the model after supervised fine-tuning (SFT) based on the training data of this embodiment has achieved significantly better results than the base model in various quantum task metrics, fully demonstrating that the data construction method proposed in this embodiment has strong knowledge coverage and training effectiveness. Furthermore, after introducing the reinforcement learning (RLVR) method of this embodiment, the model performance has achieved continuous and stable improvement on both base models, especially in computational and derivation tasks.

[0099] Table 2 Performance of each large language model

[0100] In Table 2, Qwen3-8B is the baseline model of Tongyi Qianwen 3-8B, Qwen3-8B_SFT is the model after Qwen3-8B is fine-tuned with supervision using the preset first subset, Qwen3-8B_RLVR is the model after Qwen3-8B_SFT is combined with the second subset and the validation-aware reward model for reinforcement learning, Meta-Llama-3-8B-Instruct is the baseline fine-tuned version of the Meta-Llama-3-8B series models, Meta-Llama-3-8B-Instruct_SFT is the version of Meta-Llama-3-8B-Instruct after fine-tuning with supervision using the preset first subset, and Meta-Llama-3-8B-Instruct_RLVR is the version of the Meta-Llama-3-8B-Instruct_SFT model after reinforcement learning based on the validation-aware reward model. Among them, Qwen3-8B_SFT and Qwen3-8B_RLVR are variants of the Qwen3-8B series models, while Meta-Llama-3-8B-Instruct, Meta-Llama-3-8B-Instruct_SFT, and Meta-Llama-3-8B-Instruct_RLVR are variants of the Meta-Llama-3-8B series models.

[0101] As can be observed from Table 2, the large language model, after supervised fine-tuning based on the training data of this embodiment, has achieved significantly better performance than the base model on various quantum task metrics. Taking computation problems as an example, the accuracy of Qwen3-8B and Meta-Llama-3-8B-Instruct improved by approximately 10 percentage points (from 0.512 for Qwen3-8B to 0.621 for Qwen3-8B_SFT) and 14 percentage points (from 0.374 for Meta-Llama-3-8B-Instruct to 0.517 for Meta-Llama-3-8B-Instruct_SFT), respectively, fully demonstrating that the data construction method proposed in this embodiment has strong knowledge coverage and training effectiveness. Furthermore, by introducing the reinforcement learning method of this embodiment, the model performance has achieved continuous and stable improvements on both base models. It performs particularly well in calculation and derivation tasks, with the accuracy of Qwen3-8B in calculation questions further improving to 0.680 (Qwen3-8B_RLVR) and the accuracy of short answer questions reaching 0.897 (Qwen3-8B_RLVR); at the same time, the Meta-Llama-3-8B-Instruct model also shows the same growth trend in all indicators.

[0102] Meanwhile, the Qwen3-8B (8 billion parameters) after reinforcement learning outperformed the powerful ChatGPT-5 model (0.642) in complex calculation problems (0.680), fully verifying the superiority of this mechanism in improving the model's reasoning ability in complex scientific fields. This means that this embodiment can achieve higher-precision scientific intelligence with lower computing power costs.

[0103] The experimental results above show that the overall technical solution of "structured training data + verification of the perceptual reward model + reinforcement learning optimization" in this embodiment can play a stable role under different model architectures, without relying on specific models or human preference signals, and has good versatility and reusability.

[0104] Compared with the prior art, this embodiment has the following beneficial effects: (i1) Significantly improved the quality and accuracy of quantum science datasets; Existing datasets often exhibit polarization (either overly simplistic or large-scale synthetic data lacking validation), leading to poor training performance for large language models. This embodiment effectively addresses the problem of seemingly reasonable but actually erroneous training data by employing a task-adaptive data construction strategy and dual hybrid validation (combining the Scientific Execution Suite (SES) and semantic evaluation).

[0105] Specific effects: It can automatically eliminate erroneous derivations that violate physical axioms or mathematical logic, ensuring that the quantum science dataset as training data meets high standards of scientific rigor, thereby improving the model's reasoning ability from the source.

[0106] (i2) Solves the problems of reward deception and logic failure in reinforcement learning in the scientific field; Traditional RLHF (Reinforcement Learning Based on Human Feedback) can easily lead to models generating content that caters to human preferences but violates physical constraints in order to obtain high scores. The Verification-Aware Reward Model (VRM) proposed in this embodiment introduces multi-dimensional semantic evaluation signals (mathematical correctness, physical consistency, and instruction compliance) and deterministic verification signals from the Scientific Execution Suite (SES).

[0107] Specific effects: This embodiment enables large language models to learn not only "how to speak like an expert," but also "how to reason in accordance with physical axioms." It can effectively distinguish between simple computational errors and serious logical violations, forcing large language models to strictly adhere to the physical constraints and axiomatic systems of the scientific domain during the generation process.

[0108] (i3) It overcomes the problem of training instability caused by sparse reward signals in complex reasoning tasks; In rigorous scientific reasoning tasks, the number of steps that can be fully verified through code is small (sparse reward), making it difficult for large language models to converge; however, relying solely on large language models for scoring is not accurate enough. This embodiment adopts a verification-aware reward model, which dynamically adjusts the fusion weights of deterministic and semantic signals based on the "verifiability" of the current reasoning step.

[0109] Specific effects: While maintaining rigor, it compensates for the sparsity of deterministic feedback, achieving a dynamic balance between "general preference optimization" and "rigorous scientific supervision." This significantly improves the stability of the reinforcement learning process and prevents large language models from experiencing capability collapse during training.

[0110] (i4) has extremely high parameter efficiency and computational cost-effectiveness; Typically, large language models with hundreds of billions of parameters are needed to perform well in complex scientific reasoning. The method in this embodiment significantly reduces the dependence of large language models on parameter size by introducing a rule-based, verifiable training process.

[0111] Specific results: Table 2 shows that the smaller-scale model trained using the method of this embodiment (e.g., with 8B parameters) can outperform traditional supervised fine-tuning (SFT) models on scientific reasoning tasks, and even surpasses existing proprietary large language models (e.g., ChatGPT-5) in some metrics. This means that this embodiment can achieve higher-precision scientific intelligence at a lower computational cost.

[0112] (i5) Possesses broad domain adaptability and interdisciplinary transferability; The core architecture proposed in this embodiment is highly modular and not limited to the single field of quantum mechanics. By replacing the specific verification rules in the Science Execution Suite (SES), the method of this embodiment can be adapted to other highly constrained fields.

[0113] Specific results: This demonstrates the universal effectiveness of the method across different disciplines, and can significantly reduce the threshold and data cost of developing high-precision reasoning large language models in other vertical fields such as chemistry and materials science.

[0114] Example 2 This embodiment provides a specific application scenario for a large language model training method based on hybrid verification. To illustrate in detail the technical implementation of the training mechanism proposed in this embodiment when handling tasks with stringent physical boundaries and strong logical constraints, this embodiment uses the typical quantum mechanics reasoning task of "solving quantum states and calculating wave function evolution in a one-dimensional infinite potential well" as an example to deeply explain the data flow, weight allocation, and policy network iteration process of the model training method. This embodiment aims to solve the technical challenges of large language models easily deviating from the physical axiomatic system when generating complex physical derivations, and the optimization collapse caused by sparse feedback in long-range reasoning chains.

[0115] In the construction and partitioning phase of the quantum science dataset, the system automatically generates a set of quantum mechanical problems containing different potential field parameters, initial states, and boundary conditions, based on the target physics task and employing a "seed-evolution" mechanism. It also generates candidate wavefunction solution sequences based on instructions from a large language model. The system performs dual hybrid verification. The Scientific Execution Suite (SES) not only invokes symbolic computation on the Schrödinger equation... The differential solution steps undergo mathematical form verification, and physical constraint verification is also enforced, such as verifying whether the generated wave function strictly satisfies the global normalization condition. (That is, the total probability of a particle appearing in the entire space must be 1) and whether continuity and convergence are satisfied at the potential field boundary. The large language model simultaneously performs semantic evaluation on the consistency of the use of normative and physical concept interpretations of Dirac notations in the derivation text. After this round of hybrid verification and human-machine collaborative auditing loop, the system eliminates invalid data that are mathematically valid but violate basic physical conservation laws, and samples the quantum science dataset in layers and by difficulty to obtain the first subset and the second subset.

[0116] in, Represents spatial coordinates, Indicates the system's location The probability amplitude at that location, For Hamiltonian operators, this represents the total energy operator of the system; This represents the energy eigenvalues ​​of the system.

[0117] In the knowledge injection and initialization phase of the model, this embodiment utilizes a first subset to perform supervised fine-tuning (SFT) on the pre-trained large language model. By optimizing the objective through maximum likelihood estimation, the large language model gradually internalizes proprietary expressions and problem-solving paradigms in areas such as Hamiltonian construction and operator commutation relation derivation when facing specific contexts in the quantum domain. This process significantly reduces the exploration entropy of the large language model when facing complex physical problems, endowing it with a solid quantum knowledge foundation, thereby providing an initial policy model with a high-quality initial policy distribution for the subsequent reinforcement learning phase.

[0118] In the reinforcement learning optimization phase based on reward signals, the system uses a second subset and a validation-aware reward model to drive the initialized policy model to iterate parameters. Given the numerous and complex steps involved in quantum state evolution derivation, relying solely on terminal result comparison can lead to sparse reward signals or vanishing gradients. Therefore, the validation-aware reward model used in this embodiment calculates rewards for each multi-step derivation process. The policy model updates network weights based on the reward signals to maximize the expected value of generating a reasoning sequence that conforms to physical laws.

[0119] To prevent the policy model from overfitting the reward function during optimization and generating invalid outputs (such as generating physically meaningless forced formulas), this embodiment introduces a KL divergence deviation penalty term into the optimization objective function of reinforcement learning. This penalty term, as a hard safety constraint mechanism, strictly limits the divergence distance between the currently updated policy distribution and the policy distribution of the model after SFT (i.e., the baseline policy distribution). This mechanism not only effectively curbs policy collapse but also ensures that while the policy model discovers high-yield problem-solving paths, it remains anchored within a space of fluent natural language expression and strict adherence to the axioms of quantum physics, significantly improving the robustness and scientific credibility of the generated content.

[0120] Based on the above description of the embodiments, those skilled in the art will understand that the large language model training method, system, and medium based on hybrid verification described in this embodiment can be implemented in pure software or deployed and run on general-purpose or dedicated computing hardware platforms. Based on this essence, the technical solution of this embodiment can be specifically implemented in the form of a software product containing program instructions. This software product can be stored on various non-volatile storage media or directly deployed as a local or cloud service. The program instructions are used to cause computer devices with processing capabilities—including but not limited to personal computers, server clusters, mobile terminals, or other network devices—to execute the steps described in this embodiment.

[0121] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.< / think> < / think>

Claims

1. A method for training large language models based on hybrid validation, characterized in that, include: Construct a quantum science dataset that includes quantum knowledge questions and answer labels and divide it into a first subset and a second subset; The large language model is fine-tuned under supervision using the first subset, thereby obtaining the initial policy model after quantum knowledge injection; A large language model with scientific reasoning capabilities is obtained by optimizing the policy model through reinforcement learning based on the verification-aware reward model and the second subset. The verification-aware reward model evaluates multiple candidate answers generated by the policy model based on quantum knowledge questions in the second subset and outputs multiple reward values ​​that integrate deterministic verification signals and multidimensional semantic evaluation signals from the scientific execution suite. An optimization objective is constructed based on these multiple reward values, and the parameters of the policy model are updated by maximizing the expected return to increase the probability of the policy model generating answers with high reward values. The scientific execution suite is a symbolic and numerical solver for performing mathematical operations and deterministic verification of physical constraints.

2. The method according to claim 1, characterized in that, The specific process of constructing the quantum science dataset is as follows: Expanding the topic breadth based on the seed evolution paradigm to obtain initial data; The initial data is processed using a task-adaptive strategy: for retrieval-intensive tasks, concise answer tags are generated; for complex reasoning tasks, answer tags and thought chains containing detailed derivation steps are generated. Perform dual hybrid validation on the processed data, specifically as follows: The first layer of automated hybrid validation: uses a scientific execution suite to perform deterministic validation of the physical consistency and mathematical correctness of the processed data, and uses an independent large language model to perform semantic evaluation of logic and format. The second layer of human-machine collaborative auditing: For data that has passed the first layer of verification, a tiered sampling manual review is performed based on the difficulty label; Data filtered through a dual hybrid verification mechanism will be used as a quantum science dataset.

3. The method according to claim 2, characterized in that, The second layer of human-machine collaborative auditing includes a quality feedback closed-loop mechanism, specifically: Set a batch rejection threshold; The verification error rate of the current batch of data stratified sampling is calculated. If the verification error rate exceeds the batch rejection threshold, the current batch of data is determined to be invalid. The error patterns of the failed data are analyzed, and the generation instructions in the seed evolution paradigm are corrected accordingly, triggering the regeneration process of the initial data.

4. The method according to claim 1, characterized in that, The construction process of the verification perception reward model is as follows: Configure the scientific execution suite to perform deterministic validation on verifiable dimensions among multiple candidate answers generated by the strategy model and output validation indicators; A dual-head parallel prediction network is constructed based on a pre-trained Transformer encoder, including a shared encoder backbone network, a multi-dimensional scoring head, and a dynamic weight allocation head; The shared encoder backbone network generates contextual representations based on the input question and the corresponding candidate answers; The multidimensional scoring head maps the contextual representation to a semantic evaluation score, which is used to evaluate the semantic quality of the generated content. The dynamic weight allocation header concatenates the context representation with the verification indicator and outputs the dynamic weights for each evaluation dimension. Dynamic reward calibration mechanism: For each evaluation dimension, the fusion score is calculated using the semantic evaluation score and the confidence adjustment coefficient; The final reward value is calculated by weighting and aggregating the dynamic weights and fusion scores.

5. The method according to claim 4, characterized in that, The fusion score The calculation formula is as follows: ； in, for The confidence level adjustment coefficient, For the first Semantic evaluation scores in each dimension For the first Verification indicators for each dimension; The confidence adjustment coefficient satisfy: when hour, Verify that the perceived reward model fully adopts deterministic rewards; when hour, ; when hour, The default value is close to 0.

6. The method according to claim 4, characterized in that, The verification-aware reward model is initialized through Oracle-guided distillation training, specifically as follows: Using a heterogeneous large language model ensemble as judges, soft target scores and ideal sample importance weights are generated for sample data containing the results of scientific execution suites. Using the soft target score and the importance weight of the ideal sample as supervision signals, the verification perception reward model is trained through a multi-task loss function to fit the judge's evaluation distribution and align it with their discrimination preferences.

7. The method according to claim 1, characterized in that, The division into a first subset and a second subset is as follows: The quantum science dataset is directly divided into a first subset and a second subset; or, Construct an auxiliary dataset containing general domain instruction data; The first subset is constructed by proportionally mixing a predetermined number of data points from the quantum science dataset with auxiliary datasets. The second subset is constructed by proportionally mixing a predetermined number of data from the remaining quantum science dataset, the auxiliary dataset, and a predetermined number of high-difficulty long-chain inference samples from the first subset.

8. The method according to claim 1, characterized in that, The reinforcement learning optimization of the policy model based on the verification-aware reward model and the second subset specifically includes: The advantage function is estimated based on the reward value output by the verification-aware reward model; wherein, the calculation of the reward value assigns a higher weighting coefficient to the deterministic verification signal from the scientific execution suite than to the multidimensional semantic evaluation signal; Construct a proximal policy optimization objective function that includes a truncation function, wherein the objective function is optimized with the advantage function as the optimization guide; Finally, the parameters of the policy model are updated by maximizing the objective function.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of programs for being invoked by a processor and executing the method as described in any one of claims 1-8.

Images