Alignment method and related device during weak-to-strong reasoning of large language model

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses an alignment method and a related device for big language model weak-to-strong reasoning. The large language model weak-to-strong reasoning alignment method comprises the following steps of obtaining cue words; based on the obtained cue word, generating a final output result by utilizing a selected first large language model; furthermore, in the process of generating a final output result by using the selected first large language model, the generation process of the first large language model is modeled as a Markov decision process, and decision search is performed by using a Monte Carlo tree search algorithm. According to the technical scheme disclosed by the invention, by combining two theories of weak-to-strong generalization and Monte Carlo tree search, extremely low deployment cost and efficient dynamic refined control are realized; in addition, the technical scheme of the invention also has high general expandability while realizing a more accurate alignment effect.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to a large language model weak-to-strong reasoning alignment method and related device. BACKGROUND

[0002] Large language models (LLM) have shown the ability to approach or surpass humans in natural language understanding, text generation, and complex reasoning tasks. However, the output of the model often does not align with human preferences, and may contain biased, harmful, or false content. This can lead to discriminatory decisions, serious misinformation dissemination, and even loss of life and property in critical scenarios such as medical diagnosis, legal consultation, recruitment screening, or financial investment. For example, a large language model trained on biased data may discriminate against a particular gender in the recruitment scenario, leading to unfair screening results; in legal consultation, it may fabricate non-existent legal provisions or cases, misleading users and causing serious legal risks; in the medical field, incorrect diagnosis recommendations or treatment plans may directly endanger patients' lives.

[0003] To solve the problem of inconsistency between the output of large language models and human preferences, alignment measures need to be taken. Existing alignment measures can be divided into two categories: training-time alignment and inference-time alignment.

[0004] Training-time alignment aims to modify model parameters so that they can learn human preferences during the training phase. Representative existing methods include reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO). Although these methods can improve the alignment effect of the model to some extent, they all have common drawbacks. Specifically, first, these existing methods highly depend on large-scale, high-quality human-labeled data, resulting in high cost, time-consuming, labor-intensive, and difficult-to-scale training processes. Second, they are limited to providing sequence-level, post-hoc feedback during the training phase, and cannot control the model output in real time and fine-grained during the inference phase. More importantly, once the model is trained, it becomes extremely difficult to modify it, and the operability is very weak. When the model needs to adapt to new application scenarios or correct specific problems, fine-tuning the model not only risks "catastrophic forgetting" and potential performance degradation, but also brings great uncertainty due to the need for large-scale labeling, training, and evaluation.

[0005] To address the limitations of training-time alignment, inference-time alignment methods have gained attention. The core idea of ​​these methods is to introduce preference signals during the generation process to guide the model output without modifying the model parameters. However, existing inference-time alignment methods still face many challenges. For example, while Best-of-N (BOC) alignment is simple and effective, its efficiency is limited by the number of candidates and it cannot be finely controlled during generation. Other methods, when dealing with high-dimensional generation spaces, often suffer from low exploration efficiency and insufficient control granularity, making it difficult to systematically find the optimal alignment path. Summary of the Invention

[0006] The purpose of this invention is to provide an alignment method and related apparatus for weak-to-strong inference of a large language model, in order to solve one or more of the aforementioned technical problems. Specifically, the technical solution disclosed in this invention is an alignment method for weak-to-strong inference of a large language model based on Monte Carlo Tree Search. Without modifying the parameters of the strong model (i.e., the first large language model), it achieves extremely low deployment costs and efficient dynamic fine-grained control by combining two major theories: weak-to-strong (W2S) generalization and Monte Carlo Tree Search (MCTS). Furthermore, while achieving more accurate alignment results, it also possesses high versatility and scalability.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, this invention discloses an alignment method for weak-to-strong inference in a large language model, comprising the following steps:

[0009] Get the prompt words;

[0010] Based on the obtained prompt words, the final output result is generated using the selected first large language model; wherein, in the process of generating the final output result using the selected first large language model, the generation process of the first large language model is modeled as a Markov decision process (MDP), and the Monte Carlo Tree Search (MCTS) algorithm is used for decision search.

[0011] In the process of decision search using the MCTS algorithm: through repeated iterations of selection, expansion, evaluation and backpropagation, a search tree containing multiple generated paths is constructed; based on the constructed search tree, the global alignment score is calculated using the sequence-level alignment reward function according to all explored nodes, and the candidate sequence with the highest global alignment score is determined as the final alignment output.

[0012] In the process of repeated iterations of selection, expansion, evaluation, and backpropagation: in the selection stage, information entropy is introduced into the exploration reward; in the evaluation stage, a reference large language model and an alignment large language model are introduced, and the alignment value of any prefix sequence is quantified by an intermediate value function that reflects the strength of the alignment large language model's preference for the generated state.

[0013] A further improvement to the technical solution of the present invention lies in the following specific steps for obtaining the reference large language model and the aligned large language model:

[0014] The selected second-largest language model is trained under supervised fine-tuning using a dataset that conforms to human preferences to obtain the reference large language model.

[0015] Based on the dataset that conforms to human preferences, the direct preference optimization method is used to introduce preference information into the reference large language model to obtain the aligned large language model;

[0016] The second large language model is an open-source large language model with fewer than or equal to a preset threshold of parameters.

[0017] A further improvement to the technical solution of this invention lies in modeling the generation process of the first large language model as a Markov Decision Process (MDP), and utilizing the Monte Carlo Tree Search (MCTS) algorithm for decision search in the specific steps.

[0018] In the tree structure of MCTS, each node represents a generation state, corresponding to a generated prefix sequence; the action of moving from the current generation state to the next generation state corresponds to selecting the next word token or token sequence, and the action refers to a new token sequence of a certain length generated by the first large language model.

[0019] The search tree of MCTS is rooted at the prompt word, and each path from the root node to a leaf node represents a possible generating sequence.

[0020] A further improvement to the technical solution of this invention lies in that, during the selection phase, information entropy is introduced into the exploration reward step, employing an entropy-aware PUCT selection strategy EA-PUCT, expressed as:

[0021]

[0022] In the formula, EA-PUCT(s) represents the entropy-aware prior UCT; R(s) represents the immediate reward; c and w represent coefficients; and P(s) represents the first large language model from the parent node s. p The prior probability of taking an action to reach the state s of a child node; N(s) p ) represents the parent node s pThe historical access count; N(s) represents the historical access count of the child node state s; H(s) represents the information entropy;

[0023] P(s) is defined as the geometric mean of the word-level probabilities, and its expression is:

[0024]

[0025] In the formula, L represents the length of each generated sequence; t represents the time step of the sequence generation process; p(y t |s p ,y <t ) represents a given input x and a previously generated sequence y. <t Under the condition of generating the next word y t The probability of;

[0026] H(s)=-∑ a P(s,a)·logP(s,a);

[0027] In the formula, a represents the action that the strong model, as the first language big model, can take; P(s,a) represents the probability that the strong model takes action a in state s.

[0028] A further improvement to the technical solution of this invention lies in the step of introducing a reference large language model and an alignment large language model in the evaluation stage, and quantifying the alignment value of any prefix sequence through an intermediate value function that reflects the strength of the alignment large language model's preference for the generated state.

[0029] Each newly generated node s' represents a portion of the sequence. Immediate feedback is obtained by calculating the proxy value; the expression for the intermediate value function used in the calculation is as follows:

[0030]

[0031] In the formula, This indicates that the newly generated node s' represents a partial sequence. The intermediate value function of alignment value; This represents the relationship between the current sequence y' and the newly generated sequence y'. 1:L splicing; This indicates alignment with the large language model. This indicates a reference to a large language model;

[0032] Each node uses the intermediate value function V. proxyInitialize the node as its immediate reward R(s') in the search tree; if a child node meets the termination condition, its immediate reward R(s') will be set to -∞ to prevent it from being selected again in subsequent MCTS iterations. These termination nodes will be retained as candidates for subsequent global evaluation phases.

[0033] A further improvement to the technical solution of this invention lies in the following step: based on the constructed search tree, calculating the global alignment score using a sequence-level alignment reward function according to all explored nodes, and determining the candidate sequence with the highest global alignment score as the final alignment output:

[0034] All complete generated sequences are collected from the root node to the terminal node using a recursive depth-first search. From these collected sequences, the candidate sequence with the second-to-last node having the highest MCTS evaluation reward is identified. For each candidate sequence, the final global alignment score is calculated based on the sequence-level alignment reward function r. proxy The expression for (x,y) is:

[0035]

[0036] In the formula, x represents the input prompt word, and y represents the complete generated sequence;

[0037] Based on the global alignment score calculation, each candidate sequence is re-ranked, and the candidate sequence with the highest global alignment score is determined as the final alignment output.

[0038] A further improvement to the technical solution of this invention lies in the fact that, during the process of repeated iterations of selection, expansion, evaluation, and backpropagation:

[0039] In the expansion phase, the pre-trained parameters of the first language model are used to generate new child nodes for the selected leaf nodes. First, based on the distribution predicted by the first language model in the current state, the Top-N words with the highest probabilities are identified as candidate words. Then, from the Top-N candidate words, K different candidate words are randomly selected based on their probabilities to initiate the formation of new blocks. Each randomly selected candidate word forms a new block of a set length, and the new block is connected to the current sequence to generate a new state.

[0040] Additionally, during the backpropagation phase, the reward R(s') obtained from the newly simulated node propagates back along the path to the root node, with the visit count of each ancestor state increasing along the way; among which, the reward R(s) of the parent state... p The result will be updated to the maximum value of all observed child node returns, following the formula:

[0041] R(sp )←max(R(s c ));

[0042] In the formula, R(s) c ) represents the real-time report of each child node.

[0043] A second aspect of the present invention discloses an alignment system for weak-to-strong inference in a large language model, comprising:

[0044] The prompt word acquisition module is used to acquire prompt words;

[0045] The result generation module is used to generate the final output result based on the acquired prompt words using the selected first language model; wherein, in the process of generating the final output result using the selected first language model, the generation process of the first language model is modeled as a Markov decision process (MDP), and the Monte Carlo Tree Search (MCTS) algorithm is used for decision search.

[0046] In the process of decision search using the MCTS algorithm: through repeated iterations of selection, expansion, evaluation and backpropagation, a search tree containing multiple generated paths is constructed; based on the constructed search tree, the global alignment score is calculated using the sequence-level alignment reward function according to all explored nodes, and the candidate sequence with the highest global alignment score is determined as the final alignment output.

[0047] In the process of repeated iterations of selection, expansion, evaluation, and backpropagation: in the selection stage, information entropy is introduced into the exploration reward; in the evaluation stage, a reference large language model and an alignment large language model are introduced, and the alignment value of any prefix sequence is quantified by an intermediate value function that reflects the strength of the alignment large language model's preference for the generated state.

[0048] In a third aspect, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the alignment method for weak-to-strong inference of a large language model as described in any one of the first aspects of the present invention.

[0049] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the weak-to-strong inference alignment method for a large language model as described in any one of the first aspects of the present invention.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The technical solution disclosed in this invention is a weak-to-strong inference alignment method for large language models based on Monte Carlo tree search. It aims to achieve accurate and controllable value alignment of the strong model's output without modifying the strong model's parameters. This invention combines weak-to-strong generalization and Monte Carlo tree search theories to achieve extremely low deployment costs and efficient dynamic fine-grained control. Furthermore, while achieving more accurate alignment results, it possesses high versatility and scalability. The method of this invention does not rely on expensive and unstable training processes or large-scale labeled data, allowing for plug-and-play operation. Additionally, by using the weak model as a proxy to provide alignment signals, MCTS can finely adjust the model output in real-time, step-by-step during inference, thereby achieving more accurate alignment results. Moreover, the entropy-aware PUCT selection strategy (EA-PUCT) of this invention can adaptively balance exploration and utilization, significantly improving the efficiency of MCTS in finding the optimal alignment path in massive generation spaces. Furthermore, the implementation and deployment of the technical solution framework of this invention are independent of specific LLM architectures, thus possessing high versatility and scalability, facilitating rapid deployment to various generation tasks.

[0052] Further, this invention, based on weak-to-strong generalization theory, effectively addresses the core problems of existing technologies, such as high training costs, dependence on large-scale data, and difficulty in modifying trained models. This invention demonstrates that even small-scale, low-cost weak models can effectively learn and internalize human preferences through alignment training. More importantly, this invention uses this alignment signal as a quantifiable reward function, dynamically transferring it to the unaligned strong model during the inference phase. This fundamentally solves the drawbacks of methods like RLHF and DPO, which heavily rely on expensive and unstable training processes. This mechanism enables low-cost transfer of alignment capabilities to strong models, making this method plug-and-play, with extremely low deployment costs, and possessing general scalability.

[0053] To further explain, this invention elegantly models the text generation process of a large language model as a Markov Decision Process (MDP) and creatively introduces MCTS as its online decision search algorithm, thereby completely overcoming the shortcomings of existing alignment methods during inference, such as lack of fine-grained control, low exploration efficiency, and lack of systematicity. Unlike traditional static, locally optimal decoding strategies such as greedy decoding or bundle search, MCTS, by constructing a dynamic search tree, achieves an optimal balance between "exploration" (encouraging visits to branches with high uncertainty to discover new possibilities) and "exploitation" (pursuing high-reward paths for rapid convergence). In this dynamic search process, the intermediate value function provided in real time by the weak model (i.e., the second largest language model) serves as the immediate reward for MCTS nodes, continuously and finely guiding the search direction. This ensures that the algorithm does not simply select the most likely generated content, but systematically searches and discovers content that both conforms to the powerful generation capabilities of the strong model and highly satisfies alignment preferences. Finally, the best output is selected by sorting all candidate complete sequences by sequence-level alignment reward. This hierarchical decision-making process combines fine alignment in the generation process with the global quality of the final output, ultimately achieving the unique advantage of fine control in the generation process. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0055] Figure 1 This is a flowchart illustrating an alignment method for weak-to-strong inference in a large language model, as described in an embodiment of the present invention.

[0056] Figure 2 This is a schematic diagram of the online generation and alignment process of MCTS in an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the global evaluation and final selection process of candidate results in an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of an alignment system for weak-to-strong inference in a large language model, as described in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention; obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0060] Based on the technical solutions disclosed in the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0061] Please see Figure 1 The present invention provides an alignment method for weak-to-strong inference in a large language model, comprising the following steps:

[0062] Step 1, obtain the clue words;

[0063] Step 2: Based on the prompt words obtained in Step 1, generate the final output result using the selected first large language model; wherein, in the process of generating the final output result using the selected first large language model, the generation process of the first large language model is modeled as a Markov decision process (MDP), and the Monte Carlo Tree Search (MCTS) algorithm is used for decision search.

[0064] In the process of decision search using the Monte Carlo Tree Search (MCTS) algorithm: through multiple iterative selection, expansion, evaluation and backpropagation stages, a search tree containing multiple generated paths is constructed; based on the constructed search tree, the global alignment score is calculated using the sequence-level alignment reward function according to all explored nodes, and the candidate sequence with the highest global alignment score is determined as the final alignment output.

[0065] In the process of repeated iterations of selection, expansion, evaluation, and backpropagation: in the selection stage, information entropy is introduced into the exploration reward; in the evaluation stage, a reference large language model and an alignment large language model are introduced, and the alignment value of any prefix sequence is quantified by an intermediate value function that reflects the strength of the alignment large language model's preference for the generated state.

[0066] Interpretationally, incorporating Monte Carlo Tree Search (MCTS) into LLM inference optimization is considered a promising solution, but related research is still in its early stages, mainly focusing on mathematical reasoning or planning tasks, and has not yet formed a systematic alignment exploration. The shortcomings and deficiencies of existing technologies mainly include: high training costs and data dependence, lack of fine-grained control during the inference phase, and limitations in the application of MCTS in inference-time alignment; specifically, both RLHF and DPO rely on large amounts of human-labeled data or high-quality reward models to guide training, resulting in huge training resource requirements and difficulty in quickly adapting to new tasks or deploying in resource-constrained environments; existing training-time methods can only obtain sequence-level, "post-hoc" feedback during the training phase, and cannot achieve real-time, gradual preference guidance during the generation process, making it difficult to meet the needs for interpretability and dynamic control; although MCTS has been successfully applied to offline tasks such as mathematical reasoning and planning due to its ability to balance exploration and utilization in a large-scale search space, it has rarely been applied in the field of alignment, with a few methods only using it for offline data generation, and there is no systematic framework for using it to guide strong models in real-time, dynamically, and dynamically during the inference phase of alignment exploration.

[0067] In a further specific exemplary technical solution of the present invention, the specific steps for obtaining the reference large language model and the aligned large language model are as follows:

[0068] Using a selected dataset that conforms to human preferences, supervised fine-tuning training is performed on the selected second-largest language model to obtain a reference large language model (i.e., a reference version of the subsequent aligned weak model). Further, based on the selected dataset that conforms to human preferences, the direct preference optimization method is used to introduce preference information into the reference large language model to obtain the aligned large language model (i.e., the subsequent aligned weak model). For specific examples, the selected second-largest language model is an open-source large language model with fewer than or equal to a preset threshold, which can be set to 200M.

[0069] As a specific embodiment of the technical solution of this invention, the generation process of the first large language model is modeled as a Markov decision process (MDP), and the Monte Carlo Tree Search (MCTS) algorithm is used for decision search. In the specific steps of the MCTS tree structure, each node represents a generation state s, corresponding to a generated prefix sequence y'. The action a from generation state s to the next generation state s' corresponds to selecting the next token (lexical) or token sequence. Action a refers to a new token sequence y of length L generated by the first large language model (i.e., the strong model). 1:L The search tree of MCTS is rooted at the prompt word x. Each path from the root node to a leaf node represents a possible generation sequence, and the final complete generated response is represented as y.

[0070] The technical solution of this invention, in an explanatory manner, involves the user inputting the prompt word x during the inference phase; the first major language model is a strong model π used to generate high-quality text, but which is not aligned. strong The alignment large language model is a weak alignment model used to evaluate the quality of sequence alignment. The reference large language model is a reference version of the alignment weak model. The core of the technical solution in this invention lies in integrating the alignment signal of the weak model with the generative capability of the strong model, modeling the generation process of the large language model as a Markov decision process (MDP), and using the Monte Carlo Tree Search (MCTS) algorithm for decision search. Based on this, this invention defines two key theoretical functions (i.e., the intermediate value function and the sequence-level alignment reward function) to quantify alignment quality. These functions transform the preferences of the weak model into computable reward signals to guide the MCTS search and the selection of the final result. This design also meets the reward sparsity requirement of "post-hoc reward" in traditional RLHF.

[0071] Please see Figure 2 In a specific embodiment of the present invention, Figure 2 This illustrates the process of constructing a search tree containing multiple generated paths by repeatedly iterating through four stages: selection, expansion, evaluation, and backpropagation. Specifically, this is achieved by repeatedly executing the following four steps, iterating m times to construct a search tree containing multiple generated paths.

[0072] 1. Selection Phase

[0073] The selection strategy employs EA-PUCT: In the selection phase of MCTS, an entropy-aware prioritized upper confidence bound apply to the tree (EA-PUCT) rule is designed to adaptively balance exploration and exploitation. Interpretatively, while standard UCT guides exploration by balancing node visits and the average of all historical rewards, PUCT combines prior probabilities from the policy network with UCT; however, π strong The output distribution often exhibits a "peak" effect, causing MCTS to converge prematurely to a local optimum, sacrificing diversity and the opportunity to discover better alternatives.

[0074] To overcome the above problems, this invention introduces the concept of information entropy into exploration rewards and proposes EA-PUCT, which is defined as follows:

[0075]

[0076] In the formula, EA-PUCT(s) represents the entropy-aware prior UCT proposed in this invention; R(s) represents the immediate reward rather than the average of all historical rewards, since the task is set to find an optimal solution; c and w are coefficients; P(s) represents the first large language model (strong model) from the parent node s p The prior probability of taking an action to reach the state s of a child node; N(s) p ) represents the parent node s p The historical access count; N(s) represents the historical access count of the child node state s; H(s) represents the information entropy;

[0077] Furthermore, for a generated short sequence y of length L... 1:L P(s) is defined as the geometric mean of the probabilities at the word level:

[0078]

[0079] In the formula, L represents the length of each generated sequence; t represents the time step of the sequence generation process; p(y t |s p ,y <t ) represents a given input x and a previously generated sequence y. <t Under the condition of generating the next word y t The probability of.

[0080] The approach described in this embodiment of the invention better penalizes low-probability words, thereby promoting robust path exploration.

[0081] Furthermore, the information entropy H(s) measures the uncertainty of the strong model's output distribution, and is defined as:

[0082] H(s)=-∑ a P(s,a)·logP(s,a);

[0083] In the formula, a represents the action that the strong model can take (e.g., the decision to generate a specific word); P(s,a) represents the probability that the strong model will take action a in state s.

[0084] In the rules proposed in this embodiment of the invention, by guiding the search with the perceived uncertainty of (1+w·H(s)) as the exploration reward, EA-PUCT endows the search with awareness of information gain: (1) When the entropy value is large, it indicates that the distribution of the next word in the model has high uncertainty; at this time, the (1+w·H(s)) term will increase the exploration reward, prompting MCTS to explore diverse trajectories more deeply. (2) When the entropy value is low, it indicates that the model is confident in the best action; at this time, the exploration reward is suppressed, and the focus of the search shifts to utilizing the identified high-reward paths. The mechanism proposed in this embodiment of the invention effectively alleviates the premature convergence problem of MCTS in LLM alignment, significantly enhances the ability to explore diverse candidate answers in complex generative spaces, and prevents excessive randomness caused by naive entropy maximization.

[0085] 2. Expansion Phase

[0086] The node expansion phase directly utilizes the pre-trained parameters of the strong model to generate new child nodes for the selected leaf nodes. To reflect the inherent bias of the strong model, it is first based on the distribution π predicted by the model in the current state s. strong The top-N most probable lexical units are identified to ensure generation capability. To diversify the exploration, K different lexical units are randomly selected from these top-N candidate lexical units, using their probabilities as the selection probabilities, to initiate the formation of new blocks. Each selected lexical unit then forms a new block y of length L. 1:L The block is then connected to the current sequence y' to generate a new state s'.

[0087] The above-described design means of the technical solution of the embodiments of the present invention allow for flexible setting of the step size L to adapt to different tasks:

[0088] (1) Fine-grained decision-making: When L=1, MCTS performs precise word-by-word meta-decision making to achieve high-precision control of alignment at each step.

[0089] (2) High-level branching: When L>1, MCTS expands a short sequence at once, effectively reducing the depth of the tree, thereby enabling more efficient and higher-level exploration in high-dimensional space.

[0090] 3. Evaluation Phase

[0091] Each newly generated node s' represents a portion of the sequence. That is, the current sequence y' and the newly generated y 1:L By piecing these together, it obtains immediate feedback by calculating surrogate values, a crucial evaluation step.

[0092] Specifically, using the intermediate value function V proxy :

[0093]

[0094] In the formula, the intermediate value function V proxy Used to quantize the newly generated node s' representing a partial sequence. The alignment value reflects the weak alignment model. The function effectively transforms the alignment preference in the intermediate process of a weak model into a quantifiable value signal to guide the online search of MCTS, based on the preference strength for the current generated state s'.

[0095] Each node uses V proxy Initialize the node as its immediate reward R(s') in the tree. If a child node satisfies a termination condition (e.g., reaches the maximum length or generates the terminator EOS), its reward R(s') will be explicitly set to -∞ to prevent it from being selected again in subsequent MCTS iterations. These termination nodes will be retained as candidates for the second-stage evaluation.

[0096] 4. Backpropagation phase

[0097] The reward R(s') obtained from the newly simulated node is propagated precisely back along its path to the root node, with the visit count of each ancestor state increasing along the way. Crucially, the reward R(s) of the parent state... p The result will be updated to the maximum value of all observed child node returns, following the formula:

[0098] R(s p )←max(R(s c ));

[0099] In the formula, R(s) c ) represents the real-time report of each child node.

[0100] The technical improvements in this invention enable MCTS to propagate maximum rewards rather than typical average rewards, treating LLM alignment as the optimal search for a single highest-reward sequence, rather than pursuing long-term average rewards as in adversarial games. This design ensures that MCTS can preserve high-value trajectories, effectively prune suboptimal branches, and concentrate computational resources on the most promising paths.

[0101] Please see Figure 3 After MCTS iteration, this invention obtains a V tree using intermediate step level. proxy A search tree is constructed, but the final decision still requires global evaluation. For example... Figure 3 As shown, the second phase aims to identify the most aligned complete response from all explored nodes using a robust strategy.

[0102] Specifically, firstly, all complete sequences are collected from the root node to the terminal node using a recursive depth-first search (DFS). Then, candidate sequences with the second-to-last node having the highest MCTS evaluation reward are identified from these sequences, reflecting the most promising path to alignment. For each candidate sequence, its final global alignment score is calculated according to the following formula; where the expression for the sequence-level alignment reward function is:

[0103]

[0104] In the formula, x represents the input prompt word, and y represents the complete generated response. This function is used to globally evaluate a complete generated sequence y, reflecting the strength of the weak model's preference for the complete response, and is the final alignment target of this invention.

[0105] The scores calculated in this embodiment of the invention match the complete sentence-level alignment target in the formula, and are then used to re-rank candidate sequences; the candidate sequence with the highest global alignment score is ultimately determined as the final aligned output. Additionally, if no terminating node is found during MCTS iterations (e.g., due to insufficient search budget or an extremely difficult problem), this invention also designs a backup mechanism to select the node with the highest MCTS reward to ensure stable and consistent output. In summary, the second stage integrates step-level guidance and sequence-level evaluation, thereby significantly improving the reliability, fidelity, and quality of the results.

[0106] In summary, the technical solution disclosed in this invention achieves efficient, dynamic, and fine-grained alignment during LLM inference by combining weak-to-strong generalization with MCTS. MCTS provides a powerful search mechanism during the generation process, balancing exploration and exploitation through the EA-PUCT rule, while the weak model serves as an alignment signal, guiding the search direction of MCTS in real time. This method of the present invention is plug-and-play without expensive training costs and labeled data, effectively improving the output quality of LLM.

[0107] In summary, the core improvement measures of the technical solutions in the embodiments of the present invention include at least the following:

[0108] A weak-strong model fusion inference alignment framework: The weak and strong models are used together during the inference phase, and the generation of results is guided in real time through tree search to achieve dynamic alignment.

[0109] Entropy-aware balancing exploration strategy: An entropy-aware balancing mechanism is added to the search and selection stage to adaptively balance the exploration of new paths and the utilization of high-scoring paths, thereby improving the diversity of generation and the convergence speed.

[0110] Hierarchical decision-making process: First, a small segment search is performed locally at each step, and then all candidate results are globally sorted, taking into account both the fine alignment of the generation process and the quality of the final output.

[0111] Flexible and adjustable expansion strategy: Supports adjusting the generation length and number of candidate branches for each expansion as needed, and can be flexibly configured according to different computing power and task requirements.

[0112] Plug-and-play deployment: It works entirely during the inference phase without the need to retrain large-scale models, and can be used in conjunction with existing models to achieve efficient deployment.

[0113] In summary, the significant improvements achieved by the above-described core improvement methods in the embodiments of the present invention include at least the following:

[0114] Extremely low deployment cost: This framework is plug-and-play during the inference phase, requiring no additional training or fine-tuning of strong models, and does not rely on large-scale labeled data to achieve alignment optimization, significantly reducing manpower, computing power and time investment.

[0115] Dynamic fine-grained control: Combining Monte Carlo tree search and weak model scoring, it can intervene in real time during the generation process and finely adjust the model output in step increments to achieve more accurate alignment results.

[0116] Efficient Path Search: The Entropy-Aware PUCT Selection Strategy (EA-PUCT) adaptively balances exploration and exploitation, significantly improving the efficiency of MCTS in finding the optimal alignment path in a massive generation space.

[0117] General and Scalable: The implementation and deployment of this framework are independent of specific LLM architectures. It can be used in combination with any weak or strong model, has high compatibility and scalability, and can be easily and quickly deployed to various generation tasks.

[0118] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0119] Please see Figure 4 In this embodiment of the invention, an alignment system for weak-to-strong inference in a large language model is disclosed, comprising:

[0120] The prompt word acquisition module is used to acquire prompt words;

[0121] The result generation module is used to generate the final output result based on the acquired prompt words using the selected first language model; wherein, in the process of generating the final output result using the selected first language model, the generation process of the first language model is modeled as a Markov decision process (MDP), and the Monte Carlo Tree Search (MCTS) algorithm is used for decision search.

[0122] In the process of decision search using the MCTS algorithm: through repeated iterations of selection, expansion, evaluation and backpropagation, a search tree containing multiple generated paths is constructed; based on the constructed search tree, the global alignment score is calculated using the sequence-level alignment reward function according to all explored nodes, and the candidate sequence with the highest global alignment score is determined as the final alignment output.

[0123] In the process of repeated iterations of selection, expansion, evaluation, and backpropagation: in the selection stage, information entropy is introduced into the exploration reward; in the evaluation stage, a reference large language model and an alignment large language model are introduced, and the alignment value of any prefix sequence is quantified by an intermediate value function that reflects the strength of the alignment large language model's preference for the generated state.

[0124] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment can be used to perform alignment methods during weak-to-strong inference of large language models.

[0125] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the operating system of the terminal. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the alignment method for weak-to-strong inference of large language models in the above embodiments.

[0126] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0127] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0128] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0129] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An alignment method for weak-to-strong inference in a large language model, characterized in that, Includes the following steps: Get the prompt words; Based on the obtained prompt words, the final output result is generated using the selected first large language model; wherein, in the process of generating the final output result using the selected first large language model, the generation process of the first large language model is modeled as a Markov decision process (MDP), and the Monte Carlo Tree Search (MCTS) algorithm is used for decision search. In the process of decision search using the MCTS algorithm: through repeated iterations of selection, expansion, evaluation and backpropagation, a search tree containing multiple generated paths is constructed; based on the constructed search tree, the global alignment score is calculated using the sequence-level alignment reward function according to all explored nodes, and the candidate sequence with the highest global alignment score is determined as the final alignment output. In the process of repeated iterations of selection, expansion, evaluation, and backpropagation: in the selection stage, information entropy is introduced into the exploration reward; in the evaluation stage, a reference large language model and an alignment large language model are introduced, and the alignment value of any prefix sequence is quantified by an intermediate value function that reflects the strength of the alignment large language model's preference for the generated state.

2. The alignment method for weak-to-strong inference in a large language model according to claim 1, characterized in that, The specific steps for obtaining the reference large language model and the aligned large language model are as follows: The selected second-largest language model is trained under supervised fine-tuning using a dataset that conforms to human preferences to obtain the reference large language model. Based on the dataset that conforms to human preferences, the direct preference optimization method is used to introduce preference information into the reference large language model to obtain the aligned large language model; The second large language model is an open-source large language model with fewer than or equal to a preset threshold of parameters.

3. The alignment method for weak-to-strong inference in a large language model according to claim 1, characterized in that, The generation process of the first major language model is modeled as a Markov Decision Process (MDP), and the specific steps of using the Monte Carlo Tree Search (MCTS) algorithm for decision search are as follows: In the tree structure of MCTS, each node represents a generation state, corresponding to a generated prefix sequence; the action of moving from the current generation state to the next generation state corresponds to selecting the next word token or token sequence, and the action refers to a new token sequence of a certain length generated by the first large language model. The search tree of MCTS is rooted at the prompt word, and each path from the root node to a leaf node represents a possible generating sequence.

4. The alignment method for weak-to-strong inference in a large language model according to claim 1, characterized in that, In the selection phase, information entropy is incorporated into the exploration reward step, employing an entropy-aware PUCT selection strategy, EA-PUCT, expressed as: In the formula, EA-PUCT(s) represents the entropy-aware prior UCT; R(s) represents the immediate reward; c and w represent coefficients; and P(s) represents the first large language model from the parent node s. p The prior probability of taking an action to reach the state s of a child node; N(s) p ) represents the parent node s p The historical access count; N(s) represents the historical access count of the child node state s; H(s) represents the information entropy; P(s) is defined as the geometric mean of the word-level probabilities, and its expression is: In the formula, L represents the length of each generated sequence; t represents the time step of the sequence generation process; p(y t |s p ,y <t ) represents a given input x and a previously generated sequence y. <t Under the condition of generating the next word y t The probability of; H(s)=-∑ a P(s,a)·logP(s,a); In the formula, a represents the action that the strong model, as the first language big model, can take; P(s,a) represents the probability that the strong model takes action a in state s.

5. The alignment method for weak-to-strong inference in a large language model according to claim 1, characterized in that, In the evaluation phase, a reference large language model and an alignment large language model are introduced. The alignment value of any prefix sequence is quantified through an intermediate value function that reflects the strength of the alignment large language model's preference for the generated state. Each newly generated node s' represents a portion of the sequence. Immediate feedback is obtained by calculating the proxy value; the expression for the intermediate value function used in the calculation is as follows: In the formula, This indicates that the newly generated node s' represents a partial sequence. The intermediate value function of alignment value; This represents the relationship between the current sequence y' and the newly generated sequence y'. 1:L splicing; This indicates alignment with the large language model. This indicates a reference to a large language model; Each node uses the intermediate value function V. proxy Initialize the node as its immediate reward R(s') in the search tree; if a child node meets the termination condition, its immediate reward R(s') will be set to -∞ to prevent it from being selected again in subsequent MCTS iterations. These termination nodes will be retained as candidates for subsequent global evaluation phases.

6. The alignment method for weak-to-strong inference in a large language model according to claim 5, characterized in that, Based on the constructed search tree, the steps of calculating the global alignment score using the sequence-level alignment reward function based on all explored nodes, and determining the candidate sequence with the highest global alignment score as the final alignment output, include: All complete generated sequences are collected from the root node to the terminal node using a recursive depth-first search. From these collected sequences, the candidate sequence with the second-to-last node having the highest MCTS evaluation reward is identified. For each candidate sequence, the final global alignment score is calculated based on the sequence-level alignment reward function r. proxy The expression for (x,y) is: In the formula, x represents the input prompt word, and y represents the complete generated sequence; Based on the global alignment score calculation, each candidate sequence is re-ranked, and the candidate sequence with the highest global alignment score is determined as the final alignment output.

7. The alignment method for weak-to-strong inference in a large language model according to claim 1, characterized in that, The process involves multiple iterations of the four phases: selection, expansion, evaluation, and backpropagation. In the expansion phase, the pre-trained parameters of the first language model are used to generate new child nodes for the selected leaf nodes. First, based on the distribution predicted by the first language model in the current state, the Top-N words with the highest probabilities are identified as candidate words. Then, from the Top-N candidate words, K different candidate words are randomly selected based on their probabilities to initiate the formation of new blocks. Each randomly selected candidate word forms a new block of a set length, and the new block is connected to the current sequence to generate a new state. Additionally, during the backpropagation phase, the reward R(s') obtained from the newly simulated node propagates back along the path to the root node, with the visit count of each ancestor state increasing along the way; among which, the reward R(s) of the parent state... p The result will be updated to the maximum value of all observed child node returns, following the formula: R(s p )←max(R(s c )); In the formula, R(s) c ) represents the real-time report of each child node.

8. An alignment system for weak-to-strong inference in a large language model, characterized in that, include: The prompt word acquisition module is used to acquire prompt words; The result generation module is used to generate the final output result based on the acquired prompt words using the selected first language model; wherein, in the process of generating the final output result using the selected first language model, the generation process of the first language model is modeled as a Markov decision process (MDP), and the Monte Carlo Tree Search (MCTS) algorithm is used for decision search. In the process of decision search using the MCTS algorithm: through repeated iterations of selection, expansion, evaluation and backpropagation, a search tree containing multiple generated paths is constructed; based on the constructed search tree, the global alignment score is calculated using the sequence-level alignment reward function according to all explored nodes, and the candidate sequence with the highest global alignment score is determined as the final alignment output. In the process of repeated iterations of selection, expansion, evaluation, and backpropagation: in the selection stage, information entropy is introduced into the exploration reward; in the evaluation stage, a reference large language model and an alignment large language model are introduced, and the alignment value of any prefix sequence is quantified by an intermediate value function that reflects the strength of the alignment large language model's preference for the generated state.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the alignment method for weak-to-strong inference of a large language model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the alignment method for weak-to-strong inference of a large language model as described in any one of claims 1 to 7.

Images