Process supervision data generation method and system based on adaptive monte carlo search

Abstract

A process supervision data generation method and system based on adaptive Monte Carlo search, the method comprising generating an initial reasoning trajectory and performing feature extraction; using the standardized features to perform reasoning trajectory clustering; for each cluster, counting the number of successful reasoning trajectories and the total number of trajectories and estimating the success probability, quantifying the cluster uncertainty; performing adaptive iterative sampling, if the termination condition is not met, identifying the active cluster set, selecting the active cluster with the highest uncertainty as the additional sampling target, calculating the additional sampling number, and generating new reasoning trajectories; after sampling termination, calculating the Monte Carlo value of the node to guide the expansion direction of the search tree; for each possible successor node pair of the current node, selecting the successor node with the highest comprehensive expansion score for expansion; recording all intermediate reasoning states and corresponding Monte Carlo values during the search process to generate process supervision data. The present application can dynamically and reasonably allocate computing resources, improving generation efficiency and quality.

Description

Technical Field

[0001] This invention belongs to the field of computer and artificial intelligence technology, specifically relating to a method and system for generating process-supervised data based on adaptive Monte Carlo search. Background Technology

[0002] In recent years, large language models have achieved significant success in various natural language processing tasks. However, these models still face challenges in complex, multi-step reasoning tasks requiring precise logic and error-free deduction (e.g., mathematical problem-solving, code generation). To improve the complex reasoning capabilities of these models, an effective technical approach is process supervision. The core of this approach is to first train a process reward model that evaluates each step in the model's problem-solving process. Then, the fine-grained supervisory signals generated by this process reward model are used to guide and optimize the reasoning model (e.g., through reinforcement learning), thereby enhancing its complex reasoning capabilities. Unlike "outcome reward models," which only award a total score based on the correctness of the final answer, process reward models can evaluate each step in the model's problem-solving process, providing more fine-grained supervisory signals. Related research indicates that this step-level supervision is generally more effective in improving the complex reasoning capabilities of large language models.

[0003] Training high-quality process reward models relies on large-scale, high-quality process supervision data. Providing accurate quality labels for each step in the inference process is a time-consuming and labor-intensive process requiring extensive domain expert knowledge. Therefore, the industry has proposed using Monte Carlo algorithms to automate process label acquisition, replacing costly manual labeling. The basic idea is to view the complete problem-solving process of the model as a reasoning path, consisting of multiple interconnected reasoning steps, each of which can be considered a node on the path. To evaluate the "quality" of a certain intermediate node (i.e., a certain reasoning step), the automated process starts from this node, instructing the model to continue reasoning and generate multiple different subsequent paths; this process is called path expansion. Finally, by statistically analyzing how many of these N complete paths lead to the correct answer, the proportion is used as the quality score for the corresponding node. However, existing automated data generation processes based on Monte Carlo algorithms suffer from fundamental flaws in the following two aspects, leading to their inefficiency and lack of flexibility: (1) At the node value assessment level, it relies on a fixed sampling budget strategy. This strategy allocates a uniform and fixed number of sampling resources (i.e., the number of path extensions, for example, uniformly extending 16 paths) to each inference node. This "one-size-fits-all" approach ignores the differences in assessment difficulty among different nodes, leading to a serious waste of computing resources. For a simple, highly deterministic node, a small amount of extension is sufficient to accurately assess its value; however, for a fuzzy, highly uncertain critical node, more extensions are needed to obtain a reliable assessment. The fixed budget strategy results in resource redundancy in the former and may lead to underassessment in the latter.

[0004] (2) At the path expansion level, it neglects the adaptive balance between exploration and utilization. In the process of searching and expanding inference paths, existing methods usually adopt fixed and non-adaptive strategies, which makes it difficult for the algorithm to efficiently discover high-quality inference paths or accurately locate the key error nodes that lead to the final failure. This rigid search method has become a key bottleneck in improving the quality and efficiency of data annotation.

[0005] In summary, the core issues leading to insufficient efficiency and flexibility in existing automated process monitoring data generation technologies are the static resource allocation in the evaluation process and the rigid strategies in the path search process. Summary of the Invention

[0006] The purpose of this invention is to address the problems in the prior art by providing a process supervision data generation method and system based on adaptive Monte Carlo search, which dynamically and rationally allocates computing resources and improves the efficiency and quality of process supervision data generation.

[0007] To achieve the above objectives, the present invention provides the following technical solution: Firstly, a method for generating process-supervised data based on adaptive Monte Carlo search is provided, including: Generate the initial reasoning trajectory for a multi-step reasoning task and extract its features; The extracted features are standardized, and the standardized features are used for inference trajectory clustering. For each cluster, the number of successful inference trajectories and the total number of trajectories are counted and the success probability is estimated to quantify the clustering uncertainty. Perform adaptive iterative sampling, determine whether the termination condition is met, and if the termination condition is not met, identify the active cluster set, select the active cluster with the highest uncertainty as the additional sampling target, calculate the number of additional samples, and generate a new inference trajectory. After sampling is terminated, the Monte Carlo value of the node is calculated to guide the expansion direction of the search tree; According to the expansion direction of the search tree, for each possible successor node pair of the current node, calculate the utilization value and exploration reward, define time-adaptive weights, calculate and obtain the comprehensive expansion score, and select the successor node with the highest score for expansion; For each multi-step reasoning task, record all intermediate reasoning states and their corresponding Monte Carlo values ​​during the search process to generate process supervision data.

[0008] As a preferred approach, the multi-step reasoning task is solving mathematical problems; a set of mathematical problems to be processed is selected from the MATH500 dataset and the GSM8K dataset. Each question Includes a problem description and standard answer; Qwen2.5-Math-7B-Instruct is selected as the basic large language model. Set the initial sampling quantity Maximum sampling budget and the number of clusters .

[0009] As a preferred embodiment, in the step of generating the initial reasoning trajectory for the multi-step reasoning task and performing feature extraction, for the problem... a certain intermediate inference node Using a non-greedy decoding strategy from the basic large language model generate A complete reasoning trajectory; The intermediate inference node This represents the reasoning sequence from the beginning to step t; The reasoning trajectory refers to the complete process from the current node, where the model continues to generate reasoning steps until the final answer is obtained. For each inference trajectory Extract the two-dimensional feature vector using the following formula:

[0010] In the formula, The mean negative log-likelihood represents the confidence level when the model generates the corresponding inference trajectory. Its calculation expression is: In the formula, The number of words in the trajectory; The length of the lexical path. For numerically stable terms, This reflects the complexity of the solution.

[0011] As a preferred embodiment, the step of standardizing the extracted features and using the standardized features for inference trajectory clustering includes: right The feature vectors of each inference trajectory are standardized using z-score, and their means are calculated using the following formula: and standard deviation The standardized features are ,in, Take 10^(-8); use the K-means clustering algorithm to cluster the standardized features into A strategy cluster Each cluster represents a class of similar reasoning strategies.

[0012] As a preferred embodiment, the step of counting the number of successful inference trajectories and the total number of trajectories and estimating the success probability for each cluster, and quantifying the clustering uncertainty, includes: for each cluster... Count the number of successful inference trajectories Total number of trajectories The number of successful inference trajectories To obtain the number of trajectories for the correct answer, the probability of success is estimated using the following formula: ; Cluster uncertainty is calculated using the Wilson confidence interval method. The calculation expression is:

[0013] In the formula, This represents the critical value corresponding to the confidence level. The larger the value, the more uncertain the estimate of the success probability of the corresponding cluster; Calculate the overall uncertainty of the nodes using the following formula:

[0014] The overall uncertainty of the nodes incorporates the uncertainties of all clusters and is weighted according to the sample size of each cluster.

[0015] As a preferred embodiment, the step of determining whether the termination condition is met includes the following termination conditions: the overall uncertainty of the node is not greater than the node threshold, the total number of samples reaches the maximum budget, or the uncertainty of all clusters is not greater than the cluster threshold. If any condition is met, sampling is terminated.

[0016] As a preferred embodiment, the step of identifying active cluster sets defines an active cluster as one whose sample size has not reached the upper limit of a single cluster and whose uncertainty is still higher than the clustering threshold. The step of calculating the additional sample count defines the additional sample count as being proportional to the clustering uncertainty and limited to a minimum and maximum value. After generating a new inference trajectory, the standardized features corresponding to each new trajectory are calculated, and the trajectory is assigned to the nearest cluster by calculating the Euclidean distance, with priority given to the target cluster. The statistical information, success probability, and clustering uncertainty of the target cluster are updated, while the overall uncertainty of the nodes is also updated, until the termination condition is met.

[0017] As a preferred approach, the Monte Carlo value is a weighted average of the success probabilities based on the sample size proportion of each cluster.

[0018] As a preferred approach, the steps of calculating the utilization value and exploration reward for each possible successor node pair of the current node according to the expansion direction of the search tree, defining time-adaptive weights, calculating and obtaining a comprehensive expansion score, and selecting the successor node with the highest score for expansion include: for the current node... Each possible successor node pair Calculate the utilization value according to the following expression. In the formula, Take a value between 0.8 and 0.95. Take a value between 0.85 and 0.95. The normalized length of the problem description. The length of the subsequent steps; the exploration reward is calculated according to the following expression. In the formula, To explore coefficients, This represents the cumulative number of visits to node s. For the edge The cumulative number of visits; define the time-adaptive weight as... In the formula This represents the current search iteration number. The time constant is used; the comprehensive extended score is calculated according to the following formula. Select the successor node with the highest score for expansion.

[0019] Secondly, a process-supervised data generation system based on adaptive Monte Carlo search is provided, including: The initial reasoning trajectory generation and feature extraction module is used to generate the initial reasoning trajectory for multi-step reasoning tasks and extract features. The feature standardization and inference trajectory clustering module is used to standardize the extracted features and then use the standardized features to perform inference trajectory clustering. The cluster uncertainty quantification module is used to count the number of successful inference trajectories and the total number of trajectories for each cluster and estimate the success probability, thereby quantifying the cluster uncertainty. The additional sampling module is used to perform adaptive iterative sampling, determine whether the termination condition is met, and if the termination condition is not met, identify the active cluster set, select the active cluster with the highest uncertainty as the additional sampling target, calculate the number of additional samples, and generate a new inference trajectory. The search tree expansion direction guidance module is used to calculate the Monte Carlo value of a node after sampling is terminated, and to guide the expansion direction of the search tree. The adaptive path expansion module is used to calculate the utilization value and exploration reward for each possible successor node pair of the current node according to the expansion direction of the search tree, define time adaptive weights, calculate and obtain the comprehensive expansion score, and select the successor node with the highest score for expansion. The process supervision data generation module is used to record all intermediate inference states and corresponding Monte Carlo values ​​during the search process for each multi-step inference task, thereby generating process supervision data.

[0020] Compared with the prior art, the present invention has at least the following beneficial effects: For each cluster, the number of successful inference trajectories and the total number of trajectories are counted, and the success probability is estimated. The success probability reflects the likelihood of the corresponding inference strategy successfully solving the problem starting from the current node. At the same time, the cluster uncertainty is quantified. During adaptive iterative sampling, it is determined whether the termination condition is met. If the termination condition is not met, an active cluster set is identified, and the active cluster with the highest uncertainty is selected as the target for additional sampling. The number of additional samples is calculated, and new inference trajectories are generated. After sampling terminates, the Monte Carlo value of the node is calculated to guide the expansion direction of the search tree. According to the expansion direction of the search tree, for each possible successor node pair of the current node, the utilization value and exploration reward are calculated, and a time-adaptive weight is defined. Then, the comprehensive expansion score is calculated, and the successor node with the highest score is selected for expansion. This invention uses an uncertainty-driven adaptive sampling mechanism to dynamically allocate computing resources according to the evaluation difficulty of the inference node, which significantly reduces the overall computational load of data generation while ensuring evaluation accuracy. In addition, through dynamic path expansion and trajectory clustering strategies, this invention can generate high-quality supervised data containing longer inference chains and richer step details. The reward mechanism used in the training process of this invention can more effectively guide large language models to perform complex reasoning, enabling them to achieve better accuracy. It also allows models with fewer parameters to achieve performance levels far exceeding their size, achieving more efficient and economical supervision. This invention avoids evaluation bias caused by mixing different strategies through inference trajectory clustering, and balances exploration and utilization through time-adaptive expansion. Compared to fixed sampling methods, this invention significantly reduces computational load while significantly improving the quality of generated data, including longer inference chains and richer step details. It solves the problems of low efficiency, unstable quality, and unreasonable allocation of computational resources in existing process supervision data generation technologies. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the following drawings are only some of the embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0022] Figure 1 Flowchart of the process supervision data generation method based on adaptive Monte Carlo search in this invention embodiment; Figure 2 A flowchart illustrating the adaptive iterative sampling process in this embodiment of the invention; Figure 3 A comparative analysis chart showing the number of inference steps and the number of lexical units per step for the dataset (MathSearch) generated using the method of this invention, and two public datasets (Math-Shepherd and PRM800K). Detailed Implementation

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0024] Please see Figure 1 This invention proposes a process supervision data generation method based on adaptive Monte Carlo search, which addresses the problems of low efficiency, unstable quality, and unreasonable allocation of computing resources in existing process supervision data generation technologies. The method mainly includes the following steps: Generate the initial reasoning trajectory for a multi-step reasoning task and extract its features; The extracted features are standardized, and the standardized features are used for inference trajectory clustering. For each cluster, the number of successful inference trajectories and the total number of trajectories are counted and the success probability is estimated to quantify the clustering uncertainty. Perform adaptive iterative sampling, determine whether the termination condition is met, and if the termination condition is not met, identify the active cluster set, select the active cluster with the highest uncertainty as the additional sampling target, calculate the number of additional samples, and generate a new inference trajectory. After sampling is terminated, the Monte Carlo value of the node is calculated to guide the expansion direction of the search tree; According to the expansion direction of the search tree, for each possible successor node pair of the current node, calculate the utilization value and exploration reward, define time-adaptive weights, calculate and obtain the comprehensive expansion score, and select the successor node with the highest score for expansion; For each multi-step reasoning task, record all intermediate reasoning states and their corresponding Monte Carlo values ​​during the search process to generate process supervision data.

[0025] In one possible implementation, the multi-step reasoning task of this invention is solving a mathematical problem, and a set of mathematical problems to be processed is selected. Each question It includes a problem description and a standard answer. In this embodiment, the basic large language model is selected. As an inference generator, the model parameter size ranges from 1.5B to 72B, preferably from 7B to 32B. The initial sampling quantity is set. The value is 3 to 10, preferably 6; set the maximum sampling budget. The number of clusters is set to 20 to 50, preferably 32; The value is 2 to 5, preferably 3. Furthermore, in this embodiment of the invention, mathematical problems are selected from the MATH500 and GSM8K datasets, and Qwen2.5-Math-7B-Instruct is used as the basic inference model. The initial sampling quantity is set to 6, the maximum sampling budget to 32, and the number of clusters to 3.

[0026] In one possible implementation, for the problem a certain intermediate inference node (representing the inference sequence from the start to step t), using a non-greedy decoding strategy from the model generate A complete inference trajectory. The inference trajectory refers to the complete process from the current node, where the model continues to generate inference steps until the final answer is obtained. During decoding, the temperature parameter is set to 0.6 to 1.0, and the top-p parameter to 0.85 to 0.95 to ensure diversity in the generated data. For each inference trajectory... Extracting two-dimensional feature vectors ,in: The mean negative log-likelihood represents the confidence level of the model when generating the trajectory, and its calculation expression is: , This represents the number of words in the trajectory; the smaller the value, the more confident the model is in the reasoning path. Let ζ be the lexical length of the trajectory (a lexical is the basic unit for language models to process text; typically, one lexical corresponds to one word or subword), and let ζ be... As a numerically stable term, This reflects the complexity of the solution.

[0027] In one possible implementation, for The feature vectors of each inference trajectory are standardized using z-score, and the mean is calculated. and standard deviation The standardized features are ,in We take 10^(-8). The purpose of standardization is to eliminate the dimensional differences between two feature dimensions, ensuring they have equal weight during clustering. The standardized features are then clustered using the K-means clustering algorithm. A strategy cluster In this embodiment, the number of clustering iterations is 1 to 6, and the convergence threshold is set to... Each cluster represents a similar inference strategy, such as algebraic methods, geometric methods, or numerical methods.

[0028] In one possible implementation, for each cluster Count the number of successful inference trajectories (i.e., the number of trajectories that yielded the correct answer) and the total number of trajectories. Calculate the probability of success. This probability reflects the likelihood of this type of reasoning strategy successfully solving the problem starting from the current node. The Wilson confidence interval method is used to calculate the clustering uncertainty. This method is more robust to small samples and extreme probabilities compared to the traditional normal approximation. The calculation expression is as follows: ,in, This is the critical value corresponding to the confidence level; it is set to 1.96 at a 95% confidence level. A larger value indicates greater uncertainty in estimating the success probability of the clustering, requiring more sampling to improve accuracy. The overall uncertainty of the nodes is calculated using the following formula. This value incorporates the uncertainty of all clusters and is weighted according to the sample size of each cluster.

[0029] In one possible implementation, please refer to Figure 2 During the adaptive iterative sampling process, it is determined whether a termination condition is met. The termination condition includes: the overall uncertainty of a node is less than or equal to a node threshold, the total number of samples reaches the maximum budget, or the uncertainty of all clusters is less than or equal to a cluster threshold. The node threshold is between 0.05 and 0.15, preferably 0.1; the cluster threshold is between 0.08 and 0.12, preferably 0.1. Sampling terminates if any condition is met. If the termination condition is not met, an active cluster set is identified. An active cluster refers to a cluster whose sample count has not reached the single cluster upper limit and whose uncertainty is still higher than the cluster threshold. The single cluster sampling upper limit is set to 15 to 25, preferably 20, to prevent any single cluster from consuming excessive resources.

[0030] The active cluster with the highest uncertainty is selected as the target for additional sampling. The number of additional samples is calculated, which is proportional to the uncertainty and is limited to a minimum and maximum value. In this embodiment, the minimum number of samples is 2 to 4, preferably 3; the maximum number of samples is 5 to 10, preferably 8; and the scaling factor is 80 to 120, preferably 100. After generating a new inference trajectory, its standardized features are calculated for each new trajectory, and it is assigned to the nearest cluster by calculating the Euclidean distance, with priority given to the target cluster. The statistics, success probability estimate, and uncertainty of the target cluster are updated, along with the overall uncertainty of the nodes. This step is repeated until the termination condition is met.

[0031] After sampling is terminated, the Monte Carlo value estimate of the node is calculated. In this embodiment, the Monte Carlo value estimate is a weighted average of the success probabilities of each cluster based on its sample size proportion. The Monte Carlo value in this embodiment serves as the quality score of the corresponding node, guiding the expansion direction of the search tree.

[0032] In one possible implementation, adaptive path expansion is performed as follows: At the search tree level, for the current node Each possible successor node pair Calculate the utilization value ,in Take a value of 0.8 to 0.95 (preferably 0.9). Take a value of 0.85 to 0.95 (preferably 0.9). The normalized length of the problem description. The length of the subsequent steps. This formula is designed to ensure high-value nodes (… Approximately 1) and concise steps ( Smaller (smaller) scores higher.

[0033] And, computational exploration rewards ,in The exploration coefficient is set to 1.0 to 2.0 (preferably 1.4). This represents the cumulative number of visits to node s. For the edge The cumulative number of visits. This encourages edges with fewer visits to ensure the breadth of the search.

[0034] Define time-adaptive weights ,in, This represents the current search iteration number. The time constant is set to 50 to 150 (preferably 100). This weight decays exponentially over time, achieving a shift from primarily exploration-based (…) From ) to utilization as the main ( A smooth transition.

[0035] Calculate the comprehensive extended score using the following formula The successor node with the highest score is selected for expansion. Early stage In the early stages, exploration rewards dominate, and searches tend to involve trying different paths; later on... Small, value-driven, and search focused on high-quality paths.

[0036] In one possible implementation, for each problem The training samples are formed by recording all intermediate inference states during the search process and their corresponding Monte Carlo values. In this embodiment, through the above steps, approximately 200,000 process supervision data points were generated on the MATH500 and GSM8K datasets, forming the dataset MathSearch-200K. Each sample in the dataset contains a problem description, an intermediate inference state, and a Monte Carlo value estimate for that state.

[0037] The process supervision data generation method of this invention dynamically allocates computational resources based on the evaluation difficulty of inference steps, achieves adaptive sampling through uncertainty quantification, avoids evaluation bias caused by mixing different strategies through inference trajectory clustering, and balances exploration and utilization through time-adaptive expansion. Compared with fixed sampling methods, this invention significantly reduces computational load while significantly improving the quality of generated data, including longer inference chains and richer step details, thus improving the performance of trained models.

[0038] To verify the effectiveness of the process supervision data generation method based on adaptive Monte Carlo search in improving the quality of process supervision data and the model training effect, the following experiments were conducted.

[0039] 1. Model performance validation The following comparative experiment was conducted, using two mainstream inference models (Qwen3-8B and GLM-4-9B) to test their inference performance on four publicly available mathematical evaluation datasets (AIME, MATH, Olympiad-Bench, and OmniMATH). The core of this experiment is to compare the performance differences between the process reward model trained using the method of this invention (Qwen2.5-Math-7B-PRM-AMCS) and four existing baseline process reward models in guiding the aforementioned inference models to solve problems. The final performance is evaluated using the accuracy (%) achieved on each dataset. The results are shown in the table below:

[0040] 2. Comparative analysis of the quality of data generated during the process Please see Figure 3 This paper compares and analyzes the key quality metrics of the dataset (MathSearch) generated using the method of this invention with two publicly available datasets (Math-Shepherd and PRM800K) generated using existing technologies. The analysis focuses on two core dimensions: the average number of inference steps, reflecting inference complexity, and the average number of terms per step, reflecting inference granularity. The results clearly indicate that the data generated by the method of this invention contains longer inference chains (average 11 steps, compared to 6-7 steps for the baseline method) and richer step details (average 65 terms / step, compared to 32-46 terms / step for the baseline method).

[0041] Another embodiment of the present invention proposes a process supervision data generation system based on adaptive Monte Carlo search, which mainly includes: The initial reasoning trajectory generation and feature extraction module is used to generate the initial reasoning trajectory for multi-step reasoning tasks and extract features. The feature standardization and inference trajectory clustering module is used to standardize the extracted features and then use the standardized features to perform inference trajectory clustering. The cluster uncertainty quantification module is used to count the number of successful inference trajectories and the total number of trajectories for each cluster and estimate the success probability, thereby quantifying the cluster uncertainty. The additional sampling module is used to perform adaptive iterative sampling, determine whether the termination condition is met, and if the termination condition is not met, identify the active cluster set, select the active cluster with the highest uncertainty as the additional sampling target, calculate the number of additional samples, and generate a new inference trajectory. The search tree expansion direction guidance module is used to calculate the Monte Carlo value of a node after sampling is terminated, and to guide the expansion direction of the search tree. The adaptive path expansion module is used to calculate the utilization value and exploration reward for each possible successor node pair of the current node according to the expansion direction of the search tree, define time adaptive weights, calculate and obtain the comprehensive expansion score, and select the successor node with the highest score for expansion. The process supervision data generation module is used to record all intermediate inference states and corresponding Monte Carlo values ​​during the search process for each multi-step inference task, thereby generating process supervision data.

[0042] Another embodiment of the present invention provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the process supervision data generation method based on adaptive Monte Carlo search.

[0043] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0044] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0045] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0046] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0047] The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals. For ease of explanation, the above content only shows the parts related to the embodiments of the present invention; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. This computer-readable storage medium is non-transitory and can be stored in storage devices formed by various electronic devices, enabling the execution process described in the method of the embodiments of the present invention.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for generating process-supervised data based on adaptive Monte Carlo search, characterized in that, include: Generate the initial reasoning trajectory for a multi-step reasoning task and extract its features; The extracted features are standardized, and the standardized features are used for inference trajectory clustering. For each cluster, the number of successful inference trajectories and the total number of trajectories are counted and the success probability is estimated to quantify the clustering uncertainty. Perform adaptive iterative sampling, determine whether the termination condition is met, and if the termination condition is not met, identify the active cluster set, select the active cluster with the highest uncertainty as the additional sampling target, calculate the number of additional samples, and generate a new inference trajectory. After sampling is terminated, the Monte Carlo value of the node is calculated to guide the expansion direction of the search tree; According to the expansion direction of the search tree, for each possible successor node pair of the current node, calculate the utilization value and exploration reward, define time-adaptive weights, calculate and obtain the comprehensive expansion score, and select the successor node with the highest score for expansion; For each multi-step reasoning task, record all intermediate reasoning states and their corresponding Monte Carlo values ​​during the search process to generate process supervision data.

2. The process supervision data generation method based on adaptive Monte Carlo search according to claim 1, characterized in that, The multi-step reasoning task involves solving mathematical problems; a set of mathematical problems to be processed is selected from the MATH500 and GSM8K datasets. Each question Includes a problem description and standard answer; Qwen2.5-Math-7B-Instruct is selected as the basic large language model. Set the initial sampling quantity Maximum sampling budget and the number of clusters .

3. The process supervision data generation method based on adaptive Monte Carlo search according to claim 2, characterized in that, In the step of generating the initial inference trajectory and extracting features for the multi-step inference task, for the problem... a certain intermediate inference node Using a non-greedy decoding strategy from the basic large language model generate A complete reasoning trajectory; The intermediate inference node This represents the reasoning sequence from the beginning to step t; The reasoning trajectory refers to the complete process from the current node, where the model continues to generate reasoning steps until the final answer is obtained. For each inference trajectory Extract the two-dimensional feature vector using the following formula: In the formula, The mean negative log-likelihood represents the confidence level when the model generates the corresponding inference trajectory. Its calculation expression is: In the formula, The number of words in the trajectory; The length of the lexical path. For numerically stable terms, This reflects the complexity of the solution.

4. The process supervision data generation method based on adaptive Monte Carlo search according to claim 1, characterized in that, The step of standardizing the extracted features and using the standardized features for inference trajectory clustering includes: right The feature vectors of each inference trajectory are standardized using z-score, and their means are calculated using the following formula: and standard deviation The standardized features are ,in, Take 10^(-8); use the K-means clustering algorithm to cluster the standardized features into A strategy cluster Each cluster represents a class of similar reasoning strategies.

5. The process supervision data generation method based on adaptive Monte Carlo search according to claim 1, characterized in that, The steps of counting the number of successful inference trajectories and the total number of trajectories for each cluster and estimating the success probability, and quantifying the uncertainty of clustering, include: for each cluster... Count the number of successful inference trajectories Total number of trajectories The number of successful inference trajectories To obtain the number of trajectories for the correct answer, the probability of success is estimated using the following formula: ; Cluster uncertainty is calculated using the Wilson confidence interval method. The calculation expression is: In the formula, This represents the critical value corresponding to the confidence level. The larger the value, the more uncertain the estimate of the success probability of the corresponding cluster; Calculate the overall uncertainty of the nodes using the following formula: The overall uncertainty of the nodes incorporates the uncertainties of all clusters and is weighted according to the sample size of each cluster.

6. The process supervision data generation method based on adaptive Monte Carlo search according to claim 1, characterized in that, The step of determining whether the termination condition is met includes the following termination conditions: the overall uncertainty of the node is not greater than the node threshold, the total number of samples reaches the maximum budget, or the uncertainty of all clusters is not greater than the cluster threshold. If any condition is met, sampling is terminated.

7. The process supervision data generation method based on adaptive Monte Carlo search according to claim 1, characterized in that, The step of identifying active cluster sets refers to clusters whose sample count has not reached the upper limit of a single cluster and whose uncertainty is still higher than the cluster threshold; the step of calculating the additional sampling number is that the additional sampling number is proportional to the cluster uncertainty and is limited to between the minimum and maximum values; after generating a new inference trajectory, the corresponding standardized features are calculated for each new trajectory, and the trajectory is assigned to the nearest cluster by calculating the Euclidean distance, with priority given to the target cluster; Update the statistical information, success probability, and cluster uncertainty of the target cluster, and update the overall uncertainty of the nodes until the termination condition is met.

8. The process supervision data generation method based on adaptive Monte Carlo search according to claim 1, characterized in that, The Monte Carlo value is a weighted average of the success probabilities based on the sample size proportion of each cluster.

9. The process supervision data generation method based on adaptive Monte Carlo search according to claim 1, characterized in that, The steps of calculating the utilization value and exploration reward for each possible successor node pair of the current node according to the expansion direction of the search tree, defining time-adaptive weights, calculating and obtaining a comprehensive expansion score, and selecting the successor node with the highest score for expansion include: for the current node... Each possible successor node pair Calculate the utilization value according to the following expression. In the formula, Take a value between 0.8 and 0.

95. Take a value between 0.85 and 0.

95. The normalized length of the problem description. The length of the subsequent steps; the exploration reward is calculated according to the following expression. In the formula, To explore the coefficient, This represents the cumulative number of visits to node s. For the edge The cumulative number of visits; define the time-adaptive weight as... In the formula This represents the current search iteration number. The time constant is used; the comprehensive extended score is calculated according to the following formula. Select the successor node with the highest score for expansion.

10. A process-supervised data generation system based on adaptive Monte Carlo search, characterized in that, include: The initial reasoning trajectory generation and feature extraction module is used to generate the initial reasoning trajectory for multi-step reasoning tasks and extract features. The feature standardization and inference trajectory clustering module is used to standardize the extracted features and then use the standardized features to perform inference trajectory clustering. The cluster uncertainty quantification module is used to count the number of successful inference trajectories and the total number of trajectories for each cluster and estimate the success probability, thereby quantifying the cluster uncertainty. The additional sampling module is used to perform adaptive iterative sampling, determine whether the termination condition is met, and if the termination condition is not met, identify the active cluster set, select the active cluster with the highest uncertainty as the additional sampling target, calculate the number of additional samples, and generate a new inference trajectory. The search tree expansion direction guidance module is used to calculate the Monte Carlo value of a node after sampling is terminated, and to guide the expansion direction of the search tree. The adaptive path expansion module is used to calculate the utilization value and exploration reward for each possible successor node pair of the current node according to the expansion direction of the search tree, define time adaptive weights, calculate and obtain the comprehensive expansion score, and select the successor node with the highest score for expansion. The process supervision data generation module is used to record all intermediate inference states and corresponding Monte Carlo values ​​during the search process for each multi-step inference task, thereby generating process supervision data.

Images