A breadth-first thought tree reasoning acceleration method based on speculative exploration

Abstract

The application provides a breadth-first thinking tree reasoning acceleration method based on speculative exploration, and belongs to the field of deep learning and computer architecture. The method takes a reward-balanced search algorithm as a basic breadth-first thinking tree reasoning strategy, constructs a producer-consumer execution framework decoupling branch expansion execution and search control logic, designs two core mechanisms of inter-query speculative budget allocation and intra-query speculative branch selection; according to system hardware constraints and query reasoning characteristics, global speculative resources are adaptively allocated, and the optimal node is subjected to speculative exploration by using the idle computing slot position of the GPU in the inter-layer synchronization waiting stage, thereby covering the computing power gap caused by the long-tail branch waiting, breaking the reward dependence barrier, improving the GPU hardware utilization and the system throughput in the multi-concurrent scenario, and shortening the execution delay of complex reasoning tasks; the hit checking mechanism is used to ensure that the reasoning path and the search quality are unchanged, and the efficient acceleration of the thinking tree reasoning is realized without losing the reasoning accuracy.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and computer architecture, specifically to a speculative exploration hardware acceleration and system scheduling method for tree-of-thought (ToT) reasoning tasks in the form of breadth-first search (BFS) for large language models. Background Technology

[0002] Recent studies have shown that the reasoning ability of Large Language Models (LLMs) significantly improves with increasing inference computation budget. To fully leverage the multi-step reasoning capabilities of LLMs in complex logical reasoning tasks in real-world applications such as mathematical problem solving, code generation, and strategy planning, two classic reasoning paradigms have been proposed: Chain-of-Thought (CoT) deepens the reasoning steps through single-path linear deduction; while Tree-of-Thought (ToT) expands the breadth of reasoning by relying on a tree-like structure with multi-path parallel search.

[0003] Specifically, structurally, a mind tree is a tree structure with the problem as the root node and thoughts as child nodes. Starting from the problem, the mind tree models the reasoning process as an extension of discrete steps, each step constituting a coherent thought fragment. Therefore, nodes represent partial solution states reached through steps, and the sequential trajectory from the problem to the final answer constitutes a complete reasoning path. In terms of its operational mechanism, during the expansion of the mind tree, ToT generates a reward signal for each newly expanded node through a specialized reward model or self-evaluation of the reasoning model to evaluate the quality of that thought node. Subsequently, the system determines, based on this reward signal, whether the next step should involve in-depth exploration or pruning and backtracking. Depending on the expansion and traversal strategies, the ToT algorithm can generally be divided into Depth-First Search (DFS) and Breadth-First Search (BFS).

[0004] In the actual execution process, the breadth-first search-based ToT maintains a set of currently optimal candidate nodes at each search depth. Specifically, the system evaluates newly generated candidate nodes based on the reward value fed back from the reward signal, and waits until all candidate nodes at the current depth have been generated and scored before summarizing all scores and performing a global sort to select the nodes to be expanded in the next layer. A representative algorithm is Reward Balanced Search (REBASE), which uses a Softmax mechanism to dynamically allocate the expansion budget. Given a search depth... Total sampling budget ,node Extended width The calculation formula is:

[0005]

[0006] in, Represents a node The reward value; Indicates the equilibrium temperature; Indicates the current depth The REBASE algorithm expands the width of the best candidate node with the highest reward value, allowing for more branches to be added to that node. This dynamically balances the total GPU computing resources and improves the quality of LLM inference.

[0007] However, such benchmark processes are severely constrained by the "RewardDependency Barrier" when executed on the underlying hardware system. Specifically, because branch expansion decisions depend on the global aggregation of reward signals from all nodes at the same depth, the system must wait for the generation and scoring of all candidate nodes at the current depth to be completed before it can unblock the synchronization and move to the next layer. This mechanism leads to a severe long-tail problem: in actual inference, the length of tokens generated by each branch usually varies significantly. When some shorter branches complete ahead of schedule, the corresponding GPU threads and computing slots are forced to remain idle to wait for the slowest "long-tail" branches to complete, resulting in a significant increase in overall inference latency.

[0008] Furthermore, existing LLM inference systems (such as vLLM and SGLang) are primarily designed for single-path linear CoT inference. Their underlying paging attention and key-value cache management mechanisms are not adapted to the tree-like concurrency characteristics of BFS-type ToT, and cannot proactively utilize idle GPU resources during reward synchronization. Existing algorithm optimizations also largely focus on compressing token length, failing to overcome the physical barrier of inter-layer reward synchronization at the system scheduling level. These dual system and algorithmic deficiencies make it difficult for existing technologies to overcome the performance bottleneck of breadth-first tree-like inference, resulting in low memory bandwidth utilization and an inability to achieve efficient parallel acceleration.

[0009] Currently, inference optimization for breadth-first search trees still faces the following key technical challenges: First, in multi-query concurrent service scenarios, due to significant differences in the depth of the thought trees between different queries, and considering the constraints of limited system resources, existing strategies fail to effectively utilize the search state, speculative utility, and overall system capacity of queries to prioritize different queries and dynamically allocate budgets among them. Second, breadth-first search requires aggregating the reward signals of all branches layer by layer before proceeding to the next expansion. Strict inter-layer synchronization mechanisms can lead to severe long-tail blocking problems. Because branch lengths within the same query vary significantly, short branches result in long periods of idle computing power after completion, significantly increasing overall inference latency. Third, existing inference optimizations are all designed for single-path linear inference and lack scheduling mechanisms adapted to tree-like multi-branch searches. They cannot pre-expand high-potential branches without relying on reward synchronization, making it difficult to balance inference efficiency and accuracy. Summary of the Invention

[0010] To address the problems existing in the prior art, this invention proposes a breadth-first thinking tree reasoning acceleration method based on speculative exploration. For a breadth-first thinking tree reasoning service system for large language models, it innovatively designs and implements two core mechanisms: inter-query budget allocation and intra-query speculative selection, in order to break through the reward dependency barrier and improve the execution efficiency of underlying hardware resources.

[0011] For ease of understanding, the following lists the names and meanings of relevant terms in this invention:

[0012] Main branch: The original, actual exploration path of the underlying thought tree algorithm when speculative exploration strategies are not applied. In this paper, the underlying thought tree algorithm is the reward-balanced search REBASE algorithm.

[0013] Speculative Branch: When applying a speculative exploration strategy, the query in-internal speculative branch selection mechanism predicts the future main branch.

[0014] Main branch node: A node that belongs to the main branch.

[0015] Speculative nodes: Nodes that belong to the speculative branch.

[0016] Extension: The producer thread invokes the backend's large language model to generate child nodes for a given parent node.

[0017] Explore: Expand the node and collect its reward value.

[0018] The technical solution provided by this invention is as follows:

[0019] A breadth-first thinking tree reasoning acceleration method based on speculative exploration is applied to a large language model breadth-first thinking tree reasoning service system. This system comprises a service-side and an application-side. The service-side includes an asynchronous frontend and an execution backend, while the application-side includes a general producer-consumer framework. In the general producer-consumer framework, the producer thread waits for nodes to be expanded in the request queue, obtains a node, calls the large language model in the execution backend to generate child nodes, and writes the generation result to the completion queue. The consumer thread listens to the completion queue to obtain node expansion results, maintains the thinking tree topology based on node type and hit verification, and selects speculative nodes to add to the request queue when there are idle producers, thus decoupling branch expansion execution from search control logic. This method uses the reward-balanced search REBASE algorithm as the basic breadth-first thinking tree reasoning strategy and is characterized by the following steps:

[0020] Step 1: Initialize the system and task acceptance; First, complete the loading and construction of the service layer and application layer, including the initialization of the service layer front-end and back-end framework and the execution back-end large language model, and the initialization of the application layer producer-consumer execution framework; Next, receive natural language query data (such as reasoning requests for tasks like mathematical solving, code generation, and strategy planning) from users in complex logical reasoning scenarios through the asynchronous front-end of the service layer, forming a set of concurrent reasoning query requests. Each query corresponds to a mind tree task based on a breadth-first search strategy, and the root node of the mind tree is registered in the application layer's request queue.

[0021] Step 2: First, the system determines the global projected total budget through roofline analysis. This refers to the batch concurrency at the intersection of the maximum computing power and the maximum memory. Subsequently, the consumer thread performs a query for each inference operation in the collection. Calculate its predicted utility score The calculation formula is:

[0022]

[0023] in, Indicates the current inference query Internally available parallel capacity; This indicates the prediction hit rate of the speculative branch; Indicates the size of the parameters in a large language model; Indicates reasoning query The key-value state that can be reused by its inferred branches.

[0024] Step 3: The system executes the inter-query inferred budget allocation. Based on the inferred utility score obtained in Step 2. The consumer thread will globally estimate the total budget. Distribute to each concurrent inference query, and obtain each inference query The projected budget for actual allocation The calculation formula is:

[0025]

[0026] in, Used to control the distribution of sharpness; Indicates rounding down; Indicates the current inference query Internally available parallel capacity; summation index Traverse the set of concurrent reasoning query requests ; This equals the maximum number of producer threads that the query can use simultaneously.

[0027] Step 4: Obtaining the projected budget Each reasoning query The consumer thread performs speculative branch selection within the query. When the short branch completes its exploration early (i.e., token generation and reward scoring, freeing up idle GPU computing slots), it must wait for the long branch to complete its exploration before the synchronization block can be lifted and the next layer can proceed. Utilizing the idle GPU computing slots created during this time, the consumer thread proactively and speculatively explores the completed nodes to overcome the "reward dependency barrier." This includes the following sub-steps:

[0028] 4-1. At the current depth of the mind tree The producer thread waits for nodes to be expanded in the request queue, calls the large language model in the execution backend to complete the generation of child nodes, and then writes the generation result to the completion queue; the consumer thread listens to the completion queue and obtains the set of nodes that have been generated. This gives the number of idle GPU computing slots. .

[0029] 4-2. Obtain the set of nodes that have been generated through self-evaluation using a specialized reward model or inference model. Each node that has been generated Reward value At this point, these nodes are marked as having been explored. Due to synchronization bottlenecks, the system needs to wait for all nodes currently being explored at the current depth to complete their exploration before it can proceed to the main branch of the next depth. The main branch represents the original, actual exploration path of the underlying thought tree algorithm when speculative exploration strategies are not applied.

[0030] 4-3. Take the projected budget allocated to this query in step 3. (Equal to the maximum number of producer threads that the query can occupy simultaneously) and the number of currently available GPU computing slots. The smaller of (equal to the number of currently idle producer threads) As the actual speculative budget, this budget represents the total number of speculative child nodes currently allowed for speculative exploration. Subsequently, the actual speculative budget... and the set of nodes whose child nodes have been generated Each node in Reward value As input, the REBASE algorithm is used to calculate the inferred expansion width of each node. :

[0031]

[0032] in, Indicates the equilibrium temperature. The value represents the predicted child node to be generated for the corresponding node n. The number of.

[0033] 4-4. Add all nodes to be expanded According to its reward value Push into priority queue Press in Next, among them For nodes The number of speculative child nodes being generated or already generated indicates the number of nodes that still need to be generated based on the current number. Each predictive child node selects a corresponding reward value for each time it leaves the queue. The highest node to be expanded .

[0034] Step 5: Speculative exploration and main branch exploration. Consumer threads will prioritize the queue. elements in According to its reward value Dequeue nodes of varying quality one by one and write them to the request queue; the producer thread retrieves nodes to be expanded from the request queue. The large language model in the execution backend is invoked to perform operations at the next deeper level. Inferred child nodes The actual generation. This process is related to the current depth. The unfinished "long-tail" main branch is computed in parallel, using computing power reuse to mask the waiting delay caused by inter-layer reward synchronization. If the depth is at this point... If all main branch nodes have been explored, proceed to step 6; otherwise, return to step 4 and continue exploring other main branch nodes while simultaneously performing the exploration of speculative nodes in parallel.

[0035] Step 6: Depth at this point All main branch nodes have been explored, and the consumer thread performs inter-layer global reward synchronization and speculative node selection operations:

[0036] 6-1. The system aggregates the current depth Rewards for all main branch nodes Remove the synchronization barrier and execute the REBASE algorithm to calculate the actual expansion width of each main branch node. :

[0037]

[0038] in, Indicates depth Total sampling budget; Indicates the equilibrium temperature; Indicates depth All nodes. Using a hash table. Establish a mapping between each main branch node and its expansion requirements, where the key is the main branch node and the value is the number of main branch child nodes to be generated; for each main branch node... , The initial value is .

[0039] 6-2. For the speculative child nodes generated by the large language model in step 5, which are executed by the idle producer thread calling the backend. As a result, the actual expansion requirements Cross-matching is used for hit verification. The predicted child nodes are recorded. The parent node is ,like If the prediction is successful, then the predicted child node will be selected. Access to Mind Tree Depth The corresponding position of the layer is added to the mind tree as a child node of the main branch, thereby saving the generation time of that node, and at the same time... The corresponding value was modified to ;like If the prediction fails, the prediction child node is immediately terminated. The generated value releases the occupied GPU memory and computing resources, and is not added to the mind tree as a main branch child node.

[0040] 6-3. The process terminates when the mind tree reaches the system's preset maximum reasoning depth, or when all branch nodes at the current depth generate an end marker indicating that the solution is complete; otherwise, the consumer thread writes the remaining main branch nodes into the request queue, waiting for the producer thread to expand its child nodes, until all nodes are completed. The number of child nodes reached Update the current depth to Proceed to step 4 to explore the next layer of nodes.

[0041] After the process is terminated, the system traces back along the mind tree to the complete thought path with the highest accumulated reward value, and returns it to the user as the final reasoning solution string for the mathematical problem-solving, code generation, or strategy planning task.

[0042] The beneficial effects of this invention are as follows:

[0043] This invention provides a breadth-first reasoning acceleration method based on speculative exploration, targeting breadth-first reasoning scenarios for large language models. It constructs a producer-consumer execution framework that decouples branch expansion execution and search control logic, and designs two core mechanisms: inter-query speculative budget allocation and intra-query speculative branch selection. Global speculative resources are adaptively allocated based on system hardware constraints and query reasoning characteristics. Idle GPU computing slots during inter-layer synchronization waiting phases are used to conduct speculative exploration of optimal nodes, simultaneously covering the computational gaps caused by waiting for long-tail branches and breaking down reward dependency barriers. By adopting the technical solution provided by this invention, idle computing windows during the reasoning process can be effectively filled, improving GPU hardware utilization and system throughput in multi-concurrency scenarios, significantly shortening the execution latency of complex reasoning tasks. Simultaneously, a hit verification mechanism ensures that the reasoning path and search quality remain unchanged, achieving efficient acceleration of reasoning tree inference without sacrificing inference accuracy. This method is widely adaptable to complex logical reasoning scenarios such as mathematical solving, code generation, and strategy planning, improving the deployment efficiency and engineering practicality of large language model reasoning services. Attached Figure Description

[0044] Figure 1This is a flowchart of the breadth-first thinking tree reasoning acceleration method based on speculative exploration, which is part of the present invention.

[0045] Figure 2 This is an example diagram of query-based inference branch selection in an embodiment of the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0047] This invention addresses the reward dependency barrier in breadth-first reasoning of large language models by proposing a method for accelerating breadth-first reasoning based on speculative exploration. The flowchart of this method is shown below. Figure 1 As shown, the query internal inference branch selection process is as follows: Figure 2 As shown. The method of this invention is applied to a breadth-first thinking tree reasoning service system based on a large language model. This embodiment uses an NVIDIA A6000 GPU hardware environment, deploys the Llemma-7B large language model, and uses the MATH-500 mathematical reasoning dataset as the input for natural language query data. The system includes a service layer and an application layer; the service layer includes an asynchronous front-end and an execution back-end, and the application layer includes a general producer-consumer execution framework; in the general producer-consumer execution framework, the producer thread is used to wait for nodes to be expanded in the request queue, and after obtaining the node, it calls the large language model of the execution back-end to complete the generation of child nodes and writes the generation result into the completion queue; the consumer thread is used to listen to the completion queue to obtain the node expansion result, maintain the thinking tree topology structure according to the node type and the hit verification, and select the speculative node to add to the request queue when there is an idle producer, thereby decoupling the branch expansion execution and the search control logic; the method uses the reward-balanced search REBASE algorithm as the basic breadth-first thinking tree reasoning strategy, including the following steps:

[0048] Step 1: System Initialization and Task Acceptance. First, the system completes the initialization of the service layer's SGLang front-end and back-end framework and the execution back-end Llemma-7B model, and initializes the producer-consumer execution framework. Next, the service layer asynchronously receives the MATH-500 mathematical inference dataset, forming a set of concurrent inference query requests. Each query corresponds to a mind tree task based on a breadth-first search strategy, and the root node of the mind tree is registered in the application layer's request queue.

[0049] Step 2: First, based on the A6000 GPU's computing power limit (approximately 38.7 TFLOPS) and memory limit (approximately 768 GB / s), the system determines the global inference total budget through roofline analysis. Subsequently, the consumer thread performs a reasoning query for each item in the collection. Calculate its predicted utility score The calculation formula is:

[0050]

[0051] in, Indicates the current inference query The internally available parallel capacity is configured to be 16 in this embodiment; This indicates the prediction accuracy of the speculative branch, which is approximately 0.5. This indicates the parameter size of the Lemma-7B model, which is approximately 14GB. Indicates reasoning query The key-value state quantity that can be reused by its inferred branches varies dynamically with different MATH-500 inference tasks, and is approximately in the MB range.

[0052] Step 3: The system executes the inter-query inferred budget allocation. Based on the inferred utility score obtained in Step 2. The consumer thread will globally estimate the total budget. Distribute to each concurrent inference query, and obtain each inference query The projected budget for actual allocation The calculation formula is:

[0053]

[0054] in, Used to control the allocation of sharpness, this embodiment takes... ; Indicates rounding down; Indicates the current inference query The internally available parallel capacity is configured to be 16 in this embodiment; summation index. Traverse the set of concurrent reasoning query requests The projected budget for each query assignment is calculated. .

[0055] This concludes the query for inferred budget allocation. The following section will examine single queries. Expand the query's internal inference branch selection example, such as... Figure 2 The other two query execution processes are similar and are executed in parallel with them, so they will not be discussed further here.

[0056] Step 4: Obtaining the projected budget Each reasoning query The consumer thread performs speculative branch selection within the query. When the short branch completes its exploration early (i.e., token generation and reward scoring, freeing up idle GPU computing slots), it must wait for the long branch to complete its exploration before the synchronization block can be lifted and the next layer can proceed. Utilizing the idle GPU computing slots created during this time, the consumer thread proactively and speculatively explores the completed nodes to overcome the "reward dependency barrier." This includes the following sub-steps:

[0057] 4-1. At the current depth of the mind tree The producer thread waits for nodes to be expanded in the request queue, calls the large language model in the execution backend to complete the generation of child nodes, and then writes the generation result to the completion queue; the consumer thread listens to the completion queue and obtains the set of nodes whose child nodes have been generated. This gives the number of idle GPU computing slots. .

[0058] 4-2. Obtain the set of nodes that have completed child node generation through self-evaluation using a specialized reward model or inference model. Each node that has been generated Reward value At this point, these nodes are marked as having been explored. Due to synchronization bottlenecks, the system needs to wait for all nodes currently being explored at the current depth to complete their exploration before it can proceed to the main branch of the next depth. The main branch represents the original, actual exploration path of the underlying thought tree algorithm when speculative exploration strategies are not applied.

[0059] 4-3. Take the projected budget allocated to this query in step 3. (Equal to the maximum number of producer threads that the query can occupy simultaneously) and the number of currently available GPU computing slots. The smaller of (equal to the number of currently idle producer threads) As the actual speculative budget, this budget represents the total number of speculative child nodes currently allowed for speculative exploration. Subsequently, the actual speculative budget... and the set of nodes whose child nodes have been generated Each node in Reward value As input, the REBASE algorithm is used to calculate the inferred expansion width of each node. :

[0060]

[0061] in, Indicates the equilibrium temperature. The value represents the predicted child node to be generated for the corresponding node n. The number of.

[0062] 4-4. Add all nodes to be expanded According to its reward value Push into priority queue Press in Next, among them For nodes The number of speculative child nodes being generated or already generated indicates the number of nodes that still need to be generated based on the current number. Each predictive child node selects a corresponding reward value for each time it leaves the queue. The highest node to be expanded .

[0063] Step 5: Speculative exploration and main branch exploration. Consumer threads will prioritize the queue. elements in According to its reward value Dequeue nodes of varying quality one by one and write them to the request queue; the producer thread retrieves nodes to be expanded from the request queue. The backend is invoked to execute the process at the next deeper level. Inferred child nodes The actual generation. This process is related to the current depth. The unfinished "long-tail" main branch is computed in parallel, using computing power reuse to mask the waiting delay caused by inter-layer reward synchronization. If the depth is at this point... If all main branch nodes have been explored, proceed to step 6; otherwise, return to step 4 and continue exploring other main branch nodes while simultaneously performing the exploration of speculative nodes in parallel.

[0064] In this example, steps 4 and 5 are implemented as follows:

[0065] Step 4: Obtaining the projected budget Reasoning query The consumer thread performs inference branch selection within the query.

[0066] 4-1. At the current depth of the mind tree The producer thread waits for nodes to be expanded in the request queue, calls the large language model in the execution backend to complete the generation of child nodes, and then writes the generation result to the completion queue; the consumer thread listens to the completion queue and obtains the set of nodes that have been generated. This gives the number of idle GPU computing slots. .

[0067] 4-2. Obtain the set of nodes that have been generated through self-evaluation using a specialized reward model or inference model. The reward value of each completed node 2 in the middle At this point, node 2 is marked as having completed exploration. Due to synchronization bottlenecks, the system needs to wait for all nodes currently being explored at the current depth to complete their exploration before it can proceed to the main branch exploration of the next depth.

[0068] 4-3. Take the projected budget allocated to this query in step 3. (Equal to the maximum number of producer threads that the query can occupy simultaneously) and the number of currently available GPU computing slots. The smaller of (equal to the number of currently idle producer threads) As the actual speculative budget, this budget represents the total number of speculative child nodes currently allowed for speculative exploration. Subsequently, the actual speculative budget... and the set of nodes that have been generated Each node in Reward value As input, the REBASE algorithm is used to calculate the inferred expansion width of each node. :

[0069]

[0070] in, Indicates the equilibrium temperature. The value represents the predicted child node to be generated for the corresponding node n. The number of. In this embodiment ; Calculated .

[0071] 4-4. Expand node 2 according to its reward value. Push into priority queue Press in Next, among them This represents the number of speculative child nodes that node 2 is currently generating or has already generated. It indicates that one more speculative child node needs to be generated beyond the current number. The corresponding reward value is selected for each dequeue operation. The highest node to be expanded .

[0072] Step 5: The consumer thread prioritizes the queue. elements in According to its reward value Dequeue nodes of varying heights one by one and write them to the request queue; the producer thread retrieves node 2 to be expanded from the request queue and calls the large language model in the execution backend to process nodes located at the next depth level. The actual generation of the inferred child node 5. This process is related to the current depth. The unfinished "long tail" main branch is computed in parallel, using computing power reuse to mask the waiting delay caused by inter-layer reward synchronization. At this point, the depth... There are also main branch nodes The exploration is incomplete, so proceed to step 4.

[0073] Step 4: Obtaining the projected budget Each reasoning query The consumer thread performs inference branch selection within the query.

[0074] 4-1. At the current depth of the mind tree The producer thread waits for nodes to be expanded in the request queue, calls the large language model in the execution backend to complete the generation of child nodes, and then writes the generation result to the completion queue; the consumer thread listens to the completion queue and obtains the set of nodes that have been generated. This gives the number of idle GPU computing slots. .

[0075] 4-2. Obtain the set of nodes that have completed child node generation through self-evaluation using a specialized reward model or inference model. The reward value of node 4 that has been generated is... At this point, node 4 is marked as having completed exploration. Due to synchronization bottlenecks, the system needs to wait for all nodes currently being explored at the current depth to complete their exploration before it can proceed to the main branch exploration of the next depth.

[0076] 4-3. Take the projected budget allocated to this query in step 3. (Equal to the maximum number of producer threads that the query can occupy simultaneously) and the number of currently available GPU computing slots. The smaller of (equal to the number of currently idle producer threads) As the actual speculative budget, this budget represents the total number of speculative child nodes currently allowed for speculative exploration. Subsequently, the actual speculative budget... and the set of nodes whose child nodes have been generated Each node in Reward value As input, the REBASE algorithm is used to calculate the inferred expansion width of each node. :

[0077]

[0078] in, Indicates the equilibrium temperature. The value represents the predicted child node to be generated for the corresponding node n. The number of. In this embodiment , Calculated , .

[0079] 4-4. Add all nodes to be expanded According to its reward value Push into priority queue In this embodiment, node 2 is pushed in. Next, node 4 is pushed 0 times, indicating that one more speculative child node of node 2 needs to be generated based on the current situation. Each time a node is dequeued, the corresponding reward value is selected. The highest node to be expanded .

[0080] Step 5: The consumer thread prioritizes the queue. Element 2 in the middle is based on its reward value Dequeue nodes of varying heights one by one and write them to the request queue; the producer thread retrieves node 2 to be expanded from the request queue and calls the large language model in the execution backend to process nodes located at the next depth level. The actual generation of the inferred child node 6. This process is related to the current depth. The unfinished "long tail" main branch is computed in parallel, using computing power reuse to mask the waiting delay caused by inter-layer reward synchronization. At this point, the depth... There are also main branch nodes The exploration is incomplete, so proceed to step 4.

[0081] Step 4: Obtaining the projected budget Each reasoning query The consumer thread performs inference branch selection within the query.

[0082] 4-1. At the current depth of the mind tree The producer thread waits for nodes to be expanded in the request queue, calls the large language model in the execution backend to complete the generation of child nodes, and then writes the generation result to the completion queue; the consumer thread listens to the completion queue and obtains the set of nodes that have been generated. This gives the number of idle GPU computing slots. .

[0083] 4-2. Obtain the set of nodes that have been generated through self-evaluation using a specialized reward model or inference model. Each node that has been generated Reward value At this point, these nodes are marked as having been explored. Due to synchronization bottlenecks, the system needs to wait for all nodes currently being explored at the current depth to complete their exploration before it can proceed to the main branch of the next depth. The main branch represents the original, actual exploration path of the underlying thought tree algorithm when speculative exploration strategies are not applied.

[0084] 4-3. Take the projected budget allocated to this query in step 3. (Equal to the maximum number of producer threads that the query can occupy simultaneously) and the number of currently available GPU computing slots. The smaller of (equal to the number of currently idle producer threads) As the actual speculative budget, this budget represents the total number of speculative child nodes currently allowed for speculative exploration. Subsequently, the actual speculative budget... and the set of nodes that have been generated Each node in Reward value As input, the REBASE algorithm is used to calculate the inferred expansion width of each node. :

[0085]

[0086] in, Indicates the equilibrium temperature. The value represents the predicted child node to be generated for the corresponding node n. The number of. In this embodiment , , Calculated , , .

[0087] 4-4. Add all nodes to be expanded According to its reward value Push into priority queue In this embodiment, node 1 is pushed in. Next, node 2 is pushed 0 times, and node 4 is pushed 0 times, indicating that one more speculative child node of node 1 needs to be generated based on the current situation. Each time a node is dequeued, the corresponding reward value is selected. The highest node to be expanded .

[0088] Step 5: The consumer thread prioritizes the queue. elements in According to its reward value Dequeue nodes of varying heights one by one and write them to the request queue; the producer thread retrieves node 1 to be expanded from the request queue and calls the execution backend to proceed to the next level. Inferred child nodes The actual generation. This process is related to the current depth. The unfinished "long-tail" main branch is computed in parallel, using computing power reuse to mask the waiting delay caused by inter-layer reward synchronization. Further details are needed. All main branch nodes All explorations have been completed, so proceed to step 6.

[0089] Step 6: Depth at this point All main branch nodes have been explored, and the consumer thread performs inter-layer global reward synchronization and speculative node selection operations:

[0090] 6-1. The system aggregates the current depth Rewards for all main branch nodes Remove the synchronization barrier and execute the REBASE algorithm to calculate the actual expansion width of each main branch node. :

[0091]

[0092] in, Indicates depth Total sampling budget; Indicates the equilibrium temperature; Indicates depth All nodes. Using a hash table. Establish a mapping between each main branch node and its expansion requirements, where the key is the main branch node and the value is the number of main branch child nodes to be generated; for each main branch node... , The initial value is .

[0093] 6-2. For the speculative child nodes generated by the large language model in step 5, which are executed by the idle producer thread calling the backend. As a result, the actual expansion requirements Cross-matching is used for hit verification. The predicted child nodes are recorded. The parent node is ,like If the prediction is successful, then the predicted child node will be selected. Access to Mind Tree Depth The corresponding position of the layer is added to the mind tree as a child node of the main branch, thereby saving the generation time of that node, and at the same time... The corresponding value was modified to ;like If the prediction fails, the prediction child node is immediately terminated. The generated value releases the occupied GPU memory and computing resources, and is not added to the mind tree as a main branch child node.

[0094] 6-3. The process terminates when the mind tree reaches the system's preset maximum reasoning depth, or when all branch nodes at the current depth generate an end marker indicating that the solution is complete; otherwise, the consumer thread writes the remaining main branch nodes into the request queue, waiting for the producer thread to expand its child nodes, until all nodes are completed. The number of child nodes reached , will the current depth Updated to Proceed to step 4 to explore the next layer of nodes.

[0095] After the process is terminated, the system traces back along the mind tree to find the complete thought path with the highest accumulated reward value, and returns it to the user as the final reasoning solution string for the mathematical problem-solving task.

[0096] Experimental results show that, compared to the naive breadth-first search tree reasoning method without speculative exploration, this embodiment achieves an end-to-end inference speedup of 1.2 to 1.9 times under the same GPU hardware, model, and task configuration, effectively shortening the inference execution time. This method utilizes only idle GPU computing resources throughout the process for speculative execution, keeping the additional overhead within 15%, significantly improving system throughput in multi-concurrency scenarios. At the same time, through global synchronization and hit verification mechanisms, the inference solution accuracy remains consistent with the benchmark REBASE algorithm without precision loss, effectively solving the technical problems of long inter-layer synchronization wait times, idle computing power, and low inference efficiency in existing technologies.

[0097] Finally, it should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the claims.

Claims

1. A breadth-first thought tree reasoning acceleration method based on speculative exploration, applied to a large language model breadth-first thought tree reasoning service system, which includes a service layer and an application layer; wherein, The service layer includes an asynchronous front-end and an execution back-end, while the application layer includes a general producer-consumer execution framework. In this framework, the producer thread waits for nodes to be expanded in the request queue, obtains a node, calls the large language model of the execution back-end to generate child nodes, and writes the generation result to the completion queue. The consumer thread listens to the completion queue to obtain node expansion results, maintains the thought tree topology based on node type and hit verification, and selects a predicted node to add to the request queue when an idle producer exists. This method uses the reward-balanced search REBASE algorithm as the basic breadth-first thought tree inference strategy, and is characterized by the following steps: Step 1: Initialize the system and task acceptance; First, complete the loading and construction of the service layer and application layer, including the initialization of the service layer front-end and back-end framework and the execution back-end large language model, and the initialization of the application layer producer-consumer execution framework; Next, receive natural language query data from complex logical reasoning scenarios input by users through the asynchronous front-end of the service layer, forming a set of concurrent reasoning query requests. Each query corresponds to a mind tree task based on a breadth-first search strategy, and the root node of the mind tree is registered to the request queue of the application layer. Step 2: First, the system determines the global projected total budget through roofline analysis. This refers to the batch concurrency at the intersection of the maximum computing power and the maximum memory; subsequently, the consumer thread performs a query for each inference query in the set. Calculate its predicted utility score ; Step 3: The system performs inter-query inferred budget allocation; based on the inferred utility score obtained in Step 2. The consumer thread will globally estimate the total budget. Distribute to each concurrent inference query, and obtain each inference query The projected budget for actual allocation ; Step 4: Obtaining the projected budget Each reasoning query The consumer thread performs inference branch selection within the query; specifically, it includes the following sub-steps: 4-1. At the current depth of the mind tree The producer thread waits for nodes to be expanded in the request queue, calls the large language model in the execution backend to complete the generation of child nodes, and then writes the generation result to the completion queue; the consumer thread listens to the completion queue and obtains the set of nodes that have been generated. This gives the number of idle GPU computing slots. ; 4-2. Obtain the set of nodes that have been generated through self-evaluation of the reward model or inference model. Each node that has been generated Reward value At this point, these nodes are marked as having been explored. 4-3. Take the projected budget allocated to this query in step 3. Number of currently available GPU compute slots smaller value As the actual speculative budget, this budget represents the total number of speculative child nodes currently allowed for speculative exploration; subsequently, the actual speculative budget... and the set of nodes whose child nodes have been generated Each node in Reward value As input, the REBASE algorithm is used to calculate the inferred expansion width of each node. ; 4-4. Add all nodes to be expanded According to its reward value Push into priority queue Press in Next, among them For nodes The number of speculative child nodes being generated or already generated indicates the number of nodes that still need to be generated based on the current number. Each predictive child node selects a corresponding reward value for each time it leaves the queue. The highest node to be expanded ; Step 5: Speculative exploration and main branch exploration; the consumer thread prioritizes the queue. elements in According to its reward value Dequeue nodes of varying quality one by one and write them to the request queue; the producer thread retrieves nodes to be expanded from the request queue. The large language model in the execution backend is invoked to perform operations at the next deeper level. Inferred child nodes The actual generation; if the depth at this time If all main branch nodes have been explored, proceed to step 6; otherwise, return to step 4 and continue exploring other main branch nodes while simultaneously performing the exploration of speculative nodes in parallel. Step 6: Depth at this point All main branch nodes have been explored, and the consumer thread performs inter-layer global reward synchronization and speculative node selection operations: 6-1. The system aggregates the current depth Rewards for all main branch nodes Remove the synchronization barrier and execute the REBASE algorithm to calculate the actual expansion width of each main branch node. Use a hash table Establish a mapping between each main branch node and its expansion requirements, where the key is the main branch node and the value is the number of main branch child nodes to be generated; for each main branch node... , The initial value is ; 6-2. For the speculative child nodes generated by the large language model in step 5, which are executed by the idle producer thread calling the backend. As a result, the actual expansion requirements Cross-comparison is used for hit verification; inferred child nodes are recorded. The parent node is ,like If the prediction is successful, then the predicted child node will be selected. Access to Mind Tree Depth The corresponding position of each layer is added to the mind tree as a child node of the main branch, and at the same time... The corresponding value was modified to ;like If the prediction fails, the prediction child node is immediately terminated. The generation of the node releases the occupied GPU memory and computing resources, and does not add it as a main branch child node to the mind tree; 6-3. The process terminates when the mind tree reaches the system's preset maximum reasoning depth, or when all branch nodes at the current depth generate an end marker indicating that the solution is complete; otherwise, the consumer thread writes the remaining main branch nodes into the request queue, waiting for the producer thread to expand its child nodes, until all nodes are completed. The number of child nodes reached , will the current depth Updated to Proceed to step 4 to explore the next layer of nodes; After the process is terminated, the system traces back along the mind tree to the complete thought path with the highest accumulated reward value, and returns it to the user as the final reasoning solution string for the reasoning query task.

2. The method as described in claim 1, characterized in that, Step 2 calculates the predicted utility score. The calculation formula is: in, Indicates the current inference query Internally available parallel capacity; This indicates the prediction hit rate of the speculative branch; Indicates the size of the parameters in a large language model; Indicates reasoning query The key-value state that can be reused by its inferred branches.

3. The method as described in claim 1, characterized in that, In step 3, the consumer thread will globally estimate the total budget. Distribute to each concurrent inference query, and obtain each inference query The projected budget for actual allocation The calculation formula is: in, Used to control the distribution of sharpness; Indicates rounding down; Indicates the current inference query Internally available parallel capacity; summation index Traverse the set of concurrent reasoning query requests ; This equals the maximum number of producer threads that the query can use simultaneously.

4. The method as described in claim 1, characterized in that, In step 4-3, the actual estimated budget is used. and the set of nodes whose child nodes have been generated Each node in Reward value As input, the REBASE algorithm is used to calculate the inferred expansion width of each node. The calculation formula is: in, Indicates the equilibrium temperature. The value represents the predicted child node to be generated for the corresponding node n. The number of.

5. The method as described in claim 1, characterized in that, In step 6-1, the system aggregates the current depth. Rewards for all main branch nodes The REBASE algorithm is executed to calculate the actual expansion width of each main branch node. The calculation formula is: in, Indicates depth Total sampling budget; Indicates the equilibrium temperature; Indicates depth All nodes.

Images