Data processing method and device of large language model, equipment, storage medium and program product

Abstract

This application provides a data processing method, apparatus, device, storage medium, and program product for a large language model. The method includes: initializing a search tree based on input text and performing multiple search iterations. Each search iteration includes: selecting a target node from the search tree; invoking the large language model; generating at least one child node based on a first reference node set, a second reference node set, and the target node; obtaining the evaluation value of each child node and updating the first and second reference node sets according to the evaluation value of each child node; determining a new target node from the at least one child node based on the evaluation value of each child node, until the new target node is a termination node; updating the search tree according to the search path corresponding to the termination node; and, if the search termination condition is met, determining the response text corresponding to the input text based on the updated search tree. This application can improve the inference speed and accuracy of the large language model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, storage medium, and program product for a large language model. Background Technology

[0002] In related technologies, large language models, when handling complex tasks, generate multiple inference paths by constructing search trees, perform path planning and quality feedback on the generated intermediate inference steps, and thus filter to obtain the final response data. However, the parallel exploration process with multiple branches is usually accompanied by repetitive computational overhead, and the quality evaluation results of intermediate inference steps are prone to bias, thereby affecting the inference speed and accuracy of large language models. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, storage medium, and program product for large language models, which can improve the reasoning speed and accuracy of large language models.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a data processing method for a large language model, the method comprising: The search tree is initialized based on the input text, and multiple search iterations are performed, wherein each search iteration includes: Select a target node from the search tree. If the target node is not a terminal node, call the large language model to generate at least one child node of the target node based on the first reference node set, the second reference node set, and the target node. The first reference node set is used to provide globally shared positive reasoning references, and the second reference node set is used to provide globally shared negative reasoning references. Obtain the evaluation value of each child node, and update the first reference node set and the second reference node set according to the evaluation value of each child node; Based on the evaluation value of each child node, a new target node is determined from the at least one child node until the new target node is a termination node; The search tree is updated according to the search path corresponding to the termination node, and if the search termination condition is met, the response text corresponding to the input text is determined according to the updated search tree.

[0005] This application provides a data processing apparatus for a large language model, including: The data processing module is used to initialize a search tree based on the input text and perform multiple search iterations, wherein each search iteration includes: Select a target node from the search tree. If the target node is not a terminal node, call the large language model to generate at least one child node of the target node based on the first reference node set, the second reference node set, and the target node. The first reference node set is used to provide globally shared positive reasoning references, and the second reference node set is used to provide globally shared negative reasoning references. Obtain the evaluation value of each child node, and update the first reference node set and the second reference node set according to the evaluation value of each child node; Based on the evaluation value of each child node, a new target node is determined from the at least one child node until the new target node is a termination node; The search tree is updated according to the search path corresponding to the termination node, and if the search termination condition is met, the response text corresponding to the input text is determined according to the updated search tree.

[0006] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data processing method for the large language model provided in the embodiments of this application.

[0007] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data processing method for a large language model provided in this application when executed by a processor.

[0008] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the data processing method for a large language model provided in this application.

[0009] The embodiments of this application have the following beneficial effects: By generating child nodes based on a first set of reference nodes providing globally shared positive reasoning references and a second set of reference nodes providing globally shared negative reasoning references during multiple search iterations of the search tree, the large language model can balance positive experience guidance and negative directional constraints when generating nodes, thereby improving the reasoning quality and logical rigor of the newly generated child nodes. Simultaneously, the first and second reference node sets are dynamically updated based on the evaluation values ​​of the child nodes, prompting subsequent new target nodes determined based on the evaluation values ​​to focus more on high-potential search paths. Combining this dynamic evolution mechanism to update the search tree and determine the response text achieves efficient allocation of search computing resources, thereby improving the overall reasoning speed of the large language model and the accuracy of the response text. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the data processing system architecture for a large language model provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 3A This is a first flowchart illustrating the data processing method for a large language model provided in an embodiment of this application; Figure 3B This is a second flowchart illustrating the data processing method for a large language model provided in an embodiment of this application; Figure 3C This is a schematic diagram of the third process of the data processing method for a large language model provided in the embodiments of this application; Figure 4 This is a comparative diagram of the data processing principles provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the first principle of the data processing method for a large language model provided in this application embodiment; Figure 6 This is a schematic diagram of the second principle of the data processing method for large language models provided in the embodiments of this application; Figure 7A This is a schematic diagram of the first experimental results of the data processing method for the large language model provided in the embodiments of this application; Figure 7B This is a schematic diagram of the second experimental results of the data processing method for the large language model provided in the embodiments of this application; Figure 7C This is a schematic diagram of the third experimental result of the data processing method for the large language model provided in the embodiments of this application.

[0011] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0014] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0015] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0016] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0017] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0018] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0019] 1) Large Language Models (LLMs), also known as large models, are large-scale language models designed to understand and generate human language. They are trained on massive amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. The defining characteristic of large language models is their sheer size, containing billions of parameters that help them learn complex patterns in language data. They are typically based on deep learning architectures. Large language models refer to deep learning models trained on vast amounts of text data, containing billions or even more parameters. They can be used to generate natural language text and understand its meaning. Through training, the models learn the statistical regularities and semantic relationships of language to build a vast language knowledge base, thereby simulating human language comprehension and generation capabilities.

[0020] 2) Monte Carlo Tree Search (MCTS) is a heuristic cross-search algorithm used in decision-making and planning processes. It explores the solution space by progressively building a search tree, including four steps: selection, expansion, simulation, and backpropagation.

[0021] 3) The Process Reward Model (PRM) is a model used to assess the quality of intermediate steps in the reasoning process of a large language model (rather than just the final diagnosis or result).

[0022] In related technologies, large language models, when handling complex tasks, generate multiple inference paths by constructing search trees, perform path planning and quality feedback on the generated intermediate inference steps, and thus filter to obtain the final response data. However, the parallel exploration process with multiple branches is usually accompanied by repetitive computational overhead, and the quality evaluation results of intermediate inference steps are prone to bias, thereby affecting the inference speed and accuracy of large language models.

[0023] This application provides a data processing method, apparatus, device, computer-readable storage medium, and computer program product for large language models, which can improve the inference speed and accuracy of large language models. The following describes exemplary applications of the electronic devices provided in this application. The electronic devices provided in this application can be implemented as various types of terminal devices such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and vehicle terminals, or as servers.

[0024] See Figure 1 , Figure 1This is a schematic diagram of the data processing system architecture for a large language model provided in an embodiment of this application. Figure 1 The system involves server 100, terminal device 200, and network 300. Terminal device 200 is connected to server 100 through network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0025] In some embodiments, server 100 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal devices and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0026] In some embodiments, the terminal device 200 sends the input text to the server 100. The server 100 receives the input text, generates a response text using the large language model data processing method provided in this application embodiment, and sends the response text to the terminal device 200.

[0027] The data processing method for large language models provided in this application can be used in data processing scenarios, as illustrated below: 1) In a code analysis scenario, for example, a software development engineer inputs a piece of code and a prompt text that "requires locating the root cause of the exception and outputting the refactored code" in the "deep code review" function of an intelligent programming assistance tool; the terminal device sends the input text to the server; the server generates a response text including an analysis report and refactored code through the data processing method of the large language model provided in the embodiments of this application, and sends the response text to the terminal device.

[0028] 2) In a mathematical problem reasoning scenario, for example, a student inputs a complex math competition problem text and a prompt text that "a rigorous logical formula derivation and final answer are required" in the "step-by-step analysis of difficult problems" function of an intelligent education and training software; the terminal device sends the input text to the server; the server generates a response text including the distributed derivation process and the final solution through the data processing method of the large language model provided in the embodiments of this application, and sends the response text to the terminal device.

[0029] 3) In a scientific knowledge Q&A scenario, for example, a researcher enters a complex interdisciplinary question in the field of biology or physics into the "Professional Theory Analysis" function of an academic tutoring platform, along with a prompt text that "logical deduction and analysis based on professional theory is required"; the terminal device sends the input text to the server; the server generates a response text including the theoretical deduction context and professional answer through the data processing method of the large language model provided in the embodiments of this application, and sends the response text to the terminal device.

[0030] 4) Low-resource business review scenarios, for example, a business review specialist in the "dynamic rule reasoning" function of the intelligent operation platform inputs a complex business data text to be reviewed, a structured business rule document, and a prompt text "perform multi-step reasoning based on the input business rules and output the review conclusion" for a newly launched cold start business that lacks a large amount of labeled data; the terminal device sends the input text to the server; the server uses the data processing method of the large language model provided in the embodiments of this application to generate a reply text including the rule determination reasoning path and the final review result, and sends the reply text to the terminal device.

[0031] 5) Content generation scenarios on content creation and operation platforms: For example, a self-media operations manager inputs a set of industry hot keywords, high-conversion articles from the platform's past (as a positive logical reference), and pre-set content to be avoided (as a negative reference) into the "topic planning" function of the content creation and operation platform, along with a prompt text that "requires combining current hot topics to deduce a series of creative topic selection schemes that conform to the platform's traffic distribution logic." The terminal device sends the input text to the server; the server, through the large language model data processing method provided in this application embodiment, generates a response text including a multi-step creative derivation chain and a recommended topic selection plan, and sends the response text to the terminal device. This assists operations personnel in identifying the content strategies with the greatest traffic potential through large-scale logical searches.

[0032] In other embodiments, the large language model data processing method provided in this application can be implemented by a terminal device or a server independently. The terminal device 200 or the server 100 generates response text corresponding to the input text using the large language model data processing method provided in this application.

[0033] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2The illustrated electronic device 400 includes at least one processor 410, a memory 430, and at least one network interface 420. The various components of the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0034] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0035] The memory 430 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 430 may optionally include one or more storage devices physically located away from the processor 410.

[0036] The memory 430 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 430 described in this application embodiment is intended to include any suitable type of memory.

[0037] In some embodiments, memory 430 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0038] Operating system 431 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 432 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2A data processing device 433 for a large language model stored in memory 430 is shown. This device may be software in the form of programs and plug-ins, and includes a data processing module 4331. The function of the module will be described below.

[0039] In some embodiments, the terminal device or server can implement the large language model data processing method provided in this application embodiment by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0040] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method for the large language model provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0041] The following will describe the data processing method for large language models provided in this application embodiment, using the server as the execution subject, with exemplary applications and implementations of the server provided in the embodiments of this application. See also Figure 3A , Figure 3A This is a first flowchart illustrating the data processing method for a large language model provided in this application embodiment, which will be combined with... Figure 3A The steps shown are explained.

[0042] In step 101, a search tree is initialized based on the input text, and multiple search iterations are performed, wherein each search iteration performs the following steps 201 to 204: In step 201, a target node is selected from the search tree. If the target node is not the terminal node, the large language model is invoked to generate at least one child node of the target node based on the first reference node set, the second reference node set, and the target node. The first reference node set is used to provide globally shared positive inference references, and the second reference node set is used to provide globally shared negative inference references.

[0043] In some embodiments, initializing the search tree based on the input text can be achieved as follows: create a tree-like data structure as the search tree and set the initial depth to zero; create an initial node (root node) in the search tree, using the character sequence or token sequence of the input text as the text content corresponding to the root node; configure attribute parameters (including historical access count and evaluation value) for the root node for node selection and quality evaluation, that is, initialize the historical access count of the root node to zero to indicate that the root node has not been explored, and initialize the evaluation value of the root node to a preset initial value (e.g., zero); finally, set the root node as the starting point of the search tree, thereby completing the initialization of the search tree.

[0044] Here, a search tree refers to a data topology used to structurally represent the state space of a large language model in a multi-step reasoning process. It consists of nodes representing the reasoning state and edges representing the state transition process.

[0045] Here, the root node refers to the baseline node in the initial layer of the logical hierarchy of the search tree, which carries the original input information (such as the input text mentioned above) that has not been expanded by any reasoning steps in the large language model.

[0046] Here, input text refers to the data input into the large language model to trigger inference or generation tasks, such as user-generated questions, contextual prompts, constraint descriptions, or code snippets to be analyzed and processed.

[0047] For example, taking the task of solving a mathematical logic word problem as an example, suppose the input text is "Request to solve the linear equation in one variable: 2X + 4 = 10". The top of the search tree is the root node containing the above input text. The root node will derive one or more edges representing the state transition process, and each edge will connect to a child node representing the result of subsequent single-step reasoning. For example, the first edge from the root node connects to the first child node, and the text content (reasoning text) stored in the first child node is "First, combine the constant terms on both sides of the equation to get 2X = 6", while also maintaining the historical access count and evaluation value of the first child node; parallel to this, the second edge from the root node may connect to the second child node, and the text content (reasoning text) stored in the second child node is "First, divide both sides of the equation by 2 to get X + 2 = 5", while also maintaining the historical access count and evaluation value of the second child node.

[0048] As the multi-step reasoning process deepens and expands, the aforementioned child nodes can further branch downwards, leading to new edges and connecting to deeper nodes that carry textual content such as "the final answer X=3 has been calculated," until reaching the terminal node where no further deduction is needed. In this data topology, each node records the intermediate step text derived by the large language model at a certain stage of thinking, along with the corresponding evaluation value; while the edges connecting the nodes represent the evolutionary trajectory of the complex logic as it progresses forward or shifts towards exploration.

[0049] In some embodiments, selecting a target node from the search tree can be achieved as follows: taking the root node of the search tree as the starting point of traversal, the currently traversed node is taken as a candidate node, where the text content corresponding to the root node is the input text; if the candidate node is a leaf node, then the candidate node is determined as the target node; if the candidate node is not a leaf node, then a new candidate node is determined from the set of child nodes corresponding to the candidate node, and the target node selection is continued based on the new candidate node until the new candidate node is a leaf node, and the new candidate node is determined as the target node.

[0050] For example, assume the current search tree depth is 3 (i.e., it has expanded downwards from the root node to form a three-level logical child node connection structure). In the initial stage of this search iteration, the root node is set as the initial candidate node. Since there are further branches extending from the root node (i.e., the root node is not a leaf node), assume that the set of child nodes corresponding to the root node contains child node A and child node B. Calculate the selection values ​​for child node A and child node B respectively. If the calculation shows that the selection value of child node A is higher than that of child node B, then proceed to child node A and make it a new candidate node. Further detection reveals that child node A is also not a leaf node, and the set of child nodes corresponding to child node A still contains further extended child node C. After comparing the selection values ​​of the branches under child node A, proceed to child node C and update child node C as a new candidate node. Further detection reveals that child node C currently has no attached child nodes, so child node C is determined to be a leaf node. Finally, stop traversing downwards and determine child node C as the target node required for this search iteration.

[0051] In some embodiments, determining a new candidate node from the set of child nodes corresponding to a candidate node can be achieved as follows: obtaining the historical access count of the candidate node; for each child node in the set of child nodes corresponding to the candidate node, obtaining the historical access count of the child node and the evaluation value of the child node; for each child node in the set of child nodes corresponding to the candidate node, performing a nonlinear transformation based on the historical access count of the candidate node and the historical access count of the child node to obtain the exploration weight of the child node, wherein the exploration weight is positively correlated with the historical access count of the candidate node and negatively correlated with the historical access count of the child node; for each child node in the set of child nodes corresponding to the candidate node, obtaining the product of a preset exploration constant and the exploration weight of the child node, and using the sum of the product and the evaluation value of the child node as the selection value of the child node; selecting the child node with the largest selection value from the set of child nodes corresponding to the candidate node and determining it as a new candidate node.

[0052] Here, the selection value of a child node refers to the numerical index obtained by summing the evaluation value of the child node with the exploration gain component derived from the number of historical visits (i.e., the product of the preset exploration constant and the exploration weight of the child node). It is used to characterize the priority of a child node being selected as the direction of the next stage of deduction extension after comprehensively considering the potential for in-depth utilization of known high-quality reasoning logic and the need for extensive exploration of unexplored logical paths in the hierarchical traversal operation of the search tree.

[0053] In some embodiments, the historical access count of candidate nodes is obtained; for each child node in the set of child nodes corresponding to a candidate node, the historical access count of the child node and the evaluation value of the child node are obtained. This can be achieved by retrieving the historical access count of the candidate node, the historical access count of its child nodes, and the evaluation value of the child nodes from the node attribute metadata storing the search tree structure. For example, for the currently polled candidate node, the access statistics count value corresponding to the candidate node is retrieved and confirmed as the historical access count of the candidate node; for each child node in the set of child nodes to which the candidate node belongs, the access statistics count value and quality score value corresponding to the child node are retrieved and confirmed as the historical access count and evaluation value of the child node, respectively.

[0054] It should be noted that the historical access counts and evaluation values ​​mentioned above are dynamically updated through previous search iterations, which can be achieved through backpropagation. For an explanation of how to dynamically update the attribute metadata of each node in the search tree through multiple search iterations, please refer to step 204 below (updating the search tree based on the search path corresponding to the terminating node).

[0055] In some embodiments, the exploration weight of a child node is obtained by performing a nonlinear transformation based on the historical visit counts of the candidate node and the historical visit counts of its child nodes. This can be achieved as follows: First, the natural logarithm of the historical visit counts of the candidate node is performed to smooth the numerical characteristics of the visit frequency; then, the ratio of the result of the natural logarithm operation to the historical visit counts of the child node is calculated; finally, the arithmetic square root of the obtained ratio is performed to obtain the calculation result, which is then confirmed as the exploration weight of the child node. In this process, since the visit counts of the candidate node are in the logarithmic term of the numerator, while the visit counts of the child node are in the denominator, the exploration weight of the child node is positively correlated with the historical visit counts of the candidate node and negatively correlated with the historical visit counts of the child node.

[0056] For example, suppose a candidate node has been visited 100 times. Of the candidate node's two child nodes, A and B, child node A has been visited 5 times, and child node B has been visited 50 times. When calculating the exploration weight, because the historical visit count of child node A is smaller as a denominator, the square root of the resulting ratio will be larger. Therefore, the exploration weight for child node A will be higher than that for child node B. This non-linear transformation can give higher compensation weights to nodes with lower visit frequencies, thus balancing the exploration intensity of low-frequency nodes during the search process.

[0057] In some embodiments, obtaining the product of a preset exploration constant and the exploration weight of a child node, and summing the product with the evaluation value of the child node as the selection value of the child node, can be achieved as follows: The exploration constant is read from a preset configuration file; the product of the exploration constant and the established exploration weight is calculated to obtain an exploration gain component used to adjust the search tendency; subsequently, the exploration gain component is added to the read evaluation value of the child node to obtain the selection value of the child node.

[0058] For example, determining a new candidate node from the set of child nodes corresponding to a candidate node can be expressed by formula (1): (1) in, Indicates candidate nodes Any child node in the corresponding set of child nodes; Indicates candidate nodes The set of child nodes; Represents child nodes The evaluation value; Represents child nodes obtained based on nonlinear transformation The exploration weight; This represents the preset exploration constant; Represents child nodes The selected value; Represents the set of child nodes The operation is used to find the child node that maximizes the selected value. This indicates the historical number of visits to the candidate node; Represents child nodes Historical visit count; It represents logarithmic operations.

[0059] By integrating the evaluation value representing the quality of known reasoning logic with the exploration weight derived from a nonlinear transformation based on the number of historical visits, a selection value for determining the branch direction is constructed. This effectively balances the dual requirements of "deep utilization" and "breadth exploration" in the multi-step deduction process. Since the nonlinear transformation mechanism gives higher compensation weights to low-frequency access nodes, it prevents the traversal process of the search tree from getting trapped in local suboptimal paths too early. While ensuring that high-confidence, high-quality reasoning logic is prioritized for expansion, it also provides ample room for exploring unknown logic branches, thereby significantly improving the global optimization capability and node search efficiency of large language models in complex solution spaces.

[0060] In some embodiments, see Figure 3BBased on the first reference node set, the second reference node set, and the target node, at least one child node of the target node is generated. This can be achieved through the following steps 301 to 305, which are explained in detail below.

[0061] Here, the first reference node set (e.g., denoted as) The first reference node set refers to the set of node information determined to have positive reference value during the reasoning process. Positive reference value refers to the logical attribute where the evaluation value associated with the reasoning text carried by the corresponding node satisfies a logical attribute greater than or equal to a first threshold. The first reference node set is used to provide globally shared positive reasoning references for subsequent reasoning in the large language model, and drives the large language model to reuse corresponding high-quality reasoning logic in subsequent branch expansion processing.

[0062] Here, the second reference node set (e.g., denoted as) The second reference node set refers to the set of node information determined to have negative reference value during the reasoning process. Negative reference value refers to the logical attribute where the evaluation value associated with the reasoning text carried by the corresponding node is less than or equal to a second threshold (the second threshold is less than the first threshold). This second reference node set provides globally shared negative reasoning references for subsequent reasoning in the large language model and constrains the large language model to avoid corresponding logical blind spots or reasoning paths in subsequent branch expansion processing, thereby achieving proactive search space pruning.

[0063] In step 301, the inference text context corresponding to the target node is obtained, wherein the inference text context includes the inference text corresponding to each node in the search path from the root node to the target node in the search tree.

[0064] In some embodiments, the reasoning text context corresponding to the target node can be obtained in the following ways: extract the topological structure information of the search tree, determine the search path starting from the root node of the search tree and extending to the currently selected target node; extract the reasoning text carried by each node along the path in sequence according to the hierarchical progression order of the search path; finally, combine all the extracted reasoning text according to the order represented by the search path, and confirm the combined text fragment as the reasoning text context corresponding to the target node.

[0065] For example, suppose the current search path sequentially passes through the root node (input text is "Question: How to improve webpage loading speed"), the first-level intermediate node (inference text is "Step 1: Analyze network request time"), and arrives at the selected target node (inference text is "Step 2: Compress image resources"). By extracting and concatenating the inference text context corresponding to the target node in sequence, the final result is the coherent string sequence "Question: How to improve webpage loading speed; Step 1: Analyze network request time; Step 2: Compress image resources".

[0066] In step 302, the inference text corresponding to the nodes in the first reference node set is obtained as positive reference text, and the inference text corresponding to the nodes in the second reference node set is obtained as negative reference text.

[0067] In some embodiments, obtaining the inference text corresponding to the nodes in the first reference node set as positive reference text can be achieved in the following way: for each node stored in the first reference node set, extract the inference text corresponding to the node, and combine the inference text corresponding to each node into positive reference text.

[0068] For example, in order to enable large language models to clearly identify the type of reference information, preset positive guidance markers (such as "The following is the correct reasoning path:") are added before and after the extracted reasoning text. The reasoning text with positive guidance markers is then aggregated and concatenated into a complete character sequence as positive reference text.

[0069] For example, if the first reference node set The text includes two nodes, corresponding to the inference texts "using efficient quicksort to process large-scale data" and "achieving constant-level query efficiency through hash tables". The resulting positive reference text is "Positive reference 1: using efficient quicksort to process large-scale data; positive reference 2: achieving constant-level query efficiency through hash tables".

[0070] In some embodiments, obtaining the inference text corresponding to the nodes in the second reference node set as negative reference text can be achieved in the following way: for each node stored in the second reference node set, extract the inference text corresponding to the node, and combine each extracted inference text to obtain negative reference text.

[0071] For example, in order for a large language model to accurately identify and avoid erroneous reasoning paths, a preset negative guidance marker (such as "The following is an incorrect reasoning path:") can be added before and after each extracted reasoning text involving logical fallacies or inefficient reasoning. The reasoning text with the negative guidance marker is then concatenated according to preset connectors to form a complete character sequence, which serves as negative reference text.

[0072] For example, if the second reference node set The text includes two nodes, corresponding to the inference text "intensive disk I / O reads inside the loop" and "constructing SQL statements directly using raw strings input by the user". The resulting negative reference text is "Negative reference 1: Intensive disk I / O reads inside the loop; Negative reference 2: Constructing SQL statements directly using raw strings input by the user".

[0073] In step 303, the prompt text is determined based on the inference text context, positive reference text, and negative reference text.

[0074] Here, prompt words refer to the content that is input into the large language model and explicitly specifies the direction, scope, format, and logical constraints of the output of the large language model through natural language instructions. This is used to guide the large language model to generate text information that meets the expected goals in multi-step reasoning.

[0075] In some embodiments, determining the prompt text based on the inference text context, positive reference text, and negative reference text can be achieved in the following way: obtaining a preset prompt text combination template; filling the background description information field of the prompt text combination template with the inference text context corresponding to the target node; filling the positive guidance information field of the prompt text combination template with the positive reference text and attaching heuristic guidance instructions; filling the negative constraint information field of the prompt text combination template with the negative reference text and attaching avoidance control instructions; and finally, performing overall serialization processing on the prompted text combination template after filling in the content.

[0076] For example, the identified prompt text could be expressed as: "The established reasoning context is as follows: [Reasoning text context]. In the following single-step reasoning extension, please learn and reuse the following correct logic: [Positive reference text]; at the same time, you must avoid the following incorrect logic: [Negative reference text]. Please output at least one subsequent reasoning step based on the above requirements." In step 304, the prompt text is input into the large language model to obtain at least one candidate inference text output by the large language model. The positive reference text in the prompt text is used to guide the large language model to reuse the inference logic corresponding to the positive reference text, and the negative reference text is used to constrain the large language model to avoid the inference logic corresponding to the negative reference text.

[0077] In some embodiments, inputting the prompt text into a large language model to obtain at least one candidate inference text output by the large language model can be achieved in the following way: sending the generated prompt text to the inference computing unit where the large language model is located via a network interface, and configuring the corresponding generation hyperparameters (such as the number of sampling paths (i.e., the number of candidate inference texts), inference temperature, etc.); driving the large language model to respond to the logical guidance in the prompt text, and sampling at least one inference fragment from the lexical probability distribution as at least one candidate inference text.

[0078] For example, a large language model responds to logical guidance in prompt text and samples at least one inference fragment from the lexical probability distribution. This can be achieved as follows: extract semantic features from the prompt text and convert it into a corresponding sequence of lexical vectors; use the attention mechanism network within the large language model to fuse features of the lexical vector sequence to obtain an initial probability distribution for predicting subsequent text; during the calculation and prediction of the probability distribution, use the attention weights of positive reference text in the prompt text to increase the generation probability of lexical elements related to positive inference logic; use the attention weights of negative reference text in the prompt text to reduce or even block the generation probability of lexical elements related to negative inference logic; after the above probability correction, based on the configured number of sampling paths and inference temperature, use nucleus sampling or beam search strategies to sample the corrected probability distribution word by word until a preset end symbol is generated, thereby obtaining at least one inference fragment.

[0079] In step 305, for each candidate inference text, the candidate inference text is used as the inference text corresponding to the new node, and the new node is used as a child node of the target node in the search tree.

[0080] In some embodiments, for each candidate inference text, the candidate inference text is used as the inference text corresponding to the new node, and the new node is used as a child node of the target node in the search tree. This can be achieved as follows: construct a corresponding node object (i.e., a new node) for each candidate inference text; write the content of the candidate inference text into the inference text field corresponding to the new node; finally, in the logical hierarchy of the search tree, establish a directed edge from the current target node to the new node, thereby establishing the new node as a subordinate child node of the target node at the topological level, completing one branch growth of the search tree.

[0081] For example, based on the first reference node set, the second reference node set, and the target node, generating at least one child node of the target node can be represented by formula (2): (2) in, Indicates the current target node; Indicates the corresponding target node A collection of one or more child nodes; This represents the second set of reference nodes that serves as a globally shared negative reference benchmark. This represents the first set of reference nodes that serves as a globally shared positive reference benchmark. This indicates a processing operator that invokes a large language model to perform branch expansion generation (generating at least one child node of the target node).

[0082] By fusing the context of reasoning text representing known deduction progress with globally shared positive and negative reference texts to construct cue word text, successful experiences and lessons learned from failures across branches in multi-step reasoning can be explicitly injected into the generation intervention of the large language model. When driving node hierarchical expansion, positive reference text can effectively increase the sampling probability of corresponding high-quality logical units to facilitate experience reuse, while negative reference text can precisely suppress the sampling probability of erroneous directional units to implement proactive pre-pruning. This directly blocks the repeated exploration of verified invalid search paths at the source of unit generation, improves the overall logical quality of new nodes and the downward convergence efficiency of complex topology search trees.

[0083] See also Figure 3A In step 202, the evaluation value of each child node is obtained, and the first reference node set and the second reference node set are updated according to the evaluation value of each child node.

[0084] In some embodiments, the evaluation value of each child node can be obtained as follows: for each child node, the inference text context is combined with the inference text corresponding to the child node to obtain the evaluation text corresponding to the child node; for each child node, the evaluation text corresponding to the child node is feature-encoded using a pre-trained evaluation model to obtain the evaluation text features; the evaluation text features are feature-mapped to obtain the quality category corresponding to the child node; and the evaluation value is determined based on the quality category, wherein the evaluation value is used to characterize the inference logic quality of the inference text corresponding to the child node, and the inference progress of the inference path constituted by the evaluation text.

[0085] Here, the reasoning logic quality of the reasoning text refers to the correctness, coherence, and effectiveness of the single-step derivation content (reasoning text) carried by the child nodes at the technical level.

[0086] Here, evaluating the reasoning progress of the reasoning path constituted by the text refers to the degree to which the local deduction state, composed of the current child node and the preceding search path, is in the complete multi-step logical chain leading to the final correct answer.

[0087] It should be noted that the evaluation value is able to characterize the reasoning logic quality of the reasoning text corresponding to the child node, as well as the reasoning progress of the reasoning path constituted by the evaluation text, due to the combined effect of the training data construction mechanism and the two-stage training strategy of the pre-trained evaluation model during the training phase. Specifically, in the process of constructing the training dataset, the label values ​​determined for the node samples have already deeply integrated the indication information used to indicate whether the derivation direction is correct, and the reasoning distance used to quantify the depth of the current node from the level of the final correct answer; subsequently, through the first stage of preference alignment training and the second stage of fine-grained quality classification training, the dual supervision signals about logical quality and derivation progress contained in the above label values ​​are successfully transferred and internalized into the network weights of the evaluation model. Therefore, the optimized evaluation model can accurately output classification probabilities reflecting the logical quality and task achievement degree for the input text features, thereby determining the evaluation value with the above-mentioned characterization ability. The specific training process of the evaluation model (including the construction of the training dataset and the two-stage parameter update logic) will be described in the relevant embodiments of steps 401 to 403 below.

[0088] For example, for each child node, the inference text context is combined with the inference text corresponding to the child node to obtain the evaluation text corresponding to the child node. This can be achieved by performing the following processing for each child node: extract the complete search path from the root node, through the intermediate level nodes, and finally to the current child node from the topology of the search tree; sequentially obtain the inference text carried by each node in the search path, and concatenate and combine the obtained inference texts according to the path order to obtain the evaluation text corresponding to the child node.

[0089] For example, for each child node, the evaluation text corresponding to the child node is feature-encoded using a pre-trained evaluation model to obtain evaluation text features. This can be achieved by performing the following processing for each child node: converting the evaluation text corresponding to the child node into a sequence of word vectors and inputting it into the pre-trained evaluation model; using the multi-layer self-attention network in the evaluation model, calculating the association weights between each word in the word vector sequence, thereby extracting a high-dimensional vector that can represent the semantic features and logical integrity of the evaluation text, and determining it as the evaluation text feature.

[0090] For example, feature mapping of evaluation text features to obtain the quality category corresponding to the child node can be achieved by performing the following processing for each child node: inputting the obtained evaluation text features into the multilayer perceptron (MLP) or linear mapping layer at the end of the evaluation model to perform linear transformation processing on the evaluation text features; then, calculating the probability distribution of the evaluation text features on multiple preset quality levels through a classification function (such as the Softmax function), and determining the quality level corresponding to the highest probability value in the probability distribution as the quality category corresponding to the child node.

[0091] For example, the quality categories can be pre-divided into five discrete quality levels: Level 1 (indicating logical correctness and achievement of the final goal), Level 2 (indicating logical correctness and progress in the intermediate stage), Level 3 (indicating logical redundancy but correct direction), Level 4 (indicating minor logical flaws), and Level 5 (indicating serious reasoning errors or hallucinations).

[0092] For example, determining the evaluation value based on quality categories can be achieved by performing the following process for each child node: Obtain the preset mapping relationship between quality categories and score weights. For the determined quality category of the child node, find the corresponding preset score in the mapping relationship. Since each quality category is pre-associated with fixed values ​​representing the quality of reasoning logic and reasoning progress, the found preset score is directly confirmed as the evaluation value of the child node.

[0093] For example, in the mapping relationship between quality categories and score weights, it is predefined that: the first level is mapped to 1.0 points, the second level to 0.7 points, the third level to 0.4 points, the fourth level to 0.1 points, and the fifth level to 0.0 points. If the quality category of the output child node of the current evaluation model is the first level, then the corresponding evaluation value is directly determined to be 1.0; if the quality category of the output child node is the fifth level, then the corresponding evaluation value is determined to be 0.0.

[0094] In some embodiments, updating the first reference node set and the second reference node set according to the evaluation value of each child node can be achieved by performing the following processing for each child node: if the evaluation value of the child node is greater than or equal to a preset first threshold, then the child node is added to the first reference node set; if the evaluation value of the child node is less than or equal to a preset second threshold, then the child node is added to the second reference node set, wherein the first threshold is greater than the second threshold.

[0095] For example, suppose the evaluation value of the child node is labeled as The first threshold is set as the positive recognition threshold (for example, denoted as). The second threshold is set as a negative recognition threshold (for example, denoted as ). ).

[0096] If the evaluation model outputs the evaluation value of the child nodes Greater than or equal to the positive recognition threshold (For example, setting) If the value is 0.9, then the processing steps carried by the corresponding child node are considered valuable. A storage action is then triggered, directly storing the corresponding child node into the first reference node set.

[0097] If the evaluation model outputs the evaluation value of the child nodes Less than or equal to the negative recognition threshold (For example, setting) If the value is 0.2, then the processing steps carried by the corresponding child node are deemed not to be of reference value. At this time, a storage action is triggered, and the child node is stored in the second reference node set.

[0098] By introducing an evaluation model optimized based on a two-stage training strategy, the features of the reasoning text are mapped to discrete quality categories and anchored to evaluation values. This effectively enhances the generalization robustness of the evaluation process in a few-sample reasoning environment, ensuring that the output scoring signal can accurately and stably represent the logical quality of single-step reasoning and the closed-loop degree of global reasoning progress. Combined with a dynamic recognition mechanism with dual thresholds at the beginning and end, insightful reasoning experiences and typical lessons with logical fallacies can be diverted in real time and merged into the globally shared first and second reference node sets. While ensuring the typicality of the stored information, this lays a high-fidelity knowledge foundation for global heuristic guidance and automated path pruning in the subsequent generation process. This significantly reduces the computational redundancy of large language models in multi-step reasoning and improves the convergence efficiency and accuracy of complex logical reasoning.

[0099] In other embodiments, the following processing may also be performed: obtaining the inference text features of the inference text corresponding to each node in the first reference node set, and obtaining the inference text features of the inference text corresponding to each node in the second reference node set; for the first reference node set, calculating the first similarity of the inference text features between any two nodes in the first reference node set, and deleting any one of any two nodes whose first similarity is greater than or equal to a preset third threshold; for the second reference node set, calculating the second similarity of the inference text features between any two nodes in the second reference node set, and deleting any one of any two nodes whose second similarity is greater than or equal to the third threshold.

[0100] For example, obtaining the inferred text features of the inferred text corresponding to each node in the first reference node set, and obtaining the inferred text features of the inferred text corresponding to each node in the second reference node set, can be achieved by calling a pre-built text embedding model. The inferred text corresponding to a node is input into the text embedding model, which outputs a dense vector of fixed dimensions as the corresponding inferred text features. Subsequently, the distance coefficient between pairwise dense vectors is calculated using dot product or cosine similarity algorithms, serving as the corresponding first or second similarity.

[0101] For example, assuming the third threshold is set to 0.85, and the first reference node set already contains nodes A and B. The inference text for node A is "Apply the quicksort algorithm in descending order," and the inference text for node B is "Use the quicksort algorithm to sort the array in descending order." After extracting vector features, the cosine similarity between the inference text features of node A and node B is calculated to be 0.92, which is the determined first similarity of 0.92. Since 0.92 is greater than the preset 0.85 (third threshold), either node A or node B is removed from the first reference node set.

[0102] The dynamic deduplication mechanism based on threshold comparison described above ensures that the positive and negative reference texts remaining in the global cache maintain a high degree of feature diversity, enabling the large language model to receive sufficiently differentiated guidance and constraint information in subsequent branch expansions.

[0103] In step 203, based on the evaluation value of each child node, a new target node is determined from at least one child node until the new target node is the termination node.

[0104] In some embodiments, determining a new target node from at least one child node based on the evaluation value of each child node, until the new target node is a termination node, can be achieved as follows: selecting the child node with the largest evaluation value from at least one child node of the target node as the new target node; if the new target node is not a termination node and is not a fully expanded node, performing an expansion process on the new target node to generate at least one child node corresponding to the new target node; for each child node corresponding to the new target node, obtaining the evaluation value of the child node, and selecting the child node with the largest evaluation value from at least one child node corresponding to the new target node as the subsequent new target node, until the new target node is a termination node.

[0105] Here, a terminal node refers to a node that is at the end of a multi-step reasoning path in the logical hierarchy of the search tree, and whose corresponding reasoning text includes the final solution result, task completion status, or a node whose depth in the search tree has reached the preset maximum number of reasoning steps limit.

[0106] For example, the termination node can be determined in the following way: perform key character matching or semantic feature parsing on the reasoning text corresponding to the determined new target node; if the parsed reasoning text includes "the final conclusion is", "the answer is", or a predefined termination word (e.g., "...") due to the end of generation, the termination node can be determined by: performing key character matching or semantic feature parsing on the reasoning text corresponding to the determined new target node; if the parsed reasoning text includes "the final conclusion is", "the answer is", or a predefined termination word (e.g., "...") <end>"or" <eos>If a new target node is detected to be at a depth that has reached a preset depth threshold, or if the target node is identified as a terminal node, then the current target node is directly determined to be the terminal node.

[0107] Here, a fully expanded node refers to a node whose configuration status of its subordinate child nodes has reached the preset maximum node count threshold. In subsequent simulations, for fully expanded nodes, the large language model will no longer derive new child nodes for the fully expanded node; instead, it will directly select subsequent nodes from its existing set of child nodes according to logic.

[0108] Here, the extended processing refers to combining the reasoning text of the target node with the preceding path to form a context input to the large language model, and combining the globally shared positive reasoning reference and negative reasoning reference to drive the large language model to output at least one new candidate reasoning text, thereby creating a corresponding new node in the search tree (such as the processing in steps 301 to 305).

[0109] For example, suppose the currently selected target node, after expansion, generates three subordinate branches (i.e., three child nodes): child node A (with an evaluation value of 0.3), child node B (with an evaluation value of 0.9), and child node C (with an evaluation value of 0.6). During path extension, child node B with the highest evaluation value is selected from these three child nodes and established as the new target node. If child node B has not yet output the final answer (i.e., it has the attribute of a non-terminating node) and child node B has not exhausted the preset expansion width (i.e., it is an incompletely expanded node), then a new round of expansion processing is performed starting from child node B. Suppose that after expansion, child nodes B1 (with an evaluation value of 0.4) and B2 (with an evaluation value of 0.85) belonging to child node B are generated. Subsequently, child node B2 with the highest evaluation value (0.85) is selected from child nodes B1 and B2 and established as the new target node. If the inference text of child node B2 contains an end marker indicating task completion (e.g., "final calculation result is 100"), then child node B2 is determined to be a termination node. At this point, the downward greedy expansion and selection operations stop, and the target node advancement process of the current stage ends.

[0110] For example, determining a new target node from at least one child node based on the evaluation value of each child node can be expressed by formula (3): (3) in, This represents the set of all child nodes directly connected to and extended from the current target node; Represents the target node The corresponding set of child nodes child nodes in; Represents child nodes The evaluation value; This represents the operation of finding the child node from the set of child nodes that maximizes the evaluated value.

[0111] By consistently selecting the child node with the largest evaluation value as the new target node during the deduction process, and dynamically triggering expansion processing based on the node state, a greedy optimization strategy can be implemented in the depth traversal of the search tree. This ensures that the inference computing power of the large language model is always focused on the high-confidence logical path with the greatest potential for success under the current branch, avoiding excessive consumption of computing resources on low-value or suboptimal steps that deviate from the task objective. As a result, while significantly accelerating the convergence speed of a single search iteration to reach the termination node, the execution efficiency of constructing and locking the optimal search path in the entire multi-step deduction process is significantly improved.

[0112] In step 204, the search tree is updated according to the search path corresponding to the termination node, and if the search termination condition is met, the response text corresponding to the input text is determined according to the updated search tree.

[0113] In some embodiments, updating the search tree based on the search path corresponding to the termination node can be achieved in the following ways: determining the search path from the root node to the termination node in the current search iteration, wherein the text content corresponding to the root node is the input text; updating the historical visit count of each node along the search path for each node in the search path; and updating the evaluation value of each node along the search path for each node in the search path.

[0114] In some embodiments, the search path from the root node to the termination node in the current search iteration can be determined as follows: Starting from the termination node, using the parent node pointers established in the search tree, backtrack to visit the direct parent node of the current node, and store the nodes passed on the backtracking path into a temporary path stack until the root node of the search tree is reached; then, reverse the order of the nodes in the temporary path stack to establish a sequence of nodes starting from the root node, passing through the intermediate nodes generated by each level of expansion, and ending with the termination node, as the search path corresponding to the current search iteration.

[0115] For example, suppose the current search iteration finally reaches the termination condition at child node C (the termination node). By backtracking, we find that the parent node of child node C is child node A, and the parent node of child node A is the root node. Then the determined search path is "root node → child node A → child node C".

[0116] In some embodiments, updating the historical access count of each node along the search path can be achieved by performing a traversal operation on the metadata of each node in the search path, reading the value corresponding to the original historical access count from the node storage space, and incrementing the original historical access count by one, thereby completing the real-time count update of the node's exploration frequency.

[0117] For example, if child node A had 5 visits before this iteration, its historical visit count will be updated to 6 after this search path update.

[0118] In some embodiments, updating the evaluation value of each node in the search path can be achieved by: obtaining the evaluation value corresponding to the terminating node in the current search iteration as a reward value; traversing the nodes in the search path as nodes to be updated in the order from the parent node of the terminating node to the root node along the search path; and for the currently traversed node to be updated, determining the updated evaluation value corresponding to the node to be updated based on the node's historical access count before the update, its evaluation value before the update, its reward value, and its historical access count after the update.

[0119] For example, the updated evaluation value can be calculated using an incremental averaging formula, which involves multiplying the evaluation value of the node to be updated before the update by the number of historical visits before the update to obtain the cumulative contribution value; summing the cumulative contribution value with the reward value determined this time to obtain the total contribution value; and finally, dividing the total contribution value by the number of historical visits of the node to be updated after the update, and using the average value obtained, which is determined as the updated evaluation value of the node to be updated.

[0120] For example, assuming a reward value of 1.0 (representing a completely correct deduction path), the evaluation value of the node to be updated (such as child node A) before the update is 0.6, and the number of historical visits before the update is 10. Then, the evaluation value of child node A after the update = (0.6 × 10 + 1.0) / 11 ≈ 0.636. In this way, the logical quality information fed back by the terminating node can propagate backward along the search path and permeate into nodes at each level, thereby dynamically optimizing the evaluation distribution of the search tree.

[0121] By executing a backpropagation mechanism along the exploration path, the reward value (i.e., the final success or failure feedback of the path or the completeness of the task) with global guidance contained in the termination node can be passed up level by level and permeated to all the preceding intermediate nodes at the starting end. In addition, the evaluation value data of the nodes is updated incrementally and dynamically by combining the historical access count. This mechanism completely opens up the supervision and evaluation link between the reasoning result and the deduction process, so that the large language model can rely on the continuously optimized and more objective evaluation distribution as the selection basis in subsequent search iterations. This effectively enhances the self-correction ability of the search tree in the complex solution space and its adaptive evolution ability to evolve towards a better search path.

[0122] In some embodiments, determining the response text corresponding to the input text based on the updated search tree can be achieved by: extracting multiple candidate search paths from the updated search tree; for each candidate search path, determining a path score based on the evaluation values ​​of the nodes in the candidate search path; selecting a target search path from the multiple candidate search paths based on the path score; and concatenating the inference text corresponding to each node in the target search path according to the node order to generate the response text corresponding to the input text, or using the inference text corresponding to the terminating node in the target search path as the response text.

[0123] For example, the path score of a candidate search path can be determined by taking the arithmetic mean of the evaluation values ​​of all nodes on the candidate search path except the root node, or by directly reading the evaluation value of the terminal node to which the candidate search path belongs. After determining the target search path with the highest score, all intermediate reasoning text fragments on the target search path are logically connected end to end to form a complete response paragraph.

[0124] For example, the target search path includes "Node 1: First calculate the part within the parentheses," "Node 2: Then perform the addition operation," and the terminating "Node 3: The final result is 8." Through text concatenation, the final response text corresponding to the input text is "First calculate the part within the parentheses; then perform the addition operation; the final result is 8." If only the final conclusion is needed according to the task requirements, then the reasoning text "The final result is 8" in the terminating node "Node 3" is directly determined as the response text.

[0125] By generating child nodes based on a first set of reference nodes providing globally shared positive reasoning references and a second set of reference nodes providing globally shared negative reasoning references during multiple search iterations of the search tree, the large language model can balance positive experience guidance and negative directional constraints when generating nodes, thereby improving the reasoning quality and logical rigor of the newly generated child nodes. Simultaneously, the first and second reference node sets are dynamically updated based on the evaluation values ​​of the child nodes, prompting subsequent new target nodes determined based on the evaluation values ​​to focus more on high-potential search paths. Combining this dynamic evolution mechanism to update the search tree and determine the response text achieves efficient allocation of search computing resources, thereby improving the overall reasoning speed of the large language model and the accuracy of the response text.

[0126] In other embodiments, see Figure 3C The evaluation value of each child node is obtained based on a pre-trained evaluation model. Before initializing the search tree based on the input text, the following steps 401 to 403 can also be performed, which are explained in detail below.

[0127] In step 401, a training dataset is constructed based on the node samples in the search tree sample and the inference text samples corresponding to the node samples.

[0128] Here, search tree samples refer to offline tree-like data instances that are pre-generated and retained to represent the complete multi-step reasoning state topology during the historical business data collection phase or by using baseline artificial intelligence models for inference sampling.

[0129] For example, search tree samples can be obtained by: selecting a series of historical prompt task data (such as a labeled math problem bank or code requirement bank) including known correct results; inputting the historical prompt task data into a pre-configured baseline large language model to perform multiple inference operations; recording all intermediate inference steps output by the baseline large language model along with branch states on the inference path, and exporting the recorded structured state transition data to confirm the required search tree sample.

[0130] Here, a node sample refers to a basic data unit in the topological structure of a search tree sample that carries a discrete reasoning step and state attribute.

[0131] Here, the inference text sample refers to the inference text stored inside the node sample, which is composed of natural language sequences, formulas, code fragments, etc., output by the baseline large language model in the historical stages of multi-step inference.

[0132] In some embodiments, the training dataset is constructed based on the node samples in the search tree sample and the inference text samples corresponding to the node samples. This can be achieved by: for each node sample in the search tree sample, determining the evaluation value sample of the node sample and using the evaluation value sample as the label value corresponding to the node sample; and constructing the training dataset based on the inference text samples corresponding to each node sample in the search tree sample and the label values ​​corresponding to each node sample.

[0133] In some embodiments, for each node sample in the search tree sample, determining the evaluation value sample of the node sample can be achieved in the following ways: obtaining the evaluation value sample corresponding to the parent node sample of the node sample as the preceding evaluation value sample, wherein if the parent node sample is the root node sample, the evaluation value sample corresponding to the parent node sample is a preset initial value; determining the step reward value corresponding to the node sample based on the preceding evaluation value sample, the inference distance corresponding to the node sample, and the indication information corresponding to the node sample, wherein the inference distance is used to characterize the inference progress corresponding to the node sample, and the indication information is used to indicate whether the inference logic represented by the search path sample corresponding to the node sample is correct; determining the evaluation value sample corresponding to the node sample based on the preceding evaluation value sample and the step reward value.

[0134] Here, whether the reasoning logic represented by the search path sample corresponding to the node sample is correct refers to the annotation information used to reflect the effectiveness of the intermediate reasoning steps and the optimization path to which the node sample belongs in achieving the target task, determined by matching the reasoning text sample with the preset standard answer or by specific logical verification rules.

[0135] For example, the aforementioned "correct" or "incorrect" discrimination attribute can be determined in the following ways: by comparing the reasoning text sample corresponding to the node sample with the preset correct answer to the target task, or by verifying through a logic analysis program whether it is on a valid derivation chain leading to the correct conclusion. For instance, if the reasoning text sample corresponding to the node sample constitutes a valid and coherent derivation sequence pointing to the preset correct answer, or if the node sample is marked as being located on a pre-verified successful search path sample, then the reasoning logic represented by the search path sample corresponding to the node sample is determined to be correct; conversely, if the reasoning text sample is found to contain factual errors, logical contradictions, or computational errors that do not conform to objective facts or preset rules, or if the derivation branch to which the node sample is attached ultimately leads to a result contrary to the correct answer in the search topology, then the reasoning logic represented by the search path sample corresponding to the node sample is determined to be incorrect.

[0136] For example, to obtain the evaluation value sample corresponding to the parent node sample of a node sample as the preceding evaluation value sample, it can be achieved as follows: follow the hierarchical association pointer established inside the search tree sample, trace back upwards and locate the previous level node sample directly connected to the currently processed node sample, and determine it as the parent node sample; then, read the pre-recorded score value in the data field of the parent node sample, and directly confirm the read score value as the preceding evaluation value sample.

[0137] For example, if the currently processed node sample B is located at the second level of the search tree, the node sample A at the first level is found by following the connecting edge and used as the parent node sample; if the evaluation value sample corresponding to node sample A is read as 0.6, then 0.6 is used as the preceding evaluation value sample. If the parent node sample is at the top level zero (i.e., the root node sample), since no actual inference operation has been started at this time, the preset initial value (e.g., the value 0.0) is directly extracted as the preceding evaluation value sample.

[0138] For example, the step reward value corresponding to a node sample can be determined based on the inference distance corresponding to the preceding evaluation value sample and the node sample, as well as the indication information corresponding to the node sample. This can be achieved as follows: Obtain a preset target full score benchmark (e.g., normalized full score constant 1), calculate the difference between the target full score benchmark and the preceding evaluation value sample to obtain the reward margin to be allocated; add the inference distance corresponding to the node sample to the zero-prevention constant (e.g., constant 1) to form the smoothing step size denominator; divide the reward margin by the smoothing step size denominator to calculate the single-step absolute expected improvement; then, based on the data features represented by the indication information corresponding to the node sample, construct an adjustment multiplier for positive and negative polarity switching; finally, multiply the single-step absolute expected improvement with the adjustment multiplier to calculate the step reward value with positive and negative polarity signs.

[0139] For example, suppose the preceding evaluation value of the current node sample is 0.4, the remaining distance requires 2 more steps to obtain the final answer (i.e., the inference distance is 2), and inspection reveals that the inference logic of the currently processed node sample is incorrect (i.e., the indication information is represented as a penalty state, and the constructed adjustment multiplier is -1). Calculations show that the reward margin is (1 - 0.4) = 0.6, the smoothing step size denominator is (2 + 1) = 3, and the single-step absolute expected improvement is 0.6 / 3 = 0.2. Multiplying 0.2 by the adjustment multiplier -1, the final step reward value for the determined node sample is -0.2.

[0140] For example, based on the preceding evaluation value sample and the step reward value, the evaluation value sample corresponding to the node sample can be determined in the following way: perform an arithmetic addition operation to add the preceding evaluation value sample to the step reward value to obtain the intermediate state value; in order to prevent multiple negative penalties from causing the evaluation score to lose the meaning of the relative probability distribution, compare the intermediate state value with the lower limit isolation constant (e.g., the value 0) and forcibly select the target value with the larger value as the evaluation value sample corresponding to the node sample of the final truncated output.

[0141] For example, continuing with the values ​​given above, the preceding evaluation value sample is 0.4, and the step reward value is -0.5. The intermediate state value obtained by summing these values ​​is -0.1. The maximum value operation is then performed on -0.1 and the lower limit isolation constant 0, ultimately determining that the evaluation value sample corresponding to the node sample is 0.

[0142] For example, the evaluation value of a node sample can be calculated using formula (4): (4) in, Indicates being in the first Evaluation values ​​of the node samples in the step; Indicates the first The evaluation value sample corresponding to the parent node sample of the step (corresponding to the previous evaluation value sample); Indicates that for the first The step-by-step reward value is calculated based on the given inference text sample. This represents an extremum function operator that finds the maximum value between the input parameter sequence and the lower bound constant 0.

[0143] For example, the calculation of the step reward value can be expressed by formula (5): (5) in, This represents the reward value for the step to be solved; a preset constant. This represents the full score benchmark under ideal simulation conditions; This represents a sample of preceding evaluation values; This represents the inference distance corresponding to the node sample (e.g., the number of steps required to reach the final conclusion node (the termination node corresponding to the correct answer)). This represents the indication information corresponding to the node sample. If the reasoning logic is correct (i.e., the reasoning text sample corresponding to the node sample belongs to a valid derivation step pointing to the correct answer, or the node sample is located above a pre-marked correct search path sample), the value is 0. In this case, the adjustment multiplier is used. The value is 1 if the reasoning logic is wrong (i.e., the reasoning text sample corresponding to the node sample contains a logical fallacy, or the reasoning branch where the node sample is located ultimately leads to an incorrect solution result), and the adjustment multiplier is -1.

[0144] By integrating the indication information used to reveal the correctness of the reasoning logic with the reasoning distance depth used to anchor the deduction progress, high-fidelity evaluation value labels can be generated for each intermediate node sample in the training dataset. This can accurately quantify the substantial contribution of single-step logic to achieving the final goal, and lay an objective and detailed data foundation for the subsequent evaluation model to learn dense process supervision signals in a few-sample environment. In turn, it significantly enhances the perception accuracy and quantitative discrimination ability of the trained evaluation model in the evolution state of reasoning logic.

[0145] In step 402, the initial evaluation model is trained in the first stage based on the training dataset to obtain the first evaluation model. The first stage training is used to enable the first evaluation model to distinguish between high-quality inference text samples and low-quality inference text samples.

[0146] Here, the initial evaluation model refers to a deep neural network model with prior natural language understanding and attention network feature extraction capabilities.

[0147] In some embodiments, the initial evaluation model is trained in the first stage based on the training dataset to obtain the first evaluation model, which can be achieved as follows: Based on the annotation values ​​corresponding to each inference text sample in the training dataset, high-quality inference text samples and low-quality inference text samples are determined; sample pairs are constructed based on the high-quality and low-quality inference text samples; the shared preceding inference text context of the sample pairs is obtained; a preset reference model is obtained to generate a first reference generation probability of the high-quality inference text samples in the sample pairs and a second reference generation probability of the low-quality inference text samples in the sample pairs based on the preceding inference text context; the initial evaluation model is obtained to generate a first predicted generation probability of the high-quality inference text samples in the sample pairs and a second predicted generation probability of the low-quality inference text samples in the sample pairs based on the preceding inference text context; a first loss value is determined based on the difference between the first predicted generation probability and the first reference generation probability, and the difference between the second predicted generation probability and the second reference generation probability; the parameters of the initial evaluation model are updated based on the first loss value to obtain the first evaluation model.

[0148] For example, determining high-quality and low-quality inference text samples based on the labeled values ​​corresponding to each inference text sample in the training dataset can be achieved as follows: Iterate through the labeled values ​​(i.e., evaluation value samples) corresponding to each node stored in the training dataset. Set a first quality judgment threshold (e.g., 0.8) and a second quality judgment threshold (e.g., 0.2). For each inference text sample, if the corresponding annotation value is greater than or equal to the first quality judgment threshold, the corresponding content is judged as a high-quality logical step and confirmed as a high-quality inference text sample; if the corresponding annotation value is less than or equal to the second quality judgment threshold, the corresponding content is judged as a logical trap or erroneous step and confirmed as a low-quality inference text sample.

[0149] For example, in a mathematical reasoning exercise, for the preliminary steps... arrive This resulted in two subsequent texts. The annotation value corresponding to text A is... Since the value is 0.9, and 0.9 is greater than 0.8, text A is marked as a high-quality inference text sample (denoted as ). ); the annotation value corresponding to text B Since the value is 0.15, and 0.15 is less than 0.2, text B is marked as a low-quality inference text sample (denoted as ). ).

[0150] For example, constructing sample pairs based on high-quality and low-quality inference text samples can be achieved as follows: Utilizing the inference path relationships generated during Monte Carlo Tree Search (MCTS), find high-quality and low-quality inference text samples derived from the same preceding inference text context; combine these text records with explicit quality contrast relationships to construct binary pairs representing preference order (i.e., preference pairs), which serve as sample pairs for training the first evaluation model. The goal of the first stage of training is to maximize the likelihood probability difference between the preferred step and the unpreferred step generated by the first evaluation model, relative to a pre-defined reference model.

[0151] For example, given the same mathematical problem-solving context, the aforementioned high-quality text A and low-quality text B are bound together to generate a format like... The preference pairs are used for subsequent direct preference alignment optimization of the model.

[0152] For example, obtaining the first reference generation probability of a high-quality inference text sample in a sample pair and the second reference generation probability of a low-quality inference text sample in a sample pair based on the pre-defined reference model and the preceding inference text context can be achieved as follows: Input the pre-defined inference text context shared by the sample pair into the reference model which is in a parameter-frozen state; use the autoregressive prediction mechanism of the reference model to calculate the joint probability of the complete character sequence corresponding to the high-quality inference text sample given the preceding inference text context, to obtain the first reference generation probability; similarly, calculate the joint probability of the complete character sequence corresponding to the low-quality inference text sample to obtain the second reference generation probability.

[0153] For example, the reference model can be a pre-trained language model with basic natural language generation capabilities, or an instruction alignment model that has undergone supervised fine-tuning (SFT). The reference model is used to keep the network parameters fixed during the first stage of training (i.e., it does not participate in gradient updates), thereby providing a raw, unbiased lexical probability distribution baseline for the initialized evaluation model.

[0154] For example, the preset reference model calculates that, under the current context, the probability of generating a high-quality step A is 0.3 (first reference generation probability), and the probability of generating an erroneous step B is 0.1 (second reference generation probability).

[0155] For example, determining the first loss value based on the difference between the first predicted generation probability and the first reference generation probability, and the difference between the second predicted generation probability and the second reference generation probability, can be achieved as follows: First, calculate the logarithmic rate of change (i.e., log-ratio) of the predicted generation probability given by the evaluation model being trained relative to the reference generation probability given by the reference model; then, calculate the difference between the logarithmic rate of change of high-quality inference text samples and the logarithmic rate of change of low-quality inference text samples, and weight them using scaling parameters; subsequently, convert the difference into interval scores through a nonlinear mapping function; finally, perform expectation calculation on all sample pairs in the full training batch and take the negative value to obtain the first loss value characterizing the alignment degree of the model preferences.

[0156] For example, the calculation of the first loss value can be expressed by formula (6): (6) in, This represents the first loss value calculated for the first phase of training. Indicates the training dataset All sample pairs in the middle Calculate the mathematical expectation; This indicates that the sample pairs share the same preceding inference text context; This represents a high-quality inference text sample; This indicates a low-quality inference text sample; This indicates the evaluation model that is undergoing parameter update training. A reference model in which parameters remain constant; Hyperparameters representing the strength of the preset adjustment model preference and the degree of deviation from the reference model constraint; This represents the Sigmoid activation function; Represents the natural logarithm operation.

[0157] For example, based on the first loss value, updating the parameters of the initialized evaluation model to obtain the first evaluation model can be achieved as follows: Calculate the gradient direction and value of the first loss value relative to the weights of each network layer in the initialized evaluation model using an automatic differentiation algorithm; employ stochastic gradient descent or adaptive gradient optimization algorithms (such as AdamW) to correct the neuron parameters in the initialized evaluation model along directions that reduce the first loss value. Iterate multiple times until the first loss value drops to a preset range and stabilizes. At this point, solidify the trained model parameters to obtain the first evaluation model that can effectively distinguish inference quality levels.

[0158] By constructing pairs of high-quality and low-quality reasoning text samples with explicit quality comparison based on labeled values, and introducing a parameter-preserving reference model as a benchmark to perform single-step direct preference optimization, the initial evaluation model can be strongly driven to maximize the likelihood probability difference between preferred high-quality steps and logically flawed steps. This preference alignment mechanism based on relative probability deviation not only effectively eliminates the deep dependence of the evaluation model on large-scale, finely labeled data, but also endows the evaluation model with the ability to keenly perceive and distinguish the superiority or inferiority of logic at the single-step reasoning level, thus anchoring a stable and reliable cognitive foundation for accurately selecting high-value nodes in subsequent multi-step search and deduction.

[0159] In step 403, based on the training dataset, the first evaluation model is trained in the second stage to obtain the second evaluation model, and the second evaluation model is used as the trained evaluation model. The second stage training is used to enable the second evaluation model to classify the logical quality of the reasoning text samples.

[0160] In some embodiments, the second evaluation model is obtained by performing a second-stage training on the first evaluation model based on the training dataset. This can be achieved by: discretizing the labeled values ​​in the training dataset to obtain multiple quality categories; determining a category label for each inference text sample based on the multiple quality categories; inputting each inference text sample into the first evaluation model to obtain the predicted probability corresponding to the multiple quality categories; determining a second loss value based on the category label and the predicted probability; and updating the parameters of the first evaluation model based on the second loss value to obtain the second evaluation model.

[0161] For example, discretizing the labeled values ​​in the training dataset to obtain multiple quality categories can be achieved as follows: obtain continuous labeled values ​​recorded in the training dataset (e.g., floating-point values ​​with a range of [0,1]); set a uniform or non-uniform partitioning interval (e.g., configure the value 0.2 as the partitioning span); use the partitioning interval to divide the continuous range of values ​​into several non-overlapping sub-intervals, and map each sub-interval to a set of extreme value categories. The final set obtained is the multiple quality categories.

[0162] For example, if the set value range is [0,1], and it is discretized according to a uniform interval of 0.2, then five extreme value category intervals can be divided (i.e., [0.8,1.0], [0.6,0.8], [0.4,0.6], [0.2,0.4], [0.0,0.2]). Correspondingly, five qualitative classification dictionaries are established: C = {Excellent (corresponding to the first level above), Good (corresponding to the second level above), Fair (corresponding to the third level above), Poor (corresponding to the fourth level above), and Extremely Poor (corresponding to the fifth level above)}. This constitutes the multiple quality categories required for the second stage of training.

[0163] For example, determining the category label for each inference text sample based on multiple quality categories can be achieved as follows: For each inference text sample in the training dataset, read the corresponding label value; determine which sub-interval the read label value falls into; finally, assign the extreme value category associated with the sub-interval mapping to the currently processed inference text sample as the true benchmark for the second stage of training (GroundTruth).

[0164] For example, if the labeled value of the inference text sample corresponding to a certain step The calculated result is 0.85. Since 0.85 falls within the interval [0.8, 1.0], it is... The "excellent" category in the set is confirmed as the category label corresponding to this inference text sample. .

[0165] For example, inputting each inference text sample into the first evaluation model to obtain the predicted probabilities corresponding to multiple quality categories can be achieved as follows: the inference text sample combined with the preceding context is converted into a word vector and input into the first evaluation model; features are extracted using the deep multi-head self-attention network of the first evaluation model, and the extracted feature vectors are appended to the multilayer perceptron classification head at the end of the first evaluation model; the normalized exponential function (Softmax) in the classification head is used to calculate the probability value of the current text input being classified into each specific category in the preset extreme value category dictionary, thereby outputting a normalized numerical sequence distributed across various dimensions as the final predicted probability.

[0166] For example, for the inference text sample labeled "excellent", the first evaluation model, after calculating based on internal weights, gives the output distribution: the probability of belonging to "excellent" is 0.65, the probability of belonging to "good" is 0.25, and the probabilities of belonging to the other three categories are 0.05, 0.03, and 0.02, respectively.

[0167] For example, the second loss value can be determined based on the class label and the predicted probability as follows: Extract the probability value corresponding to the true class label from the predicted probability distribution output by the first evaluation model; perform a natural logarithmic operation on the extracted probability value; then, using the true class label as the target mask (similar to One-Hot encoding), calculate the cross-entropy loss between the true distribution (i.e., only the probability of the correct class is 1, and the rest are 0) and the probability distribution predicted by the first evaluation model; finally, sum the obtained cross-entropy values ​​and combine them with a negative sign operation to confirm the second loss value.

[0168] For example, the calculation of the second loss value can be expressed by formula (7): (7) in, This represents the second loss value used to optimize classification prediction ability (i.e., cross-entropy loss). This represents the set of all categories (e.g., containing five quality levels) resulting from the discretization process. Represents a set Any one of the categories; This indicates that the summation operation is performed on the calculation results of all categories in the set; This represents the input inference text sample (along with the preceding context); This represents the true category label corresponding to the inference text sample; This indicates the indicator function, which predicts the evaluation category. Exactly equal to the real label When the condition is met, the indicator function takes the value 1; otherwise, it takes the value 0. This represents the first evaluation model in the second stage of training (possessing a parameter sequence). Give the category for the input. The predicted probability; This represents the function for performing logarithmic operations.

[0169] For example, based on the second loss value, the parameters of the first evaluation model are updated to obtain the second evaluation model. This can be achieved as follows: Based on the determined second loss value, the weight gradient information that reduces the cross-entropy is calculated through the backpropagation algorithm, and the parameters of the first evaluation model are adjusted through a gradient optimizer (such as Adam); when multiple rounds of data iteration (Epoch) have been performed and the classification accuracy reaches the preset qualified level, the parameter update is terminated, and the trained second evaluation model is obtained.

[0170] By discretizing continuous labeled values ​​and reconstructing them into multiple discrete quality category labels, and then combining the predicted probability to calculate the extreme value cross-entropy loss to drive the second-stage parameter update of the evaluation model, the overfitting and noise interference problems that are prone to occur when deep neural networks accurately fit continuous regression values ​​in a few-sample environment can be effectively overcome. This strategy design, which transforms value assessment into a fine-grained classification task, makes full use of the natural advantage of clear discrete rating boundaries, greatly enhances the feature extraction and cross-domain generalization ability of the evaluation model for different logical deduction states, and ensures that the trained evaluation model can output high-fidelity logical quality rating signals with extreme stability and accuracy when facing complex multi-step reasoning. This lays a solid quantitative evaluation foundation for the global information diversion and automated path pruning of the search tree.

[0171] Through steps 401 to 403, the labeled values ​​that integrate logical correctness and deduction progress are transformed into structured preference sample pairs. Then, the evaluation model is sequentially subjected to single-step direct preference alignment training aimed at maximizing the difference in likelihood probability, and fine-grained value classification training aimed at predicting the discrete extreme value category. This two-stage training mechanism fully leverages the dual advantages of preference identification and discrete rating, enhances the cross-domain generalization efficiency and logical perception acuity of the evaluation model in a low-sample environment, and ensures that the trained evaluation model can output accurate evaluation values.

[0172] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0173] Large Language Models (LLMs) have demonstrated exceptional capabilities in numerous natural language processing and logical reasoning applications. However, when dealing with complex reasoning tasks involving high difficulty and long chains of reasoning (such as mathematical proofs, code generation, and deep question answering), directly relying on LLM-generated results often faces limitations such as easily divergent logic, susceptibility to logical fallacies, and difficulty in self-correction. To overcome the limitations of single-step linear generation by LLMs (such as Chain-of-Thought (CoT) technology), introducing tree-based or graph-based search planning mechanisms (such as Tree of Thoughts (ToT) and Graph of Thoughts (GoT)) has become an important direction for technological evolution.

[0174] In related technologies, a reasoning enhancement scheme based on Monte Carlo Tree Search (MCTS) is employed to guide the output of a large language model. The large language model serves as a policy network for generating candidate steps, and an independent Process Reward Model (PRM) is introduced as a value network for evaluating node quality. Within the standard Monte Carlo Tree Search framework, the algorithm follows a four-step loop of "Selection, Expansion, Simulation, and Backpropagation," searching for the optimal reasoning path through multiple explorations.

[0175] However, the following limitations still exist in the relevant technologies: (1) Computational Redundancy and Information Silos: In standard tree search (such as MCTS or ToT), each rollout starting from the root node is usually independent. Suppose the model discovers a complex logical fallacy in a certain rollout branch and fails as a result. When subsequent searches go through similar reasoning states, the model has no mechanism to globally "remember" past lessons and is very likely to repeatedly try and fail in the same logical trap. This lack of global memory, the "Markov property", leads to computational redundancy in the complex search space, resulting in reduced search efficiency.

[0176] (2) Sparsity and inaccuracy of process supervision signals: It is difficult to train a process reward model (PRM) that can provide accurate scores during the reasoning process. Related technologies rely on a large number of manually labeled step-level tags, which are difficult to scale up to general domains. If unsupervised reward feedback is carried out solely based on the correctness of the final answer, a serious credit assignment problem is faced—that is, an incorrect reasoning process may produce a correct solution with low confidence due to the randomness of the model (false positive), or a reasoning idea may be completely rejected due to careless calculation in the last step (false negative). Such sparse and noisy supervision signals can seriously mislead the search direction.

[0177] (3) Lack of dynamic knowledge reuse mechanism: Search enhancement schemes that incorporate external retrieval (such as RAG) in related technologies mainly focus on retrieving external document libraries in a static dimension. However, the high-scoring intermediate reasoning steps (intermediate states) in the multi-step reasoning exploration process of large language models are themselves a kind of high-value dynamic knowledge that is highly consistent with the current context. Related technologies usually discard these intermediate states after a single search, which makes it impossible for the model to achieve self-reinforcement and real-time reuse of experience in parallel reasoning branches.

[0178] In view of this, embodiments of this application provide a data processing method for large language models, which can introduce globally shared positive and negative reasoning references (i.e., heuristic memory and fallacy memory) into the search tree architecture of multi-step reasoning, and dynamically assemble these global reference knowledge into prompt words when expanding nodes. This breaks down information silos between search branches and significantly reduces the number of repeated explorations of invalid paths through "full graph pruning"; at the same time, in conjunction with a specially designed two-stage training and optimization evaluation model, it provides high-fidelity discretized logical quality classification signals, thereby significantly reducing the computational cost of reasoning while improving the accuracy and rigor of large language models in handling complex logical tasks.

[0179] The implementation details of the data processing method for large language models provided in the embodiments of this application will be described below.

[0180] For example, see Figure 4 , Figure 4 This is a comparative diagram of the data processing principles provided in the embodiments of this application. For example... Figure 4 As shown, suppose the input text (i.e., the question) is: "Given..." and The axes intersect at points A and B. Let V be its vertex. If Given an equilateral triangle, find the value of m.

[0181] like Figure 4 As shown on the left, in the related technique (Monte Carlo tree search), when a large language model performs multi-step reasoning for the above problem, each attempt to explore downwards from the root node (Rollout) is treated as an independent, isolated trajectory. For example, in a certain branch of the attempt, the model may generate a certain intermediate reasoning step (such as calculating...). In other exploratory operations, the model may still generate similar or even completely repeated intermediate derivation steps from scratch (such as...). Figure 4 The "calculation" appears in different branch paths. This results in a large amount of computing power being consumed in repeatedly deriving the correct path and repeatedly trying out incorrect paths, leading to a search tree that, although large, is inefficient and contains a lot of redundant calculations.

[0182] In comparison, such as Figure 4 As shown on the right, the data processing method for the large language model provided in the embodiments of this application can introduce a global sharing mechanism in the multi-step reasoning process, dynamically acquire and guide the reuse of heuristic successful experiences and verified logical fallacies.

[0183] like Figure 4 As shown, for the same problem, in the early exploration stage of deduction, if the large language model generates intermediate reasoning nodes that demonstrate correct logic (e.g., deriving the discriminant)... "), and the corresponding calculated evaluation value shows a high score (e.g. Figure 4 If the value marked as 0.9 is found, the current deduction step is considered to be of high-quality logic. At this point, a positive heuristic recording action is triggered, storing the corresponding intermediate inference node in the first reference node set (or heuristic memory, denoted as ). Conversely, if the large language model generates inference nodes during the random search phase that lead to deviations in thought or even incorrect application of formulas, and the corresponding calculated evaluation values ​​are low (e.g., ... Figure 4 If the value marked as 0.1 is found, it is determined that a logical fallacy has occurred, and the inference node that caused the error is stored in the second reference node set (or the Fallacies Memory, denoted as ). In subsequent search iterations, when performing expansion processing, the large language model receives positive inference references (i.e., input in the form of prompt words) provided by the first set of reference nodes. Figure 4 The "heuristic" reference marked with a "√" and the negative reasoning reference provided by the second set of reference nodes (i.e. Figure 4 The inclusion of "error" references marked with "×" allows the large language model to proactively avoid recorded erroneous deduction methods when expanding into unknown directions (equivalent to performing proactive pruning at the search tree level), while directly absorbing and using validated high-quality deduction logic (e.g., directly reusing correct discriminant conclusions to construct the side length equation of an equilateral triangle), thereby accelerating the convergence of the overall deduction logic.

[0184] By employing globally reinforced correct problem-solving strategies and proactively pruning error-prone branches, the data processing method for large language models provided in this application enables them to accurately and quickly pinpoint the optimal reasoning trajectory and reach the correct answer, even under conditions that generate extremely concise search tree branches. Compared with related technologies, it significantly reduces the breadth requirements of exploration, effectively enhancing the rigor and accuracy of complex logical reasoning while reducing computational redundancy overhead.

[0185] 1. Monte Carlo Tree Search (MCTS) To further clarify the technical basis of the embodiments of this application, preliminary knowledge will be explained first.

[0186] Monte Carlo Tree Search builds and refines the search tree by repeatedly performing four steps: Selection, Expansion, Simulation, and Back-propagation. In the selection phase, starting from the root node of the search tree, suitable child nodes are recursively selected downwards using the Upper Confidence Bound Applied to Tree (UCT) algorithm.

[0187] For example, the formula for calculating the confidence upper limit for any node n can be expressed by formula (8): (8) in, This represents the historical number of visits to node n; This represents the score (i.e., evaluation value) of node n. Indicates the parent node to which node n is associated; This refers to the historical number of visits to the parent node; This represents the preset exploration weight (i.e., the exploration constant). After completing one round of simulated exploration extending downwards, a backpropagation mechanism is executed upwards along the reached path to dynamically update the score values ​​of node n and its associated parent nodes.

[0188] The inference task of a large language model is defined as generating a coherent sequence of steps to solve a given input problem. The corresponding solution can be represented as a trajectory in the search tree. Each of the intermediate elements It represents an independent logical reasoning step, while the final element This contains the final solution to the task. Therefore, the optimization objective of this application's embodiments is to find the optimal trajectory within the complex solution space that maximizes the probability of accurate deduction.

[0189] 2. Data processing method for large language models provided in the embodiments of this application Based on the aforementioned logical principles, the overall architecture and execution flow of the embodiments of this application will be described below. For example, see [link to example]. Figure 5 , Figure 5 This is a schematic diagram illustrating the first principle of the data processing method for a large language model provided in this application embodiment.

[0190] like Figure 5 As shown, the data processing method provided in this application integrates a multi-step search tree mechanism (MCTS) and a process reward model (PRM), and couples it with a reflective memory architecture. Overall, the data processing architecture includes three core components: a first reference node set (heuristic memory, ), second reference node set (fallacy memory, ) and the memory manager.

[0191] Within this framework, the data processing method for the large language model provided in this application embodiment can be implemented in the following ways: First, receive the input text (i.e.) Figure 5 The input problem shown in the figure ), and based on the input problem Initialize the root node (node) of the search tree Following the aforementioned upper confidence limit principle, target nodes to be expanded are selected within the search tree.

[0192] Subsequently, the Large Language Model (LLM) is invoked to expand the selected target node through branch generation, outputting multiple parallel candidate inference steps. The generated nodes are then input into the Process Reward Model (PRM) for scoring. Here, the Process Reward Model can be either a Continuous Process Reward Model (C-PRM) based on continuous numerical regression (corresponding to...). Figure 5 The “Model 1” and “Model 2” in the text can be a general-purpose large language model with strong logical discrimination ability, or it can be a dual-stage process reward model (D-PRM) that has been specially trained through two stages: “Stage 1: single-step direct preference optimization” and “Stage 2: classification supervision fine-tuning” (corresponding to the evaluation model obtained by training through the first and second stages above).

[0193] Next, based on the evaluation score output by the process reward model (corresponding to the evaluation value mentioned above), bidirectional splitting and storage operations are performed: if a generated node is evaluated to have high quality (e.g., evaluation value...), then... ,like Figure 5 Middle score If a generated node is found to be of low quality (e.g., its evaluation value is low), then the corresponding node is confirmed as a valid shortcut or a verified best pattern and stored in the first reference node set (heuristic memory); if a generated node is found to be of low quality (e.g., its evaluation value is low), then the node is considered a valid shortcut or a verified best pattern and stored in the first reference node set (heuristic memory). ,like Figure 5 Middle score If a node is identified as a logical blind spot or logical error, it is stored in the second reference node set (error memory). Simultaneously, the memory manager continuously performs similarity filtering on the two reference node sets to remove highly overlapping redundant data.

[0194] Finally, for the remaining nodes (such as...) Figure 5 The nodes within the dashed box that are in a state of pending expansion When performing a new round of generation and deduction, positive guiding signals are retrieved from the first reference node set, and negative constraints are retrieved from the second reference node set. Both are simultaneously input into the large language model as reference data. The stored heuristic memory and error memory can promote global information sharing between different branches of the large language model, thereby guiding the generation of subsequent steps extremely efficiently and actively pruning invalid search spaces at the source. After reaching the terminal node, backpropagation updates are completed along the path, ultimately accurately locking in the optimal trajectory and answer.

[0195] Here, heuristic memory is used to persistently archive valid reasoning trajectories generated in historical searches, as well as validated intermediate inference states. For example, during a simulation of multi-step search tree iterations, if the evaluation value of a node is detected to exceed a preset positive recognition threshold... If the corresponding node content is not found, it will be integrated into the first reference node set. These stored inference insights serve as positive guiding signals in subsequent node expansion processing, enabling the large language model to reproduce and reuse successful inference strategies when generating new branches.

[0196] Here, the error memory serves as a global repository specifically for storing erroneous reasoning patterns, logical busy-aways, and failed deduction trajectories. For example, when a node's evaluation value is less than or equal to the negative identification threshold... At this time, the corresponding node data is archived into the second reference node set to provide globally shared negative interference feedback. This negative feedback mechanism provides the necessary prior constraints for the dynamic pruning of the search tree space, enabling the large language model to actively avoid recurring logical errors in subsequent iterations, thereby significantly improving the overall throughput and inference efficiency of the search process.

[0197] Here, memory management aims to maintain the conciseness of stored information. Its core function is to extract key inference patterns while filtering out redundant entries with highly overlapping content, thereby preventing computational overload on large language models. By ensuring that only inference and analysis insights with high fidelity and typicality are retained in the first and second reference node sets, the memory manager optimizes the overall quality of the bootstrapping information provided when expanding target nodes.

[0198] For example, see Figure 6 , Figure 6 This is a schematic diagram of the second principle of the data processing method for a large language model provided in this application embodiment. When performing a multi-step inference task, the initial parameters are first read, including the preset root node (denoted as...). The preset total number of search iterations (denoted as ) ), exploration constant (denoted as ) and the positive identification threshold used to control memory data splitting (denoted as ) ) and negative recognition threshold (denoted as Finally, the processing flow will output an updated search tree, which contains updated statistics (i.e., the evaluation values ​​of each node). and historical visit count ), and the stored memory state (i.e., the first reference node set). With the second reference node set ).

[0199] Below, in conjunction with Figure 6 This application describes the implementation details of the data processing method for large language models provided in its embodiments: In the initial phase, an initialization operation is performed, which sets the variable representing the current loop iteration. Set it to 1, and simultaneously set the first reference node set... Second reference node set Initialize to an empty set (i.e.) Next, enable the total number of search iterations. The control loop logic, i.e., executing the outer condition judgment. Under the condition that the loop is satisfied, each complete loop execution process corresponds to one "search iteration" mentioned in step 101 above.

[0200] When it is determined If the result is "yes", the execution flow for one round of search iteration is initiated. First, the node currently being processed (denoted as the current node) is... The value is assigned to the root node of the search tree. Then, the following four processing stages are executed sequentially: (1) First stage: Selection stage.

[0201] After entering the selection phase, the inner condition judgment is executed. Is it a leaf node? (This is in the process of determining the currently traversed node.) If the node is not a leaf node (i.e., the result is "no"), proceed to the downward traversal loop. Inside the loop, the node's evaluation value, exploration constant, and historical visit count are calculated using formula (1) above. Represents a node The set of subordinate child nodes, Represents any child node in the set of child nodes. This represents the evaluation value of the corresponding child node. Represents a node Historical visit count, Represents child nodes The historical number of visits. By executing the above formula (1), the child node with the largest selection value is iteratively selected from the set of child nodes, and the selected child node is updated as the node to be traversed in the next level. This inner loop proceeds downwards along the search tree topology until the current node is determined. When the node is a leaf node, the inner loop is exited. The logic at this stage corresponds in detail to the "select the target node from the search tree" operation in step 201 above.

[0202] (2) Second stage: Expansion.

[0203] During the expansion phase, conditional checks are performed first. Is it the termination node? If the node was determined through the aforementioned selection phase... If the expected target node is not the termination node (the result is "no"), then the function is executed. The process: that is, combining , and nodes The large language model is invoked to perform text inference, thereby generating at least one new child node under the target node (the set of newly generated child nodes is denoted here). This calling process corresponds to formula (2) established earlier. After generation, for the set of child nodes... Each child node in Call the update function respectively This allows us to obtain and initialize each new child node. Evaluation value If node If a node is determined to be a termination node, the aforementioned expansion and evaluation calculation operations will not be performed. The logic at this stage corresponds to the processing work of "generating at least one child node of the target node" in step 201 above and the previous processing work of "obtaining the evaluation value of each child node" in step 202 above.

[0204] (3) The third stage: Simulation.

[0205] Entering the simulation phase (i.e., the greedy deduction phase), the judgment is executed. Is it the terminal node? This refers to any node currently being processed along the exploration path. If it is not a terminating node (the result is "No"), then proceed to the internal downward derivation loop: first, perform the state judgment. Should it be fully expanded? If the node... If the node is not fully expanded (resulting in "No"), then perform a node expansion operation to obtain subsequent branches. Next, execute the split storage logic within the dashed box: traverse the nodes. Set of child nodes Each child node in If child nodes Evaluation value Then this child node join in If child nodes Evaluation value Then this child node join in After the memory data splitting is completed, according to formula (3) above, the child node with the largest evaluation value is selected from the child node set and assigned the value to a new node for further derivation. Then, it returns to the beginning of this stage and performs the judgment on the termination node again. This stage utilizes a threshold. and The logic for determining and updating the memory pool corresponds to the process in step 202 above, which involves "updating the first reference node set and the second reference node set based on the evaluation value of each child node." The logic for advancing node alternation based on the formula for finding the maximum value corresponds to the process in step 203 above, which involves "determining a new target node from at least one child node based on the evaluation value of each child node, until the new target node is the termination node."

[0206] (4) Fourth stage: Backpropagation.

[0207] Determining during the simulation phase Is it the terminating node? If the result is "yes," the single-step inference attempt ends, and the process proceeds to the backpropagation operation. For the path extending downwards from the root node to the terminating node in this iteration, the function that performs the statistical update operation is called. The process involves a reverse backtracking sequence from bottom to top, refreshing and overwriting the historical visit counts and corresponding evaluation values ​​of each node traversed along the entire search path. This logic corresponds to the "update the search tree based on the search path corresponding to the terminating node" step 204 above.

[0208] After the backpropagation step is completed, execute the operation instruction representing the increment of the iteration round. Then return to the outer loop and re-execute the loop condition. "When the loop completes the preset..." After several search iterations, a search tree is output. Finally, this updated search tree is used to extract the target path to determine the corresponding response text to the input text, thus concluding the overall processing flow.

[0209] 3. Two-stage process reward model The evaluation value of each child node is obtained based on a pre-trained evaluation model (i.e., a process reward model). This application embodiment employs a two-stage evaluation model training strategy, which is described in detail below.

[0210] (1) Data Construction (corresponding to step 401 above).

[0211] The search tree generated based on Monte Carlo Tree Search (MCTS) represents each partial solution (denoted as ) on the search path. ) Calculate the objective quality value separately The calculated quality value is then used as a reference standard (ground truth) for subsequent training alignment. Here, value... It is iteratively updated to accurately reflect the cumulative progress of each deduction to reach the correct answer. The calculation and update logic corresponds to formulas (4) and (5) above, which will not be repeated here.

[0212] (2) Preference alignment based on step-level direct preference optimization (SDPO) (corresponding to step 402 above).

[0213] In the first phase, embodiments of this application introduce a step-by-step direct preference optimization (SDPO) architecture to align the model's underlying preferences. Utilizing the real inference paths generated during MCTS, a preference pair is constructed, consisting of two texts, one positive and one negative. .in, This represents nodes that meet the criteria for high-quality inference (e.g., decision). (nodes), and This represents a node with incorrect reasoning (e.g., a decision). (nodes).

[0214] The loss function for the first stage can be found in formula (6) above, and will not be repeated here.

[0215] (3) Fine-Grained Value Classification (corresponding to step 403 above).

[0216] In the second stage, this application embodiment restructures the task of estimating evaluation values ​​from continuous numerical regression to a discrete classification task targeting logical quality. This is because, in few-shot application scenarios facing the challenge of data scarcity, large language models often have a better ability to perceive and generalize discrete hierarchical categories than to accurately fit continuous numerical values.

[0217] This application embodiment obtains the records in the training dataset that are in the current state. Continuous labeled values ​​within the interval The continuous interval is discretized according to a preset uniform step size (e.g., 0.2), thereby dividing and mapping out a set of extreme value categories consisting of five quality levels, for example, as follows: .

[0218] For each given inference text sample and associated real category labels The cross-entropy loss between the predicted probability distribution and the true category label is optimized to guide the evaluation model to learn the boundary features between different logical quality levels, thereby transforming value assessment into a more robust feature rating process. The loss function for the second stage can be found in formula (7) above, and will not be repeated here.

[0219] 4. Experimental Results To verify the effectiveness of the data processing method for large language models provided in the embodiments of this application, multi-dimensional comparative tests were conducted on different baseline configurations and various challenging public evaluation benchmarks.

[0220] The evaluation criteria are divided into two categories: one is a set of scientific and fact-verification test sets that focuses on the breadth of knowledge and factual information (including GPQA-DIAMOND and FMT test sets); the other is a set of mathematical reasoning test sets that focuses on the rigor of multi-step logic (including the general mathematics test set MATH500 and the advanced mathematics test set AIME25).

[0221] In selecting the benchmark methods for comparison, in addition to introducing the method of the embodiments in this application, we also compared the zero-shot direct inference method, the action strategy method based on prompting and retrieval (ReAct), the architecture oriented towards agent search (Search-O1), and large-scale models of Monte Carlo tree search augmented route baselines using different strategies (such as ReST-MCTS). And MCTS-RAG).

[0222] In terms of evaluation metrics, for the science and fact test set, the exact match rate (EM) and F1 score are used to measure the accuracy of reasoning results; for the mathematical reasoning test set, the accuracy score calculated directly by the algorithm is used for measurement. Meanwhile, in terms of structural morphology, the number of exploration trajectories (Traj) (a smaller value indicates lower computational power consumption) and search depth are used to evaluate search efficiency.

[0223] To eliminate interference from the native capabilities of the underlying large language model, two preprocessed language models with different scale parameters (i.e., large language model-1 and large language model-2) were uniformly connected in the experiment to perform comparative verification.

[0224] (1) Analysis of accuracy of comprehensive reasoning The overall reasoning accuracy results of the comparative test are shown in Table 1.

[0225] Table 1

[0226] Based on the data in Table 1, the embodiments of this application demonstrate outstanding performance in knowledge-intensive tasks. In the Large Language Model-1 deployment environment, the method of this application achieved an EM score of 65.08 on the extremely difficult GPQA benchmark, outperforming the MCTS-RAG baseline, which also relies on retrieval and search optimization. Simultaneously, in the FMT test set, it achieved an EM score of 70.50 and an F1 score of 70.07, indicating that the node evaluation mechanism combined with the reflection mechanism in this application can effectively suppress the frequent textual illusions and logical deviations in direct reasoning scenarios. In the Large Language Model-2 deployment environment, the method of this application also maintained a high level of performance on the extremely difficult mathematical competition logic deduction (MATH500 and AIME25), and comprehensively surpassed the conventional Zero-shot and ReAct methods in all evaluations.

[0227] (2) Search efficiency evaluation and analysis For example, see Figure 7A and Figure 7B Compared to the MCTS-RAG method, the method in this application significantly reduces the number of search trajectories across all test datasets. Specifically, taking the Large Language Model-1 as an example, the average number of trajectories decreased from 18.76 to 8.40 in the GPQA test set, a reduction of over 55%; in the AIME25 test set, the number of trajectories decreased from 12.17 to 3.67. Taking the Large Language Model-2 as an example, the number of trajectories shrank from 6.32 to 3.09 in the MATH test set. This significant reduction in the number of trajectories reflects the substantial role of the first reference node set (heuristic memory) and the second reference node set (error memory) configured above in actively pruning the search space. By limiting repeated probing of low-value branches, computational resources are ensured to be concentrated on highly promising inference directions.

[0228] See Figure 7A and Figure 7B When handling commonsense verification tasks such as GPQA and FMT, the tree structure inference depth extended by the method in this application remains at a level close to the baseline (e.g., the Large Language Model-1 score is 3.54 compared to 3.58 in the GPQA test set), empirically ensuring that the derivation logic is not excessively omitted. However, in mathematical tasks such as MATH and AIME25, the inference depth shows a slight reduction (e.g., the Large Language Model-1 score is reduced from 2.93 to 2.42 in the MATH test set). This smoothing of computational depth not only avoids redundancy caused by useless intermediate states in long text generation modes but also corroborates that the method provided in this application enables the Large Language Model to find a more direct and highly efficient solution path in its derivation.

[0229] In summary, compared to tree search schemes that rely on a large number of generated trajectories for filtering, the data processing method for large language models provided in this application achieves a higher problem-solving accuracy than the benchmark with fewer trajectories and a stable search depth.

[0230] (3) Validation of the effectiveness of the two-stage evaluation model For the evaluation model of the two-stage mechanism (i.e. steps 401 to 403) of "step size preference alignment (SDPO)" and "fine-grained value classification loss optimization (CLS)" mentioned in the above embodiments, a targeted comparative verification was carried out, and the specific comparison data is shown in Table 2.

[0231] Table 2

[0232] Wherein, "LA" is used to represent Label Accuracy, Oracle-PRM indicates that the scoring result is output using the baseline large language model in the relevant technology, which can be regarded as the operational boundary benchmark under the current technical conditions; Light-PRM indicates that the scoring result (evaluation value) is output using the lightweight two-stage process reward model trained by the embodiment of this application (the evaluation model trained by the method provided by the embodiment of this application).

[0233] As shown in Table 2, the evaluation model trained using the method provided in this application's embodiments approaches or even matches the consistency of the scoring feedback with the control group, Oracle-PRM. Whether using the Big Language Model-1 or Big Language Model-2 backbone network, Light-PRM achieves the same level of accurate matching (EM scores of 70.50 and 68.00, respectively) as the benchmark group, with negligible differences in metrics such as F1.

[0234] Furthermore, as shown in Table 2, although Light-PRM has a slightly higher number of trajectories (Traj) than Oracle-PRM (which is the theoretical upper limit, for example, 14.91 vs. 10.60 in the large language model-1), indicating that the lightweight model has slightly weaker discrimination ability in the early stages of the search and needs a slightly larger exploration scope, the search depth remains stable (2.33 vs. 2.24), successfully preventing inference from degenerating into inefficient long chains. This further proves that Light-PRM can achieve performance comparable to Oracle-PRM, reducing the cost of calling large models in practical applications while ensuring logical fidelity.

[0235] (4) Ablation test To further explore the specific contributions of each component in the architecture provided in the embodiments of this application, a first reference node set (denoted as...) is discussed. ) and the second reference node set (denoted as Ablation experiments were performed. For an example, see [link to example]. Figure 7C The horizontal axis represents the number of trajectories, used to measure search efficiency, while the vertical axis represents the model inference score, used to measure accuracy. The color intensity of the dots is determined by the color bar on the side, with values ​​ranging from 2.5 to 3.5. Darker dots (corresponding to the bottom of the color bar) indicate a smaller search depth, while lighter dots (corresponding to the top of the color bar) indicate a larger search depth.

[0236] See Figure 7C In the experimental data distribution of the MATH test set and the AIME25 test set, it can be seen that, compared with... Figure 7C The related technologies (MCTS-RAG) and the methods provided in this application embodiment (i.e., the dots marked "method of this application embodiment" in the figure) can achieve the optimal reasoning score while consuming fewer exploration trajectories.

[0237] When the second reference node set (i.e., "w / o") is removed When the first reference node set (i.e., "w / o") is removed, the model's inference score declines, and due to the loss of the negative constraint, the model needs to explore more trajectories to find the correct solution. In the case of ""), the reasoning score showed a more severe decline, indicating that positive heuristic guidance signals play a crucial role in reinforcing and guiding the derivation of complex mathematical logic. If the two memory components mentioned above (i.e., "w / o") are simultaneously removed... and ", the model's performance will drop to its lowest point.

[0238] In summary, Figure 7C The ablation experiments verified a significant synergistic effect between the first and second reference node sets. By injecting both positive successes and negative lessons learned into the inference context of the large language model, the embodiments of this application not only ensure the rigor and accuracy of the reasoning logic, but also significantly reduce the size of the search space through "full-graph pruning," making the large language model more efficient and intelligent when dealing with challenging tasks.

[0239] The data processing method for large language models provided in this application has the following beneficial effects: (1) Reduce computational redundancy in the exploration space and achieve efficient utilization of computing power in the reasoning stage.

[0240] To address the information silos existing between independent search branches in related technologies, the data processing method for a large language model provided in this application constructs a global sharing mechanism by introducing a first reference node set (heuristic memory) and a second reference node set (error memory). During node expansion, the large language model can proactively extract and reuse validated high-value intermediate states (positive experience), while simultaneously avoiding known logical pitfalls (negative constraints) at the source. This enables the data processing method for the large language model provided in this application to automatically perform proactive pruning across the entire graph. Compared to search planning enhancement schemes in related technologies (such as MCTS-RAG), the data processing method for the large language model provided in this application achieves better inference scores while consuming fewer exploration trajectories, successfully guiding the large language model to achieve a paradigm shift from "brute-force search relying on large-scale trajectory generation" to "agile deduction based on a global reflection sharing mechanism."

[0241] (2) Improve the rigor and accuracy of large language models in dealing with complex logical tasks.

[0242] To address the limitations of large language models in long-chain reasoning, which are prone to logical divergence and factual illusions, and lack a dynamic knowledge reuse mechanism, the data processing method for large language models provided in this application transforms high-scoring intermediate states in multi-step exploration into dynamic reference knowledge that directly fits the current context. Through the assembly of dynamic prompt words, it provides real-time reusable positive heuristics and negative bias prevention constraints for each expansion in unknown directions. The experimental data above demonstrates that this synergistic effect effectively prevents reasoning divergence and logical deviation when dealing with scientific fact verification tasks (such as the GPQA test set) and high-level multi-step mathematical deduction tasks (such as the AIME25 test set), achieving reasoning accuracy that comprehensively surpasses various comparative benchmark architectures.

[0243] (3) It improves the fidelity and stability of process monitoring signals, providing accurate node evaluation basis for multi-step search of large language models.

[0244] To address the challenges of training highly accurate evaluation models (process reward models) and the credit assignment difficulties arising from reliance on final answer evaluation in related technologies, this application's implementation of a large language model data processing method proposes a two-stage training mechanism combining single-step direct preference optimization (SDPO) and fine-grained value classification. This transforms continuous value regression into a more generalizable discrete-level classification task, guiding the evaluation model to learn the boundary features between different logical quality levels. This enables the model to output high-fidelity and extremely stable step-level process supervision signals even in few-shot scenarios. This accurate stage-based feedback signal avoids misleading the search direction, thus providing a highly reliable scoring basis for expanding the high-quality branches of the entire search tree while ensuring the fidelity of logical evaluation.

[0245] The following description continues to illustrate the exemplary structure of the data processing device 433 for the large language model provided in this application embodiment as a software module. In some embodiments, such as... Figure 2 As shown, the software modules in the data processing device 433 storing the large language model in the memory 430 may include: The data processing module 4331 is used to initialize a search tree based on the input text and perform multiple search iterations. Each search iteration includes: selecting a target node from the search tree; if the target node is not a termination node, invoking a large language model to generate at least one child node of the target node based on a first reference node set, a second reference node set, and the target node, wherein the first reference node set is used to provide globally shared positive inference references, and the second reference node set is used to provide globally shared negative inference references; obtaining the evaluation value of each child node and updating the first reference node set and the second reference node set according to the evaluation value of each child node; determining a new target node from the at least one child node according to the evaluation value of each child node, until the new target node is a termination node; updating the search tree according to the search path corresponding to the termination node, and determining the response text corresponding to the input text according to the updated search tree if the search termination condition is met.

[0246] In some embodiments, the data processing module 4331 is further configured to take the root node of the search tree as the starting point of traversal, and take the currently traversed node as a candidate node, wherein the text content corresponding to the root node is the input text; if the candidate node is a leaf node, then the candidate node is determined as the target node; if the candidate node is not a leaf node, then a new candidate node is determined from the set of child nodes corresponding to the candidate node, and the target node selection is continued based on the new candidate node until the new candidate node is a leaf node, and the new candidate node is determined as the target node.

[0247] In some embodiments, the data processing module 4331 is further configured to: obtain the historical access count of the candidate node; for each child node in the set of child nodes corresponding to the candidate node, obtain the historical access count of the child node and obtain the evaluation value of the child node; for each child node in the set of child nodes corresponding to the candidate node, perform a nonlinear transformation based on the historical access count of the candidate node and the historical access count of the child node to obtain the exploration weight of the child node, wherein the exploration weight is positively correlated with the historical access count of the candidate node and negatively correlated with the historical access count of the child node; for each child node in the set of child nodes corresponding to the candidate node, obtain the product of a preset exploration constant and the exploration weight of the child node, and use the sum of the product and the evaluation value of the child node as the selection value of the child node; select the child node with the largest selection value from the set of child nodes corresponding to the candidate node and determine it as the new candidate node.

[0248] In some embodiments, the data processing module 4331 is further configured to: obtain the inference text context corresponding to the target node, wherein the inference text context includes the inference text corresponding to each node in the search path from the root node to the target node in the search tree; obtain the inference text corresponding to the nodes in the first reference node set as positive reference text, and obtain the inference text corresponding to the nodes in the second reference node set as negative reference text; determine the prompt word text based on the inference text context, the positive reference text, and the negative reference text; input the prompt word text into the large language model to obtain at least one candidate inference text output by the large language model, wherein the positive reference text in the prompt word text is used to guide the large language model to reuse the inference logic corresponding to the positive reference text, and the negative reference text is used to constrain the large language model to avoid the inference logic corresponding to the negative reference text; for each candidate inference text, use the candidate inference text as the inference text corresponding to a new node, and use the new node as a child node of the target node in the search tree.

[0249] In some embodiments, the data processing module 4331 is further configured to: combine the inference text context with the inference text corresponding to the child node for each child node to obtain the evaluation text corresponding to the child node; for each child node, perform feature encoding on the evaluation text corresponding to the child node using a pre-trained evaluation model to obtain evaluation text features; perform feature mapping on the evaluation text features to obtain the quality category corresponding to the child node; and determine the evaluation value based on the quality category, wherein the evaluation value is used to characterize the inference logic quality of the inference text corresponding to the child node and the inference progress of the inference path constituted by the evaluation text.

[0250] In some embodiments, the data processing module 4331 is further configured to perform the following processing for each child node: if the evaluation value of the child node is greater than or equal to a preset first threshold, then the child node is added to the first reference node set; if the evaluation value of the child node is less than or equal to a preset second threshold, then the child node is added to the second reference node set, wherein the first threshold is greater than the second threshold.

[0251] In some embodiments, the data processing module 4331 is further configured to obtain the inference text features of the inference text corresponding to each node in the first reference node set, and obtain the inference text features of the inference text corresponding to each node in the second reference node set; for the first reference node set, calculate the first similarity of the inference text features between any two nodes in the first reference node set, and delete any one of the two nodes whose first similarity is greater than or equal to a preset third threshold; for the second reference node set, calculate the second similarity of the inference text features between any two nodes in the second reference node set, and delete any one of the two nodes whose second similarity is greater than or equal to the third threshold.

[0252] In some embodiments, the data processing module 4331 is further configured to select the child node with the largest evaluation value from at least one child node of the target node as the new target node; if the new target node is not a termination node and the new target node is not a fully expanded node, then perform expansion processing on the new target node to generate at least one child node corresponding to the new target node; for each child node corresponding to the new target node, obtain the evaluation value of the child node, and select the child node with the largest evaluation value from at least one child node corresponding to the new target node as the subsequent new target node, until the new target node is a termination node.

[0253] In some embodiments, the data processing module 4331 is further configured to determine the search path from the root node to the termination node in the current search iteration, wherein the text content corresponding to the root node is the input text; along the search path, for each node in the search path, update the historical access count of the node; along the search path, for each node in the search path, update the evaluation value of the node.

[0254] In some embodiments, the data processing module 4331 is further configured to obtain the evaluation value corresponding to the termination node in the current search iteration as a reward value; traverse the nodes in the search path as nodes to be updated in the order from the parent node of the termination node to the root node along the search path; and for the currently traversed node to be updated, determine the updated evaluation value corresponding to the node to be updated based on the historical access count of the node to be updated before the update, the evaluation value before the update, the reward value, and the historical access count of the node to be updated after the update.

[0255] In some embodiments, the data processing module 4331 is further configured to extract multiple candidate search paths from the updated search tree; for each candidate search path, determine a path score based on the evaluation value of the nodes in the candidate search path; select a target search path from the multiple candidate search paths according to the path score; and concatenate the inference text corresponding to each node in the target search path according to the node order of the target search path to generate the response text corresponding to the input text, or use the inference text corresponding to the termination node in the target search path as the response text.

[0256] In some embodiments, the acquisition of the evaluation value of each child node is implemented based on a pre-trained evaluation model. The data processing module 4331 is further configured to construct a training dataset based on node samples in the search tree samples and the inference text samples corresponding to the node samples; perform a first-stage training on the initialized evaluation model based on the training dataset to obtain a first evaluation model, wherein the first-stage training is used to enable the first evaluation model to distinguish between high-quality inference text samples and low-quality inference text samples; perform a second-stage training on the first evaluation model based on the training dataset to obtain a second evaluation model, and use the second evaluation model as the trained evaluation model, wherein the second-stage training is used to enable the second evaluation model to classify the logical quality of the inference text samples.

[0257] In some embodiments, the data processing module 4331 is further configured to determine an evaluation value sample for each node sample in the search tree sample, and use the evaluation value sample as the label value corresponding to the node sample; and construct the training dataset based on the inference text sample corresponding to each node sample in the search tree sample and the label value corresponding to each node sample.

[0258] In some embodiments, the data processing module 4331 is further configured to obtain an evaluation value sample corresponding to the parent node sample of the node sample as a preceding evaluation value sample, wherein if the parent node sample is a root node sample, the evaluation value sample corresponding to the parent node sample is a preset initial value; based on the preceding evaluation value sample, the inference distance corresponding to the node sample, and the indication information corresponding to the node sample, determine the step reward value corresponding to the node sample, wherein the inference distance is used to characterize the inference progress corresponding to the node sample, and the indication information is used to indicate whether the inference logic represented by the search path sample corresponding to the node sample is correct; and based on the preceding evaluation value sample and the step reward value, determine the evaluation value sample corresponding to the node sample.

[0259] In some embodiments, the data processing module 4331 is further configured to: determine high-quality inference text samples and low-quality inference text samples based on the annotation values ​​corresponding to each inference text sample in the training dataset; construct sample pairs based on the high-quality inference text samples and the low-quality inference text samples; obtain the pre-defined inference text context shared by the sample pairs; obtain a first reference generation probability and a second reference generation probability of the low-quality inference text samples in the sample pairs generated by a preset reference model based on the pre-defined inference text context; obtain a first predicted generation probability and a second predicted generation probability of the low-quality inference text samples in the sample pairs generated by the initialized evaluation model based on the pre-defined inference text context; determine a first loss value based on the difference between the first predicted generation probability and the first reference generation probability, and the difference between the second predicted generation probability and the second reference generation probability; and update the parameters of the initialized evaluation model based on the first loss value to obtain the first evaluation model.

[0260] In some embodiments, the data processing module 4331 is further configured to discretize the labeled values ​​in the training dataset to obtain multiple quality categories; determine a category label for each inference text sample based on the multiple quality categories; input each inference text sample into the first evaluation model to obtain a predicted probability corresponding to the multiple quality categories; determine a second loss value based on the category label and the predicted probability; and update the parameters of the first evaluation model based on the second loss value to obtain the second evaluation model.

[0261] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. An electronic device's processor reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the large language model data processing method described in this application.

[0262] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the data processing method for a large language model provided in this application. For example, ... Figure 3A The data processing method for large language models is shown.

[0263] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0264] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0265] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0266] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0267] In summary, through the embodiments of this application, child nodes are generated based on a first set of reference nodes providing globally shared positive reasoning references and a second set of reference nodes providing globally shared negative reasoning references during multiple search iterations of the search tree. This allows the large language model to consider both positive experience guidance and negative directional constraints when generating nodes, thereby improving the reasoning quality and logical rigor of the newly generated child nodes. Simultaneously, the first and second reference node sets are dynamically updated based on the evaluation values ​​of the child nodes, prompting subsequent new target nodes determined based on the evaluation values ​​to focus more on high-potential search paths. Combining this dynamic evolution mechanism with the updating of the search tree and the determination of the response text achieves efficient allocation of search computing resources, thereby improving the overall reasoning speed of the large language model and the accuracy of the response text.

[0268] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.< / eos> < / end>

Claims

1. A data processing method for a large language model, characterized in that, The method includes: The search tree is initialized based on the input text, and multiple search iterations are performed, wherein each search iteration includes: Select a target node from the search tree. If the target node is not a terminal node, call the large language model to generate at least one child node of the target node based on the first reference node set, the second reference node set, and the target node. The first reference node set is used to provide globally shared positive reasoning references, and the second reference node set is used to provide globally shared negative reasoning references. Obtain the evaluation value of each child node, and update the first reference node set and the second reference node set according to the evaluation value of each child node; Based on the evaluation value of each child node, a new target node is determined from the at least one child node until the new target node is a termination node; The search tree is updated according to the search path corresponding to the termination node, and if the search termination condition is met, the response text corresponding to the input text is determined according to the updated search tree.

2. The method according to claim 1, characterized in that, Selecting a target node from the search tree includes: Taking the root node of the search tree as the starting point of traversal, the currently traversed node is taken as a candidate node, wherein the text content corresponding to the root node is the input text; If the candidate node is a leaf node, then the candidate node is determined as the target node; If the candidate node is not a leaf node, a new candidate node is determined from the set of child nodes corresponding to the candidate node, and the target node selection is continued based on the new candidate node until the new candidate node is a leaf node, and the new candidate node is determined as the target node.

3. The method according to claim 2, characterized in that, The step of determining a new candidate node from the set of child nodes corresponding to the candidate node includes: Obtain the historical access count of the candidate node; For each child node in the set of child nodes corresponding to the candidate node, obtain the historical access count of the child node and the evaluation value of the child node; For each child node in the set of child nodes corresponding to the candidate node, a nonlinear transformation is performed based on the historical access count of the candidate node and the historical access count of the child node to obtain the exploration weight of the child node. The exploration weight is positively correlated with the historical access count of the candidate node and negatively correlated with the historical access count of the child node. For each child node in the set of child nodes corresponding to the candidate node, obtain the product of a preset exploration constant and the exploration weight of the child node, and use the sum of the product and the evaluation value of the child node as the selection value of the child node; From the set of child nodes corresponding to the candidate node, select the child node with the largest selection value and determine it as the new candidate node.

4. The method according to claim 1, characterized in that, The step of generating at least one child node of the target node based on the first reference node set, the second reference node set, and the target node includes: Obtain the inference text context corresponding to the target node, wherein the inference text context includes the inference text corresponding to each node in the search path from the root node to the target node in the search tree; Obtain the inference text corresponding to the nodes in the first reference node set as positive reference text, and obtain the inference text corresponding to the nodes in the second reference node set as negative reference text; Based on the context of the reasoning text, the positive reference text, and the negative reference text, the prompt word text is determined; The prompt text is input into the large language model to obtain at least one candidate inference text output by the large language model. The positive reference text in the prompt text is used to guide the large language model to reuse the inference logic corresponding to the positive reference text, and the negative reference text is used to constrain the large language model to avoid the inference logic corresponding to the negative reference text. For each candidate inference text, the candidate inference text is used as the inference text corresponding to the new node, and the new node is used as a child node of the target node in the search tree.

5. The method according to claim 4, characterized in that, The process of obtaining the evaluation value of each child node includes: For each child node, the inference text context is combined with the inference text corresponding to the child node to obtain the evaluation text corresponding to the child node; For each child node, the evaluation text corresponding to the child node is feature-encoded using a pre-trained evaluation model to obtain evaluation text features; the evaluation text features are then feature-mapped to obtain the quality category corresponding to the child node; and the evaluation value is determined based on the quality category, wherein the evaluation value is used to characterize the reasoning logic quality of the reasoning text corresponding to the child node, and the reasoning progress of the reasoning path constituted by the evaluation text.

6. The method according to claim 1, characterized in that, The step of updating the first reference node set and the second reference node set according to the evaluation value of each child node includes: Perform the following processing for each of the child nodes: If the evaluation value of the child node is greater than or equal to a preset first threshold, then the child node is added to the first reference node set; If the evaluation value of the child node is less than or equal to a preset second threshold, then the child node is added to the second reference node set, wherein the first threshold is greater than the second threshold.

7. The method according to claim 6, characterized in that, The method further includes: Obtain the inference text features of the inference text corresponding to each node in the first reference node set, and obtain the inference text features of the inference text corresponding to each node in the second reference node set; For the first reference node set, calculate the first similarity of the inference text features between any two nodes in the first reference node set, and delete any one of the two nodes whose first similarity is greater than or equal to a preset third threshold. For the second reference node set, calculate the second similarity of the inferred text features between any two nodes in the second reference node set, and delete any one of the two nodes whose second similarity is greater than or equal to the third threshold.

8. The method according to claim 1, characterized in that, The step of determining a new target node from the at least one child node based on the evaluation value of each child node, until the new target node is a termination node, includes: Select the child node with the largest evaluation value from at least one child node of the target node, and use it as the new target node; If the new target node is not a termination node and the new target node is not a fully extended node, then an extension process is performed on the new target node to generate at least one child node corresponding to the new target node. For each child node corresponding to the new target node, obtain the evaluation value of the child node, and select the child node with the largest evaluation value from at least one child node corresponding to the new target node as the subsequent new target node, until the new target node is the termination node.

9. The method according to claim 1, characterized in that, The step of updating the search tree according to the search path corresponding to the termination node includes: Determine the search path from the root node to the termination node in this search iteration, wherein the text content corresponding to the root node is the input text; Along the search path, for each node in the search path, update the historical access count of the node; Along the search path, for each node in the search path, update the evaluation value of the node.

10. The method according to claim 9, characterized in that, Updating the evaluation value of each node in the search path includes: Obtain the evaluation value corresponding to the termination node in the current search iteration, and use it as the reward value; Along the search path, the nodes in the search path are traversed as nodes to be updated in the order from the parent node of the termination node to the root node. For the currently traversed node to be updated, the updated evaluation value of the node is determined based on the node's historical access count before the update, its evaluation value before the update, the reward value, and the node's historical access count after the update.

11. The method according to claim 1, characterized in that, Determining the response text corresponding to the input text based on the updated search tree includes: Extract multiple candidate search paths from the updated search tree; For each candidate search path, a path score is determined based on the evaluation values ​​of the nodes in the candidate search path. Based on the path score, a target search path is selected from the plurality of candidate search paths; According to the node order of the target search path, the reasoning text corresponding to each node in the target search path is concatenated to generate the response text corresponding to the input text, or the reasoning text corresponding to the terminating node in the target search path is used as the response text.

12. The method according to any one of claims 1 to 11, characterized in that, The method of obtaining the evaluation value of each child node is based on a pre-trained evaluation model. Before initializing the search tree based on the input text, the method further includes: A training dataset is constructed based on the node samples in the search tree sample and the inference text samples corresponding to the node samples; Based on the training dataset, the initialized evaluation model is trained in the first stage to obtain the first evaluation model, wherein the first stage training is used to enable the first evaluation model to distinguish between high-quality inference text samples and low-quality inference text samples. Based on the training dataset, the first evaluation model is trained in a second stage to obtain a second evaluation model, and the second evaluation model is used as the trained evaluation model. The second stage of training is used to enable the second evaluation model to classify the logical quality of the reasoning text samples.

13. The method according to claim 12, characterized in that, The training dataset is constructed based on node samples in the search tree samples and the corresponding inference text samples, including: For each node sample in the search tree sample, an evaluation value sample is determined for the node sample, and the evaluation value sample is used as the label value corresponding to the node sample; The training dataset is constructed based on the inference text sample corresponding to each node sample in the search tree sample, and the annotation value corresponding to each node sample.

14. The method according to claim 13, characterized in that, Determining the evaluation value sample for each node sample in the search tree sample includes: Obtain the evaluation value sample corresponding to the parent node sample of the node sample as the preceding evaluation value sample. If the parent node sample is the root node sample, then the evaluation value sample corresponding to the parent node sample is a preset initial value. Based on the preceding evaluation value sample, the inference distance corresponding to the node sample, and the indication information corresponding to the node sample, the step reward value corresponding to the node sample is determined. The inference distance is used to characterize the inference progress corresponding to the node sample, and the indication information is used to indicate whether the inference logic represented by the search path sample corresponding to the node sample is correct. Based on the preceding evaluation value sample and the step reward value, the evaluation value sample corresponding to the node sample is determined.

15. The method according to claim 13, characterized in that, The first stage of training the initialized evaluation model based on the training dataset to obtain the first evaluation model includes: Based on the annotation value corresponding to each inference text sample in the training dataset, high-quality inference text samples and low-quality inference text samples are determined. Based on the high-quality inference text samples and the low-quality inference text samples, sample pairs are constructed; Obtain the shared pre-inference text context of the sample pairs; Based on the preceding inference text context, a preset reference model is obtained to generate a first reference generation probability of high-quality inference text samples in the sample pair and a second reference generation probability of low-quality inference text samples in the sample pair. The initial evaluation model is obtained based on the preceding inference text context to generate a first predicted generation probability of high-quality inference text samples in the sample pair and a second predicted generation probability of low-quality inference text samples in the sample pair. A first loss value is determined based on the difference between the first predicted generation probability and the first reference generation probability, and the difference between the second predicted generation probability and the second reference generation probability. Based on the first loss value, the parameters of the initialized evaluation model are updated to obtain the first evaluation model.

16. The method according to claim 13, characterized in that, The step of training the first evaluation model on the training dataset to obtain a second evaluation model includes: The labeled values ​​in the training dataset are discretized to obtain multiple quality categories; Based on the multiple quality categories, a category label is determined for each of the inference text samples; Each of the inference text samples is input into the first evaluation model to obtain the predicted probability corresponding to the plurality of quality categories; Based on the category label and the predicted probability, a second loss value is determined; Based on the second loss value, the parameters of the first evaluation model are updated to obtain the second evaluation model.

17. A data processing device for a large language model, characterized in that, The device includes: The data processing module is used to initialize a search tree based on the input text and perform multiple search iterations, wherein each search iteration includes: Select a target node from the search tree. If the target node is not a terminal node, call the large language model to generate at least one child node of the target node based on the first reference node set, the second reference node set, and the target node. The first reference node set is used to provide globally shared positive reasoning references, and the second reference node set is used to provide globally shared negative reasoning references. Obtain the evaluation value of each child node, and update the first reference node set and the second reference node set according to the evaluation value of each child node; Based on the evaluation value of each child node, a new target node is determined from the at least one child node until the new target node is a termination node; The search tree is updated according to the search path corresponding to the termination node, and if the search termination condition is met, the response text corresponding to the input text is determined according to the updated search tree.

18. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 16.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method described in any one of claims 1 to 16.

20. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method according to any one of claims 1 to 16.

Images