Long-chain reasoning redundancy reduction method based on semantic structure compression and direct preference optimization
By generating candidate path trees, selecting the best paths, and optimizing training using direct preferences, the redundancy problem in multi-step reasoning of large language models is solved, achieving an efficient and concise reasoning process and improving the model's reasoning efficiency and reliability.
Patent Information
- Application Number
- CN202510892618.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-30
- Publication Date
- 2025-10-17
AI Technical Summary
Existing large-scale language models suffer from redundancy, high computational overhead, and error accumulation in multi-step inference tasks, affecting inference efficiency and reliability, especially in edge device deployments where computational costs and latency are severe.
By generating multiple candidate inference paths, constructing a tree structure, selecting the preferred path, eliminating lengthy and inefficient paths, and training the model using the direct preference optimization algorithm, the generation of efficient and concise inference chains is encouraged.
It significantly reduces the number of inference steps and text length, improves inference efficiency and accuracy, is applicable to language models of different domains and sizes, reduces data acquisition costs, and enhances the model's ability to be applied in resource-constrained environments.
Smart Images
Figure CN120805987A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of large language model reasoning optimization, in particular to a long chain reasoning redundancy reduction method based on semantic structure compression and direct preference optimization. BACKGROUND
[0002] Large language models have shown strong capabilities in complex reasoning tasks, especially through the chain-of-thought (CoT) approach, which decomposes the problem-solving process into a series of intermediate steps, significantly improving the accuracy of tasks such as mathematical calculations and logical reasoning. However, existing advanced reasoning models often exhibit overthinking when performing multi-step reasoning, characterized by excessively long reasoning chains, redundant steps, and high computational overhead. Such lengthy and redundant reasoning not only reduces reasoning efficiency but also leads to the accumulation of errors, affecting the reliability of the final answer. In actual deployment on mobile devices and other edge devices, the high computational cost and delay caused by lengthy reasoning processes also severely restrict the practicality of large models.
[0003] To alleviate the above problems, researchers have proposed some methods to promote the generation of concise reasoning by large language models. For example, imposing a reward signal that encourages concise output (such as penalizing long texts) during reinforcement learning training or adding guidance in prompts that requires concise reasoning. However, these methods have two limitations: on the one hand, they often require a large number of samples to generate candidate reasoning to filter high-quality results, leading to a significant increase in computational cost; on the other hand, simply shortening the output length does not necessarily mean optimizing the reasoning quality. Our statistical analysis shows that the shortest reasoning path is not necessarily the most efficient path. Simple length rewards can sometimes cause the model to abandon necessary reasoning steps, resulting in "pseudo-concise" illusions where the number of words is reduced, but the reasoning logic still has redundancies or biases. This mismatch results in the existing methods not being ideal in improving reasoning efficiency.
[0004] In addition, some methods currently attempt to control reasoning generation through heuristic rules or step-by-step guidance to reduce unnecessary steps. However, these means often introduce additional biases, which can disrupt the logical integrity of the reasoning chain. There are also studies that aim to enable the model to perform simplified reasoning in the hidden space or use multiple small models to explore in parallel, but the implementation is complex, and it does not fundamentally solve the problem of objective criteria for redundancy evaluation.
[0005] Therefore, there is an urgent need for a more systematic and low-cost solution that can effectively evaluate and remove redundant steps while ensuring the correctness of the reasoning, to improve the efficiency and reliability of large language model reasoning. SUMMARY
[0006] In order to overcome the deficiencies of the prior art in long-chain reasoning redundancy control, the present application provides a long-chain reasoning redundancy reduction method based on semantic structure compression and direct preference optimization, which can greatly compress unnecessary intermediate steps, reduce the consumption of computing resources, and improve the efficiency and practicability of model reasoning while maintaining the accuracy of reasoning.
[0007] A long-chain reasoning redundancy reduction method based on semantic structure compression and direct preference optimization, comprising:
[0008] (1) For the problem to be solved, a plurality of chain reasoning paths are generated by a pre-trained large language model as candidate paths, each candidate path including a plurality of intermediate reasoning steps arranged in order and a final answer;
[0009] (2) The plurality of candidate paths generated in step (1) are combined into a tree structure as a reasoning decision topology;
[0010] (3) An efficiency index considering the number of steps and the length of text is calculated, and the preferred path is selected from the candidate paths based on the efficiency index;
[0011] (4) According to the set threshold condition, filter the remaining non-preferred paths in the tree structure, and eliminate the inefficient solutions compared with the preferred path;
[0012] (5) The preferred path selected in step (3) and the other inefficient paths filtered out in step (4) are constructed into a preference data set;
[0013] (6) If none of the candidate paths generated in step (1) meets the correct answer condition, or at least one correct answer path is not retained after step (4) filtering, the problem sample is considered to have no training value and is removed from the preference data set;
[0014] (7) Input the samples in the preference data set obtained through the above steps into the student model, and adjust and train the student model parameters; during the training process, the preferred path determined in step (3) is used as the ideal reference to encourage the student model to improve the probability of generating the path and reduce the preference degree for inferior paths;
[0015] (8) Use the trained student model to solve the problem.
[0016] The application is suitable for optimizing the inference process of a large language model (LLM). The method first generates a certain number of candidate solution ideas (inference paths) from the model itself, then merges these paths into a tree structure through semantic structure similarity to eliminate repeated inference segments; then, the merged inference tree is subjected to multi-factor preferred path screening to select the most concise and correct solution as the preferred target, while filtering out other obviously lengthy and inefficient paths; finally, the preferred data obtained through screening is used to directly optimize and train the model, prompting the model to tend to output more efficient inference chains.
[0017] In step (1), the generation of inference paths is performed by a large language model with strong inference ability. The model independently samples K different chain inference paths for each problem to be solved, providing diverse solution ideas for subsequent training in a structured form.
[0018] The specific process of step (2) is as follows:
[0019] First, select the path with the fewest inference steps as the backbone, denoted as S min , and determine its depth value as the reference depth; for the remaining candidate paths, gradually perform node merging under depth constraints: when a candidate path has a semantic similarity to an existing node in the current tree at a certain step that reaches a set threshold, and the depth of the merged path does not exceed the reference depth, align and merge the step of the candidate path with the existing node, thereby eliminating redundant inference steps; if the above conditions are not met, add a branch node to the current tree to retain the candidate path, thereby maintaining inference diversity while ensuring compact structure.
[0020] The semantic structure merging is achieved using a dynamic path integration mechanism, including: using the node sequence of the backbone path S min as the initial tree node set; for each other candidate path S i , search for similar nodes in the current tree along the path and attempt to align, when the semantic similarity of the t-th step of path S i to a node v in the tree reaches a threshold τ, and the depth d of node v from the root node plus the remaining step number Δl from step t to the end of path S i does not exceed the original depth limit of the backbone path (d+Δl≤origin_depth), then merge the node of path S i with the existing node v; otherwise, create a new node branch in the current tree to ensure that the depth of the merged tree structure does not exceed the predetermined threshold. Through the above method, the common segments of different candidate inference paths are merged and compressed, effectively reducing redundant calculations while avoiding path depth explosion.
[0021] In step (3), the calculation formula of the efficiency index is as follows:
[0022]
[0023] In the formula, E(τ j ) represents the efficiency index of the jth candidate path τ j , represents the total number of words of the path, represents the number of reasoning steps of the path.
[0024] The calculation of the efficiency index E considers both the reasoning steps and the length of the generated text, and measures the comprehensive cost of the reasoning path in the form of the Euclidean norm.
[0025] The preferred path is selected from the candidate paths based on the efficiency index, and the specific process is as follows:
[0026] The set of candidate paths that reach the expected answer and are logically correct is denoted as τ correct , and the preferred path τ + is selected from the set τ correct according to the following formula:
[0027]
[0028] When there is only one Pareto optimal path that meets the above conditions, the path is directly selected as the preferred path; if there are multiple candidate paths that are not dominated and are located on the Pareto frontier, the path with the least number of words is selected as the preferred path.
[0029] In step (4), the redundant and inefficient solutions that are compared with the preferred path are removed, and the specific process is as follows:
[0030] For each correct reasoning path in the set τ correct , when the number of reasoning steps is more than the preset step threshold and the text length exceeds the preset multiple of the preferred path τ + , the path is determined to be a redundant and long suboptimal solution and is filtered out; at the same time, any invalid solution path that has more steps and words than the preferred path and has an incorrect final answer is removed to ensure that only high-quality and efficient solution examples are retained in the preference data.
[0031] In the implementation process, for any correct solution path τ j , if and , it is determined that the path is too long and is eliminated.
[0032] For any incorrect solution path τ j , if and Then, the path is determined to be inefficient and incorrect and is eliminated.
[0033] In step (5), the preferred path τ + As the positive winning answer, each eliminated inefficient path is regarded as the corresponding negative inferior answer, and a plurality of preference sample data in pairs of winning and inferior are formed to constitute a preference data set.
[0034] By pairing the preferred path with other paths, the preferred path is regarded as a positive sample, and the remaining eliminated paths are regarded as negative samples. Each pair of samples is used for preference learning of the supervised model, wherein the model will be trained to be more inclined to generate a reasoning process similar to the positive sample path and away from the inefficient reasoning mode embodied by the negative sample path.
[0035] In step (6), when a certain problem does not appear in the correct answer among a preset number of reasoning paths, all candidate paths corresponding to the problem are discarded and are not included in the training preference data set, so as to avoid invalid samples interfering with the learning process of the model.
[0036] In step (7), a direct preference optimization algorithm (DPO) is used to adjust and train the student model parameters.
[0037] During the training process, the target function introduces a cross-entropy supervision item on the basis of the preference pair comparison loss, so as to ensure that the learning model pursues concise reasoning while strictly following the correct answer, thereby optimizing the reasoning chain generation strategy of the model.
[0038] In step (7), the direct preference optimization training combines the joint action of the preference comparison loss and the supervised fine-tuning loss. When the direct preference optimization algorithm is used to fine-tune the model, a log-likelihood loss constraint based on cross-entropy is introduced to punish the case that the probability of generating the next correct reasoning content in each step of the preferred path is reduced, and to encourage the model to generate more concise reasoning chains under the premise of ensuring the correctness of reasoning. Through the above training strategy, the model significantly reduces unnecessary reasoning redundancy while maintaining high accuracy, and improves the coherence and efficiency of the reasoning process.
[0039] Compared with the prior art, the present application has the following beneficial effects:
[0040] 1. Efficiency and practicality are significantly improved: the method can effectively reduce the number of reasoning steps and the length of the generated text under the premise of ensuring the correctness of the answer. According to the experimental statistics, the average length of the reasoning chain of the model optimized by the invention is reduced by about 26.8%, the text output length is reduced by about 28.1%, and the accuracy rate is not reduced but increased. At the same time, due to the avoidance of a large amount of redundant calculation, the reasoning overhead is significantly reduced, which is conducive to the deployment of the model in the environment with limited computing resources.
[0041] 2. Improve reasoning quality and reliability: the invention uses the direct preference optimization strategy to train the model not only to pursue simplicity but also to ensure the correctness of each step of reasoning logic. During the optimization process, supervision constraints are added to correct answer chains to avoid logical omissions caused by simply shortening answers and reduce false reasoning and invalid reflection. Therefore, the model maintains the coherence and rigor of the reasoning process while generating shorter answers.
[0042] 3. High data efficiency and wide applicability: the method does not require artificial construction of large-scale chain reasoning data sets, but relies on model self-sampling and screening to generate high-quality training samples, greatly reducing data acquisition costs. Compared with methods that rely on manual annotation or rule guidance, the invention can be applied to language models of different domains and different sizes, and can realize the automatic optimization of reasoning redundancy through a unified preference learning framework. It is universal and effective for mathematical, logical question answering, code reasoning and other tasks that require multi-step reasoning. BRIEF DESCRIPTION OF DRAWINGS
[0043] Figure 1 The figure is a flowchart of the method of the invention.
[0044] Figure 2 The figure is a schematic diagram of the system framework corresponding to the method of the invention. DETAILED DESCRIPTION
[0045] The invention will be described in further detail below in conjunction with the drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the invention and do not limit it in any way.
[0046] In the embodiment of the invention, it is assumed that there is a mathematical application question that requires multi-step reasoning, and the problem Q is: "A person walks at a speed of 5 kilometers per hour, how many kilometers can he walk in 8 hours?". For this problem, the correct answer should be 40 kilometers. The existing large language model can give a complete solution process (expert path) through chain reasoning, for example: the first step is to list the speed and time, the second step is to calculate the walking distance, and finally the answer is 40. However, we assume that the model may produce multiple different reasoning paths when solving autonomously, some of which are lengthy or repetitive. The method of the invention is applied to evaluate and optimize these reasoning paths of the model.
[0047] As Figure 1 shown, a long-chain reasoning redundancy reduction method based on semantic structure compression and direct preference optimization includes the following steps:
[0048] Step S1: Generate candidate reasoning paths.
[0049] First, use a large language model to independently generate multiple solutions for the question Q. The model may give different reasoning processes, for example: path 1 lists the formula "distance = speed x time" and directly calculates 40; path 2 calculates the cumulative distance walked per hour in detail and sums up to 40; path 3 may mistakenly attempt other unnecessary calculation steps, etc. Assume that the model generates K=5 initial reasoning paths, including the correct answer 40 and individual biased answers. All these paths will be used as candidates and input into the next redundancy evaluation process.
[0050] Step S2: Semantic structure compression.
[0051] The above 5 reasoning paths are combined to form a tree-like reasoning structure. As Figure 2 shown, path 1 with the least steps is selected as the backbone, and each reasoning step is sequentially taken as a tree node. Then, for path 2 5, compare their steps with the contents of the existing nodes in the tree step by step: for example, the first half of path 2 is exactly the same as path 1, so these nodes are merged with the backbone; path 3 starts to appear different redundant calculations from the second step, and it is detected that its semantic similarity with the second step of path 1 is low and the depth after merging will exceed the depth of the backbone, so it is not merged and a branch node is created in the tree to retain the subsequent steps of path 3. In this way, all candidate paths are integrated into this reasoning tree.
[0052] After merging, each node in the tree may correspond to the same behavior of multiple original paths (such as paths 1 and 2 sharing the first two steps), and these repeated parts are folded into a single node; while different ideas tried by the model are retained in different branches.
[0053] Step S3: Preference path screening.
[0054] In the obtained reasoning tree, one of the best problem-solving paths needs to be selected from the many merged paths as a training example. For this purpose, a multi-criteria evaluation index E is introduced to quantify the "efficiency" of each path, which considers both the number of steps and the number of words.
[0055] For Figure 2Each complete path in the above table calculates E value, assuming the result shows that the E value of path 1 is the lowest, path 2 is the second, and path 3 and other paths are much higher. Therefore, path 1 is determined as the preferred path τ + . This path has the least steps and concise statements, and the correct answer 40 can be deduced. In comparison, path 2 is also a correct answer but has a few more steps, path 3 not only has more steps but also has repeated logic, and path 4 is not considered because the reasoning is wrong. Thus, τ + is locked as the problem-solving process corresponding to path 1.
[0056] Step S4: Threshold condition filters other paths.
[0057] Next, the system filters other paths except τ + . According to the preset threshold standard: any correct path with more than 2 steps than the preferred path and more than 50% of the long text description is considered as a redundant answer. For example, path 2 has 3 more steps than path 1 and more words, which meets the condition of "long correct path" and is eliminated; path 3 is directly eliminated as "inefficient incorrect path" because the final answer is incorrect.
[0058] After this step of filtering, only path 1 is left as a high-quality answer, and the remaining paths are either discarded as redundant and too long (path 2) or discarded as invalid because of errors (path 3, path 4, etc.).
[0059] Step S5: Construct preference training samples.
[0060] The system converts the screening results into training data: the preferred path τ + (path 1) is marked as a positive sample, and the eliminated paths 2 and 3 are marked as negative samples to form several preference comparison pairs. For example, for the question Q, the positive sample is "speed x time = distance → 40 kilometers", the negative sample 1 is "hourly cumulative summation → 40 kilometers", and the negative sample 2 is "incorrect steps… → answer deviation". In this way, the model learns to optimize the target positive sample and learns that path 1 is a better reasoning method.
[0061] Step S6: Direct preference optimization model.
[0062] Finally, the above preference samples are input into the model, and the model parameters are adjusted using DPO combined with supervised fine-tuning loss. During training, the model will be rewarded to prefer the reasoning process similar to the positive sample and avoid the lengthy reasoning like the negative sample. At the same time, the consistency constraint of the standard answer is added to ensure that the model will not ignore the correctness because of the pursuit of simplicity. After several rounds of training iterations, the model gradually learns to solve similar problems with more refined steps without blindly expanding too many useless inferences.
[0063] Through the above examples, it is verified that the method of the present application can effectively reduce the redundant steps of large language models in chain reasoning. The optimized model can automatically produce a concise process close to the human optimal solution when facing problems that require multi-step reasoning. For example, for the above problem Q, the trained model will directly give the answer 40 in 2 steps of reasoning, rather than possibly listing a large number of redundant steps as before. Therefore, the scheme provided by the present application can greatly improve the reasoning efficiency while ensuring the correctness of the answer, which is of great significance for improving the performance of large language models in practical applications.
[0064] The above examples have described the technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the present application. Any modification, supplement and equivalent replacement made within the principle range of the present application shall be included in the protection scope of the present application.
Claims
1. A long-chain reasoning redundancy reduction method based on semantic structure compression and direct preference optimization, characterized by: include: (1) For the question to be answered, the pre-trained large language model generates multiple chained reasoning solution paths as candidate paths. Each candidate path includes multiple intermediate reasoning steps arranged in sequence and the final answer. (2) Merge the multiple candidate paths generated in step (1) into a tree structure as the inference decision topology; (3) Calculate the efficiency index that comprehensively considers the number of steps and text length, and select the optimal path from the candidate paths based on the efficiency index; (4) Filter the remaining non-optimal paths in the tree structure according to the set threshold conditions, and eliminate solutions that are lengthy and inefficient compared to the optimal path; (5) constructing a preference data set by combining the preferred path selected in step (3) and other inefficient paths screened out in step (4); (6) If none of the candidate paths generated in step (1) meets the correct answer condition, or if at least one correct answer path is not retained after filtering in step (4), the problem sample is considered to have no training value and is removed from the preferred data set; (7) Input the samples in the preference data set obtained in the above steps into the student model and adjust the parameters of the student model for training; during the training process, the preferred path determined in step (3) is used as an ideal reference to encourage the student model to increase the probability of generating the path while reducing the preference for inferior paths; (8) Use the trained student model to solve the problem.
2. The long-chain reasoning redundancy reduction method based on semantic structure compression and direct preference optimization according to claim 1 is characterized in that: The specific process of step (2) is: First, select the path with the least number of reasoning steps as the backbone, denoted as S min , determine its depth value as the reference depth; For the remaining candidate paths, depth-constrained node merging is performed step by step: when the semantic similarity of a candidate path in one step with an existing node in the current tree reaches a set threshold, and the depth of the merged path does not exceed the reference depth, the step of the candidate path is aligned and merged with the existing node, thereby eliminating repeated and redundant reasoning steps; if the above conditions are not met, a new branch node is added to the current tree to retain the candidate path, thereby maintaining reasoning diversity while ensuring structural compactness.
3. The long-chain reasoning redundancy reduction method based on semantic structure compression and direct preference optimization according to claim 1 is characterized in that: In step (3), the calculation formula of the efficiency index is as follows: Where, E(τ j ) represents the jth candidate path τ j The efficiency index, l token Indicates the total number of words in the path, l step Indicates the number of inference steps of the path.
4. The long-chain reasoning redundancy reduction method based on semantic structure compression and direct preference optimization according to claim 1 is characterized in that: In step (3), the optimal path is selected from the candidate paths based on the efficiency index. The specific process is as follows: The set of candidate paths that reach the expected answer and are logically correct is denoted as τ correct , the optimal path τ + According to the following formula, from the set τ correct Select and receive: When there is a unique Pareto optimal path that meets the above conditions, this path is directly selected as the preferred path; if there are multiple non-dominant candidate paths located on the Pareto front, the path with the least number of words is selected as the preferred path.
5. The long-chain reasoning redundancy reduction method based on semantic structure compression and direct preference optimization according to claim 4 is characterized in that: In step (4), the solutions that are lengthy and inefficient compared to the optimal path are eliminated. The specific process is as follows: For the set τ correct Each correct reasoning path in + When the number of steps exceeds the preset threshold and the text length exceeds the preset multiple of the preferred path, the path is judged as a redundant and long suboptimal solution and is filtered out; at the same time, any invalid solution path with a higher number of steps and words than the preferred path and an incorrect final answer is eliminated to ensure that only high-quality and efficient solution examples are retained in the preferred data.
6. The long-chain reasoning redundancy reduction method based on semantic structure compression and direct preference optimization according to claim 1 is characterized in that: In step (5), the optimal path τ + For the positive superior solution, each inefficient path that is screened out is used as the corresponding negative inferior solution, forming a number of superior-inferior pairing preference sample data to constitute the preference data set.
7. The long-chain reasoning redundancy reduction method based on semantic structure compression and direct preference optimization according to claim 1 is characterized in that: In step (7), the direct preference optimization algorithm is used to adjust and train the student model parameters.
8. The long-chain reasoning redundancy reduction method based on semantic structure compression and direct preference optimization according to claim 1 is characterized in that: In step (7), during the training process, the objective function introduces a cross-entropy supervision term based on the preference pair comparison loss to ensure that the learning model maintains strict adherence to the correct answer while pursuing concise reasoning, thereby optimizing the model's reasoning chain generation strategy.