A thought chain compression model training method and system and a medium

Abstract

A thought chain compression model training method and system and a medium, which comprises: obtaining a training sample set; clustering the training samples based on their semantic features to obtain a plurality of task clusters; constructing a multi-level step set for each training sample; for each training sample, using a large language model to perform multiple samplings to generate a plurality of reasoning paths; each reasoning path comprises a reasoning chain and a corresponding answer; for each reasoning path, calculating a step accuracy rate according to the matching degree of its reasoning chain and the multi-level step set; updating the historical average accuracy rate of the task cluster to which the current training sample belongs according to the number of reasoning paths with correct answers in the current sampling of the current training sample, and determining the reward score of each reasoning path; normalizing the reward scores of the reasoning paths in the current sampling to obtain normalized reward scores; updating the parameters of the large language model with the optimization target of maximizing the normalized reward scores to obtain a thought chain compression model.

Description

Technical Field

[0001] This invention relates to the field of mind chain compression technology, and in particular to a mind chain compression model training method, system and medium. Background Technology

[0002] Chain-of-Thought (CoT) reasoning significantly improves the performance of models in complex tasks such as mathematical reasoning, logical deduction, and common sense question answering by guiding large language models to generate intermediate reasoning steps. However, current CoT reasoning generally suffers from the problem of "overthinking": regardless of the complexity of the input, the model generates lengthy reasoning trajectories, leading to a significant increase in computational overhead and reasoning latency; moreover, redundant reasoning does not necessarily bring improved accuracy, but rather affects the model's deployment efficiency and user experience.

[0003] To reduce the cost of reasoning in thought chains, two mainstream methods have been proposed in the industry. The first method involves generating simplified reasoning data through manual design, rule templates, or by utilizing stronger large language models, which is then used to fine-tune the target model. This type of method relies heavily on manual annotation or external strong models, resulting in high costs, unstable compression effects, and a tendency for reasoning logic to be interrupted due to excessive pruning of key steps. The second method employs a reinforcement learning framework, designing reward functions to guide the model to autonomously learn simplified strategies. However, most existing reinforcement learning methods use static length penalty mechanisms, causing the model to sacrifice the integrity and causal consistency of reasoning logic in order to obtain rewards. Although some solutions introduce task complexity evaluation to dynamically adjust penalty weights, their evaluation only relies on the superficial attributes of the task itself (such as question length and domain category), without considering the differences in the model's actual mastery of different tasks. This leads to redundancy in the reasoning chain for tasks that the model is already proficient in, while the reasoning chain is easily truncated prematurely for tasks that the model has not yet mastered, resulting in the loss of key information.

[0004] In summary, existing technologies lack a mechanism that can sense the model's own capabilities and adaptively adjust the intensity of thought chain compression accordingly, thus failing to achieve differentiated compression for tasks with different levels of mastery. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method for training a mind chain compression model, comprising the following steps: Obtain a training sample set, where each training sample must include at least a problem description; The training samples are clustered based on their semantic features to obtain multiple task clusters; A multi-level set of steps is constructed for each training sample to record the expected result of each step in the multiple inference steps corresponding to that training sample. For each training sample, multiple samplings are performed using a large language model to generate multiple inference paths; each inference path includes an inference chain and the corresponding answer. For each reasoning path, the step accuracy is calculated based on the degree of matching between its reasoning chain and the multi-level step set. Based on the number of correct inference paths in the current training sample, update the historical average accuracy of the task cluster to which the current training sample belongs, and determine the reward score for each inference path; The reward scores of each reasoning path in this sampling are normalized to obtain the normalized reward scores; With maximizing the normalized reward score as the optimization objective, the parameters of the large language model are updated to obtain the thought chain compression model.

[0006] Optionally, the training samples can be clustered based on their semantic features to obtain multiple task clusters, including: The semantic vector of each training sample is extracted through a pre-trained model, and the cosine similarity between the semantic vectors is calculated. Based on the cosine similarity, a density clustering algorithm is used to divide the training samples into multiple task clusters.

[0007] Optionally, a multi-level set of steps is constructed for each training sample, including: The question description and reasoning task corresponding to each training sample are concatenated into prompt words and input into the large language model. This guides the large language model to break down the reasoning process into multiple sequentially executed sub-steps, with each sub-step associated with an expected result. Based on the response of the prompt words to the large language model, the expected results of each sub-step are stored sequentially as a multi-level step set.

[0008] Optionally, for each inference path, the step accuracy is calculated based on the degree of matching between its inference chain and the multi-level step set, including: For each reasoning path, the reasoning content corresponding to each sub-step of the multi-level step set is parsed from the reasoning chain of the reasoning path; Determine whether the reasoning content of each sub-step matches the expected result of that sub-step, and count the number of sub-steps that successfully match. The step accuracy is obtained by dividing the number of successfully matched sub-steps by the total number of sub-steps in the multi-level step set.

[0009] Optionally, based on the number of correct inference paths in the current sampling of the current training sample, update the historical average accuracy of the task cluster to which the current training sample belongs, including: Add the total number of historical sampling inferences to the task cluster to which the current training sample belongs to the total number of inference paths in this sampling to obtain the updated total number of samplings; The updated total accuracy is obtained by adding the historical sampling accuracy of the task cluster to which the current training sample belongs to the number of correct reasoning paths in the current sampling. Divide the updated total accuracy by the updated total number of samples to obtain the updated historical average accuracy.

[0010] Optionally, determine the reward score for each reasoning path, including: Determine if the number of correct reasoning paths in the current training sample is zero. If the value is zero, the step accuracy of each reasoning path is multiplied by the first preset weight and used as the reward score for the corresponding reasoning path. If it is not zero, and the historical average accuracy of the task cluster is less than the preset threshold or the number of correct reasoning paths is equal to 1, then the reward score corresponding to the correct reasoning path is set as the first preset value, and the reward score corresponding to the incorrect reasoning path is set as its step accuracy multiplied by the second preset weight. If the number of correct reasoning paths is greater than or equal to 2, and the historical average accuracy of the task cluster to which it belongs is greater than or equal to a preset threshold, then the reward score corresponding to the incorrect reasoning path is set as the second preset value. The reward score of the correct reasoning path is calculated based on its reasoning path length, wherein the reward score is negatively correlated with the reasoning path length, and the first preset value is a positive number greater than the second preset value.

[0011] Optionally, the reward score for the correct reasoning path is calculated based on its reasoning path length, including: Based on the correct reasoning path, determine whether the length of the longest reasoning path is equal to the length of the shortest reasoning path; if so, set the normalized length value to zero; if not, divide the difference between the reasoning length and the shortest reasoning path length by the difference between the longest reasoning path length and the shortest reasoning path length to obtain the normalized length value. Multiply the difference between 1 and the normalized length value by the third preset weight to obtain the reward score for the corresponding reasoning path.

[0012] Optionally, the parameters of the large language model are updated with the goal of maximizing the normalized reward score, including: Based on the normalized reward scores of each inference path, calculate the policy gradient for this sampling. Based on the policy gradient, the thought chain generation strategy of the large language model is updated using a sequence-level or token-level optimization algorithm to obtain the updated model parameters.

[0013] Corresponding to the aforementioned method for training a mind chain compression model, this invention provides a training system for a mind chain compression model, comprising: The sample acquisition module is used to acquire the training sample set, and each training sample includes at least a problem description; The clustering module is used to cluster training samples based on the semantic features of each training sample to obtain multiple task clusters; A step set construction module is used to construct a multi-level step set for each training sample. The multi-level step set is used to record the expected result of each step in the multiple inference steps corresponding to the training sample. The sampling module is used to sample each training sample multiple times using a large language model to generate multiple inference paths; each inference path includes an inference chain and the corresponding answer. The calculation module is used to calculate the step accuracy for each inference path based on the degree of matching between its inference chain and the multi-level step set; The update module is used to update the historical average accuracy of the task cluster to which the current training sample belongs based on the number of correct inference paths in the current sampling of the current training sample. The reward score determination module is used to determine the reward score for each reasoning path; The normalization module is used to normalize the reward scores of each inference path in this sampling to obtain the normalized reward scores; The model generation module is used to update the parameters of the large language model with the optimization objective of maximizing the normalized reward score, thereby obtaining the thought chain compression model.

[0014] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a mind chain compression model training program, which, when executed by a processor, implements the steps of the mind chain compression model training method described above.

[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) Task clusters are obtained by clustering based on the semantic features of training samples. Multi-level step sets are constructed for each training sample. Multiple inference paths are generated by sampling multiple times using a large language model. The step accuracy is calculated based on the matching degree between the inference chain and the multi-level step set. The reward score of each inference path is dynamically determined and normalized in combination with the historical average accuracy of the task cluster. The parameters of the large language model are updated with the goal of maximizing the normalized reward score. This allows the model to perceive its mastery of different task clusters during training and adaptively adjust the compression intensity of the thought chain accordingly. This solves the technical problem that existing technologies cannot achieve differentiated compression for tasks with different mastery levels. At the same time, it avoids the high cost and unstable compression effect caused by relying on manual annotation or external strong models.

[0016] (2) Semantic vectors are extracted by pre-trained models and cosine similarity is calculated. Density clustering algorithm is used to divide task clusters so that training samples with similar semantics are grouped into the same task cluster. This ensures that the training samples in the same task cluster have inherent consistency in reasoning difficulty and required reasoning steps, providing a reliable task grouping basis for subsequent differentiated reward allocation based on the historical average accuracy of task clusters.

[0017] (3) By concatenating the problem description and reasoning task corresponding to each training sample into prompt words, the large language model is guided to break down the reasoning process into multiple sequentially executed sub-steps, so that the reasoning chain of each reasoning path can be parsed and evaluated step by step, thereby providing a clear reference standard for the accuracy of subsequent calculation steps and ensuring the ability to finely measure the logical integrity of the reasoning chain.

[0018] (4) By parsing out the reasoning content corresponding to each sub-step from the reasoning chain, judging whether the reasoning content matches the expected result and counting the number of successful matches, the number of successful matches is divided by the total number of sub-steps to obtain the step accuracy rate, so that the step accuracy rate can directly reflect the integrity and correctness of the reasoning chain at the logical step level, and provides a quantitative basis directly related to the reasoning logic quality for determining the reward score.

[0019] (5) The updated total number of samplings is obtained by adding the sum of the historical sampling inferences to the total number of inference paths in the current sampling. The updated total number of accurate results is obtained by adding the number of correct inference paths in the current sampling. The updated total number of accurate results is then divided by the updated total number of samplings to obtain the updated historical average accuracy. This allows the historical average accuracy of the task cluster to accumulate and update continuously during the training process, thereby dynamically reflecting the changes in the model's mastery of the task cluster and providing a real-time and reliable capability perception indicator for subsequent differentiated reward allocation.

[0020] (6) By judging whether the number of correct reasoning paths is zero, and when it is zero, the step accuracy multiplied by the first preset weight is used as the reward score; when the number of correct answers is not zero and the historical average accuracy of the task cluster is less than the preset threshold or the number of correct answers is equal to 1, the reward score of the correct reasoning path is set to a higher first preset value, and the reward score of the incorrect answer is set to the step accuracy multiplied by the second preset weight; when the number of correct answers is greater than or equal to 2 and the historical average accuracy is greater than or equal to the preset threshold, the reward score of the incorrect answer is set to a lower second preset value, and the reward score of the correct answer is calculated based on the negative correlation of the reasoning path length, so that the reward mechanism can respond in a coordinated manner to the two dimensions of the model's mastery of the task (historical average accuracy) and the current sampling quality (number of correct answers): when the model has insufficient mastery of the task, priority is given to ensuring the integrity of the reasoning logic and the correctness of the answer; when the model has mastered the task, the reward score with the negative correlation of the reasoning path length guides the model to compress the reasoning path length, thus realizing the adaptive matching between the compression intensity and the model's ability.

[0021] (7) By judging whether the length of the longest inference path is equal to the length of the shortest inference path, the normalized length value is set to zero when they are equal, and the normalized length value is obtained by dividing the difference between the length of the inference path and the length of the shortest inference path by the difference between the length of the longest inference path and the length of the shortest inference path when they are not equal. Then, the difference between 1 and the normalized length value is multiplied by the third preset weight to obtain the reward score. This allows the reward score of the correct inference path to be compared in relative length within the same sampling batch. The shorter the inference chain, the higher the reward score. Thus, a refined length competition mechanism is established within the same task cluster, guiding the model to actively choose a shorter inference path while maintaining the correct answer.

[0022] (8) The policy gradient of this sampling is calculated based on the normalized reward score of each reasoning path, and the thought chain generation strategy of the large language model is updated by using sequence-level or token-level optimization algorithms, so that the model parameters can directly respond to the guidance of the aforementioned differentiated reward signals, and internalize the task cluster perception ability and reasoning path length preference into the model's generation strategy, thereby realizing the continuous optimization and solidification of the thought chain compression capability at the parameter level. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a simplified flowchart of an embodiment of the mind chain compression model training method of the present invention; Figure 2 This is a framework diagram of an embodiment of the mind chain compression model training system of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] like Figure 1 As shown, a method for training a thought chain compression model according to the present invention includes the following steps: Obtain a training sample set, where each training sample must include at least a problem description; The training samples are clustered based on their semantic features to obtain multiple task clusters; A multi-level set of steps is constructed for each training sample to record the expected result of each step in the multiple inference steps corresponding to that training sample. For each training sample, multiple samplings are performed using a large language model to generate multiple inference paths; each inference path includes an inference chain and the corresponding answer; preferably, the number of samplings per sample is 32, and the large language model is the Qwen3-14B model; For each reasoning path, the step accuracy is calculated based on the degree of matching between its reasoning chain and the multi-level step set; Based on the number of correct inference paths in the current training sample, update the historical average accuracy of the task cluster to which the current training sample belongs, and determine the reward score for each inference path; The reward scores of each reasoning path in this sampling are normalized to obtain the normalized reward scores; By maximizing the normalized reward score as the optimization objective, the parameters of the large language model are updated to obtain the thought chain compression model.

[0026] This invention obtains task clusters by clustering semantic features based on training samples, constructs a multi-level step set for each training sample, and generates multiple inference paths through multiple sampling using a large language model. The step accuracy is calculated based on the matching degree between the inference chain and the multi-level step set. The reward score for each inference path is dynamically determined and normalized by combining the historical average accuracy of the task clusters. The parameters of the large language model are updated with the optimization objective of maximizing the normalized reward score. This allows the model to perceive its mastery of different task clusters during training and adaptively adjust the compression intensity of the thought chain accordingly. This solves the technical problem of existing technologies being unable to achieve differentiated compression for tasks with different levels of mastery, while avoiding the high cost and unstable compression effects associated with relying on manual annotation or external strong models. This invention is a training method for the RL stage of a large language model, so it has a certain degree of universality for large language models. The Qwen3-14B model is only an example and does not constitute an undue limitation on this invention. Training employs reinforcement learning methods such as GRPO and GSPO.

[0027] In this embodiment, the training samples are clustered based on their semantic features to obtain multiple task clusters, including: The semantic vector of each training sample is extracted through a pre-trained model, and the cosine similarity between the semantic vectors is calculated. Preferably, the pre-trained model is a bge-m or BERT model, which can have end-to-end vector output and does not involve the extraction of intermediate layers. Based on cosine similarity, a density clustering algorithm is used to divide the training samples into multiple task clusters.

[0028] This invention extracts semantic vectors and calculates cosine similarity through a pre-trained model, and uses a density clustering algorithm to divide task clusters, so that training samples with similar semantics are grouped into the same task cluster. This ensures that training samples within the same task cluster have inherent consistency in inference difficulty and required inference steps, providing a reliable task grouping basis for subsequent differentiated reward allocation based on the historical average accuracy of task clusters.

[0029] In this embodiment, a multi-level set of steps is constructed for each training sample, including: The question description and reasoning task corresponding to each training sample are concatenated into prompt words and input into the large language model. This guides the large language model to break down the reasoning process into multiple sequentially executed sub-steps, with each sub-step associated with an expected result. Based on the response of the large language model to the prompt words, the expected results of each sub-step are stored sequentially as a multi-level set of steps.

[0030] For example, the output of a structured task that extracts names and addresses can be broken down into the following four steps: a) Generate the reasoning content as required, and separate the reasoning process from the final result using preset markers (e.g., using similar methods). <think>< / think> (label format); b) Extract the names; c) Extract place names; d) Output according to the required format, such as using JSON format, like {"Name":"Zhang San","Address":"Beijing"}. The final output should be ["Yes","Zhang San","Beijing","Yes"]. The results of a and d can be saved as "Yes" or "No".

[0031] Alternatively, taking a mathematical reasoning problem as an example, it can be broken down into the following three steps: 1) Generate reasoning content as required, and separate the reasoning process from the final result using preset markers (e.g., using similar methods). <think>< / think> (label format); 2) Calculate the answer; 3) Output according to the required format, such as using JSON format; e.g., {"Answer": 2026}. The final value is ["Yes", "2026", "Yes"]. The results from steps 1 and 3 can be saved as "Yes" or "No".

[0032] The above description uses two specific implementation methods as examples. The marking format and step division method can be adjusted according to actual task requirements.

[0033] This invention uses the problem description and reasoning task corresponding to each training sample to concatenate prompt words into a large language model, which then guides the large language model to break down the reasoning process into multiple sequentially executed sub-steps. This allows the reasoning chain of each reasoning path to be parsed and evaluated step by step, thereby providing a clear reference standard for the accuracy of subsequent calculation steps and ensuring the ability to finely measure the logical integrity of the reasoning chain.

[0034] In this embodiment, for each inference path, the step accuracy is calculated based on the degree of matching between its inference chain and the multi-level step set, including: For each reasoning path, the reasoning content corresponding to each sub-step of the multi-level step set is parsed from the reasoning chain of the reasoning path; Determine whether the reasoning content of each sub-step matches the expected result of that sub-step, and count the number of sub-steps that successfully match. The step accuracy is obtained by dividing the number of successfully matched sub-steps by the total number of sub-steps in the multi-level step set.

[0035] This invention parses the reasoning content corresponding to each sub-step from the reasoning chain, determines whether the reasoning content matches the expected result, and counts the number of successful matches. The step accuracy rate is obtained by dividing the number of successful matches by the total number of sub-steps. This allows the step accuracy rate to directly reflect the completeness and correctness of the reasoning chain at the logical step level, providing a quantitative basis directly related to the quality of reasoning logic for determining the reward score.

[0036] In this embodiment, the historical average accuracy of the task cluster to which the current training sample belongs is updated based on the number of correct inference paths in the current sampling, including: Add the total number of historical sampling inferences to the task cluster to which the current training sample belongs to the total number of inference paths in this sampling to obtain the updated total number of samplings; The updated total accuracy is obtained by adding the historical sampling accuracy of the task cluster to which the current training sample belongs to the number of correct reasoning paths in the current sampling. Divide the updated total accuracy by the updated total number of samples to obtain the updated historical average accuracy.

[0037] This invention obtains the updated total number of samplings by adding the sum of the historical sampling inference counts to the total number of inference paths in the current sampling; it obtains the updated total accuracy by adding the historical accuracy count to the number of correct inference paths in the current sampling; and it obtains the updated historical average accuracy by dividing the updated total accuracy count by the updated total number of samplings. This allows the historical average accuracy of a task cluster to continuously accumulate and update during the training process, thereby dynamically reflecting the changes in the model's mastery of the task cluster and providing a real-time and reliable capability perception indicator for subsequent differentiated reward allocation.

[0038] In this embodiment, determining the reward score for each inference path includes: Determine if the number of correct reasoning paths in the current training sample is zero. If the value is zero, the step accuracy of each reasoning path is multiplied by the first preset weight and used as the reward score for the corresponding reasoning path. If the value is not zero, and the historical average accuracy of the task cluster is less than the preset threshold or the number of correct reasoning paths is equal to 1, then the reward score corresponding to the correct reasoning path is set to the first preset value (preferably 1.0), and the reward score corresponding to the incorrect reasoning path is set to its step accuracy multiplied by the second preset weight. If the number of correct reasoning paths is greater than or equal to 2, and the historical average accuracy of the task cluster is greater than or equal to a preset threshold, then the reward score corresponding to the incorrect reasoning path is set to the second preset value (preferably 0). The reward score of the correct reasoning path is calculated based on its reasoning path length, wherein the reward score is negatively correlated with the reasoning path length, and the first preset value is a positive number greater than the second preset value.

[0039] Preferably, the reward score calculation formula for each reasoning path is as follows: ; in, This represents the inference path obtained from different samplings of the sample, where n represents the nth inference path among multiple samplings; Represents the reward score for different sampling inference paths; This indicates the accuracy of each step in the reasoning path; the key has no actual meaning. This represents the first preset weight, which is preferably 1; This represents the number of correct inference paths in the current training sample during this sampling. This represents the historical average accuracy of the task cluster to which it belongs; This represents a preset threshold value, preferably 0.95. This indicates the second preset weight, preferably 0.8; This indicates that the nth reasoning path is labeled with the true result (i.e., the standard answer). This represents the result of the i-th sample (i.e., the actual answer output by the model). Indicates the length of the inference path for a single sample; Indicates the length of the shortest reasoning path; Indicates the length of the longest reasoning path; This represents the third preset weight, which is preferably 0.85.

[0040] This invention determines whether the number of correct inference paths is zero, and if so, uses the step accuracy multiplied by a first preset weight as the reward score. When the number of correct answers is not zero and the historical average accuracy of the task cluster is less than a preset threshold or the number of correct answers is equal to 1, the reward score for the correct inference path is set to a higher first preset value, and the reward score for the incorrect answer is set to the step accuracy multiplied by a second preset weight. When the number of correct answers is greater than or equal to 2 and the historical average accuracy is greater than or equal to a preset threshold, the reward score for the incorrect inference path is set to a lower second preset value, and the reward score for the correct answer is calculated based on the negative correlation between the inference path length and the reward score. This allows the reward mechanism to respond synergistically to information from two dimensions: the model's mastery of the task (historical average accuracy) and the current sampling quality (number of correct answers). When the model's mastery of the task is insufficient, priority is given to ensuring the integrity of the inference logic and the correctness of the answer. When the model has mastered the task, the reward score, which is negatively correlated with the inference path length, guides the model to compress the inference path length, achieving an adaptive match between compression intensity and model capability.

[0041] In this embodiment, the reward score for the correct reasoning path is calculated based on its reasoning path length, including: Based on the correct reasoning path, determine whether the length of the longest reasoning path is equal to the length of the shortest reasoning path; if so, set the normalized length value to zero; if not, divide the difference between the reasoning length and the shortest reasoning path length by the difference between the longest reasoning path length and the shortest reasoning path length to obtain the normalized length value. Multiply the difference between 1 and the normalized length value by the third preset weight to obtain the reward score for the corresponding reasoning path.

[0042] This invention determines whether the length of the longest inference path is equal to the length of the shortest inference path. When they are equal, the normalized length value is set to zero; when they are not equal, the difference between the inference length and the shortest inference path length is divided by the difference between the length of the longest inference path and the shortest inference path length to obtain the normalized length value. Then, the difference between 1 and the normalized length value is multiplied by a third preset weight to obtain the reward score. This allows the reward score of the correct inference path to be compared relative in length within the same sampling batch. The shorter the inference chain, the higher the reward score. Thus, a refined length competition mechanism is established within the same task cluster, guiding the model to actively choose a shorter inference path while maintaining the correctness of the answer.

[0043] In this embodiment, the parameters of the large language model are updated with the goal of maximizing the normalized reward score, including: Based on the normalized reward scores of each inference path, the policy gradient for this sampling is calculated. The calculation formula is as follows: ;in, This represents the maximum value of all reward scores in this sampling. This is the minimum value of all reward scores in this sampling; Based on the policy gradient, the thought chain generation strategy of the large language model is updated using sequence-level or token-level optimization algorithms to obtain the updated model parameters. The specific update objectives are as follows: Where π represents the inference path generation strategy of the current model; Under the current inference generation strategy of the model, the mathematical expectation of the reward score of the sampled inference path; C refers to a complete sequence of inference paths.

[0044] This invention calculates the policy gradient of the current sampling based on the normalized reward score of each inference path, and updates the thought chain generation strategy of the large language model using sequence-level or token-level optimization algorithms. This allows the model parameters to directly respond to the guidance of the aforementioned differentiated reward signals, internalizes the task cluster perception ability and inference path length preference into the model's generation strategy, and thus achieves continuous optimization and solidification of the thought chain compression capability at the parameter level.

[0045] like Figure 2 As shown, the present invention also provides a training system for a mind chain compression model, which includes: The sample acquisition module 10 is used to acquire a training sample set, and each training sample includes at least a problem description. Clustering module 20 is used to cluster training samples based on the semantic features of each training sample to obtain multiple task clusters; The step set construction module 30 is used to construct a multi-level step set for each training sample. The multi-level step set is used to record the expected result of each step in the multiple inference steps corresponding to the training sample. The sampling module 40 is used to sample each training sample multiple times using a large language model to generate multiple inference paths; each inference path includes an inference chain and the corresponding answer; The calculation module 50 is used to calculate the step accuracy for each reasoning path based on the degree of matching between its reasoning chain and the multi-level step set. Update module 60 is used to update the historical average accuracy of the task cluster to which the current training sample belongs based on the number of correct inference paths in the current sampling of the current training sample. The reward score determination module 70 is used to determine the reward score for each reasoning path; The normalization module 80 is used to normalize the reward scores of each inference path in this sampling to obtain the normalized reward scores; The model generation module 90 is used to update the parameters of the large language model with the optimization objective of maximizing the normalized reward score, so as to obtain the thought chain compression model.

[0046] This invention also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement... Figure 1 The training method for the thought chain compression model is shown. The computer-readable storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0047] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments and storage medium embodiments, since they are basically similar to method embodiments, the descriptions are relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0048] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0049] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A training method for a thought chain compression model, characterized in that, Includes the following steps: Obtain a training sample set, where each training sample must include at least a problem description; The training samples are clustered based on their semantic features to obtain multiple task clusters; A multi-level set of steps is constructed for each training sample to record the expected result of each step in the multiple inference steps corresponding to that training sample. For each training sample, multiple samplings are performed using a large language model to generate multiple inference paths; each inference path includes an inference chain and the corresponding answer. For each reasoning path, the step accuracy is calculated based on the degree of matching between its reasoning chain and the multi-level step set; Based on the number of correct inference paths in the current training sample, update the historical average accuracy of the task cluster to which the current training sample belongs, and determine the reward score for each inference path; The reward scores of each reasoning path in this sampling are normalized to obtain the normalized reward scores; With maximizing the normalized reward score as the optimization objective, the parameters of the large language model are updated to obtain the thought chain compression model.

2. The training method for the mind chain compression model according to claim 1, characterized in that, The training samples are clustered based on their semantic features to obtain multiple task clusters, including: The semantic vector of each training sample is extracted through a pre-trained model, and the cosine similarity between the semantic vectors is calculated. Based on the cosine similarity, a density clustering algorithm is used to divide the training samples into multiple task clusters.

3. The training method for the mind chain compression model according to claim 1, characterized in that, Construct a multi-level set of steps for each training sample, including: The problem description and reasoning task corresponding to each training sample are concatenated into prompt words and input into the large language model. This guides the large language model to break down the reasoning process into multiple sequentially executed sub-steps, with each sub-step associated with an expected result. Based on the response of the prompt words to the large language model, the expected results of each sub-step are stored sequentially as a multi-level step set.

4. The training method for the mind chain compression model according to claim 1, characterized in that, For each inference path, the step accuracy is calculated based on the degree of matching between its inference chain and the multi-level step set, including: For each reasoning path, the reasoning content corresponding to each sub-step of the multi-level step set is parsed from the reasoning chain of the reasoning path; Determine whether the reasoning content of each sub-step matches the expected result of that sub-step, and count the number of sub-steps that successfully match. The step accuracy is obtained by dividing the number of successfully matched sub-steps by the total number of sub-steps in the multi-level step set.

5. The training method for the mind chain compression model according to claim 1, characterized in that, Based on the number of correct inference paths in the current training sample, update the historical average accuracy of the task cluster to which the current training sample belongs, including: Add the total number of historical sampling inferences to the task cluster to which the current training sample belongs to the total number of inference paths in this sampling to obtain the updated total number of samplings; The updated total accuracy is obtained by adding the historical sampling accuracy of the task cluster to which the current training sample belongs to the number of correct reasoning paths in the current sampling. Divide the updated total accuracy by the updated total number of samples to obtain the updated historical average accuracy.

6. The training method for the mind chain compression model according to claim 1, characterized in that, Determine the reward score for each reasoning path, including: Determine if the number of correct reasoning paths in the current training sample is zero. If the value is zero, the step accuracy of each reasoning path is multiplied by the first preset weight and used as the reward score for the corresponding reasoning path. If it is not zero, and the historical average accuracy of the task cluster is less than the preset threshold or the number of correct reasoning paths is equal to 1, then the reward score corresponding to the correct reasoning path is set as the first preset value, and the reward score corresponding to the incorrect reasoning path is set as its step accuracy multiplied by the second preset weight. If the number of correct reasoning paths is greater than or equal to 2, and the historical average accuracy of the task cluster to which it belongs is greater than or equal to a preset threshold, then the reward score corresponding to the incorrect reasoning path is set as the second preset value. The reward score of the correct reasoning path is calculated based on its reasoning path length, wherein the reward score is negatively correlated with the reasoning path length, and the first preset value is a positive number greater than the second preset value.

7. The training method for the mind chain compression model according to claim 6, characterized in that, The reward score for the correct reasoning path is calculated based on its length, including: Based on the correct reasoning path, determine whether the length of the longest reasoning path is equal to the length of the shortest reasoning path; if so, set the normalized length value to zero; if not, divide the difference between the reasoning length and the shortest reasoning path length by the difference between the longest reasoning path length and the shortest reasoning path length to obtain the normalized length value. Multiply the difference between 1 and the normalized length value by the third preset weight to obtain the reward score for the corresponding reasoning path.

8. The training method for the mind chain compression model according to claim 1, characterized in that, The parameters of the large language model are updated with the goal of maximizing the normalized reward score, including: Based on the normalized reward scores of each inference path, calculate the policy gradient for this sampling. Based on the policy gradient, the thought chain generation strategy of the large language model is updated using a sequence-level or token-level optimization algorithm to obtain the updated model parameters.

9. A training system for a mind chain compression model, characterized in that, include: The sample acquisition module is used to acquire the training sample set, and each training sample includes at least a problem description; The clustering module is used to cluster training samples based on the semantic features of each training sample to obtain multiple task clusters; A step set construction module is used to construct a multi-level step set for each training sample. The multi-level step set is used to record the expected result of each step in the multiple inference steps corresponding to the training sample. The sampling module is used to sample each training sample multiple times using a large language model to generate multiple inference paths; each inference path includes an inference chain and the corresponding answer. The calculation module is used to calculate the step accuracy for each inference path based on the degree of matching between its inference chain and the multi-level step set; The update module is used to update the historical average accuracy of the task cluster to which the current training sample belongs based on the number of correct inference paths in the current sampling of the current training sample. The reward score determination module is used to determine the reward score for each reasoning path; The normalization module is used to normalize the reward scores of each inference path in this sampling to obtain the normalized reward scores; The model generation module is used to update the parameters of the large language model with the optimization objective of maximizing the normalized reward score, thereby obtaining the thought chain compression model.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a mind chain compression model training program, which, when executed by a processor, implements the steps of the mind chain compression model training method as described in any one of claims 1 to 8.

Images