Large language model training and inference method based on mixed field entropy dynamic alignment

Abstract

The application provides a large language model training and reasoning method based on mixed field entropy dynamic alignment. The training method comprises the following steps: acquiring a general field data set and a target field data set; generating multiple reasoning tracks corresponding to each general field sample in the general field data set and calculating corresponding reasoning uncertainty and exploration diversity; obtaining a comprehensive value score of each general field sample according to the reasoning uncertainty and the exploration diversity, and selecting the top k general field samples with the highest scores to obtain a selected general field data set; in each iteration training, a batch of data is sampled from the mixed data set for track generation and entropy dynamic extraction to obtain an entropy dynamic set of the target field sample and an entropy dynamic set of the general field sample, and an entropy dynamic alignment reward is calculated to update the parameters of the strategy model until the preset training condition is met to obtain a trained large language model; and the accuracy of the reasoning result is improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and machine learning technology, and in particular to a method for training a large language model based on dynamic alignment of hybrid domain entropy and a method for inferring a large language model based on dynamic alignment of hybrid domain entropy. Background Technology

[0002] In recent years, reasoning systems based on large language models (such as OpenAI o1 and DeepSeek-R1) have made breakthrough progress in complex reasoning tasks. Reinforcement Learning with Verifiable Rewards (RLVR) is one of the key driving forces behind this advancement. This technique utilizes rule-based binary reward signals related to the correctness of the final answer to train the model. This approach eliminates the cost of training complex reward models, thus demonstrating superior performance in data-rich fields such as mathematics.

[0003] However, existing RLVR technologies and research suffer from a fundamental limitation: their success heavily relies on the "high-resource" assumption, requiring a large number of high-quality training samples from the target domain. However, in many important real-world target domains (such as medical reasoning, law, and other specific scientific fields), a large number of high-quality training samples have not yet been accumulated. In other words, in these low-resource scenarios, target domain samples suitable for RLVR training are extremely scarce, leading to the following serious deficiencies in existing technical solutions: 1. Severe Entropy Collapse and Limited Exploration: Due to the scarcity of training samples, the policy model quickly overfits to a few successful inference trajectories, causing the model to lose exploration diversity early in training. Compared to high-resource scenarios, the entropy value of low-resource RLVR is significantly reduced, resulting in a more severe "entropy collapse" phenomenon. This fundamentally limits the model's ability to explore better solutions, leading to a decline in final inference performance.

[0004] 2. Limitations of Existing Entropy Enhancement Methods: To alleviate entropy collapse, various entropy regularization methods have been proposed. However, these methods typically do not fully consider the scarcity of training data. Directly applying these methods in low-resource scenarios often yields poor or suboptimal results because they may indiscriminately increase entropy values, thereby interfering with the exploration effectiveness of the policy model and potentially leading to training instability. They fail to achieve an effective balance between exploration and utilization under scarce samples.

[0005] In summary, existing RLVR technologies struggle to perform effective model training under conditions of scarce training data in the target domain. The core problem lies in the severe and uncontrollable entropy collapse caused by insufficient data, leading to premature exhaustion of the model's exploratory capabilities. Therefore, there is an urgent need for an RLVR training framework specifically designed for low-resource scenarios to effectively maintain and guide the model's exploratory behavior with limited samples, thereby improving its final inference performance. Summary of the Invention

[0006] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. To this end, one objective of this invention is to propose a large language model training method based on hybrid domain entropy dynamic alignment. By constructing a hybrid domain entropy dynamic alignment framework, this method effectively alleviates the entropy collapse problem in low-resource target domain RLVR, thereby improving the accuracy of inference results.

[0007] The second objective of this invention is to propose a large language model reasoning method based on dynamic alignment of hybrid domain entropy.

[0008] To achieve the above objectives, a first aspect of this invention proposes a method for training a large language model based on dynamic alignment of entropy across hybrid domains. This method includes the following steps: acquiring a general domain dataset and a target domain dataset; generating multiple inference trajectories for each general domain sample in the general domain dataset using a policy model to be trained; calculating the inference uncertainty and exploration diversity corresponding to each general domain sample based on the multiple inference trajectories; obtaining a comprehensive value score for each general domain sample based on the inference uncertainty and exploration diversity, and selecting the k general domain samples with the highest scores to obtain a selected general domain dataset; in each iteration of training, sampling a batch of data from the hybrid dataset for trajectory generation and dynamic entropy extraction to obtain a dynamic entropy set for the target domain samples and a dynamic entropy set for the general domain samples, wherein the hybrid dataset includes the selected general domain dataset and the target domain dataset; calculating the dynamic entropy alignment reward based on the dynamic entropy set of the target domain samples and the dynamic entropy set of the general domain samples, and updating the policy model parameters until a preset training condition is met to obtain a trained large language model.

[0009] The large language model training method based on dynamic alignment of hybrid domain entropy according to embodiments of the present invention has the advantage of solving the problem of severe entropy collapse and limited exploration ability caused by rapid overfitting of the model when the target domain training data is scarce in existing RLVR technology, thereby improving the accuracy of inference results.

[0010] In addition, the large language model training method based on dynamic alignment of hybrid domain entropy proposed in the above embodiments of the present invention may also have the following additional technical features: Optionally, the inference uncertainty corresponding to each general domain sample x is calculated according to the following formula. :

[0011]

[0012] in, Indicates an indicator function, This represents the true answer. This indicates the number of inference trajectories. Let represent the i-th inference trajectory. The closer the uncertainty score is to 1, the more uncertain the model's prediction for that sample is, and the higher its value.

[0013] Optionally, the exploration diversity corresponding to each general domain sample is calculated based on the multiple inference trajectories, including: selecting the top 20% of words with the highest entropy values ​​in each inference trajectory and calculating the average entropy value of these words; averaging the average entropy values ​​of the multiple inference trajectories to obtain the exploration diversity score of each general domain sample.

[0014] Optionally, obtaining the comprehensive value score of each general domain sample based on the inference uncertainty and the exploration diversity includes: obtaining the comprehensive value score of each general domain sample based on the product of the inference uncertainty and the exploration diversity.

[0015] Optionally, a batch of data is sampled from the mixed dataset for trajectory generation and entropy dynamic extraction to obtain the entropy dynamic set of the target domain samples and the entropy dynamic set of the general domain samples. This includes: generating multiple corresponding trajectories for each sample in the batch using the current strategy; calculating the word probability distribution entropy of each trajectory at each generation time step to obtain the entropy dynamic; and obtaining the entropy dynamic set of the target domain samples and the entropy dynamic set of the general domain samples based on the entropy dynamic corresponding to each trajectory.

[0016] Optionally, the entropy dynamic alignment reward is calculated based on the entropy dynamic set of the target domain samples and the entropy dynamic set of the general domain samples, including: for each entropy dynamic in the entropy dynamic set of the target domain samples, calculating its similarity with other entropy dynamics within the domain, and taking the maximum value to obtain the intra-domain similarity; for each entropy dynamic in the entropy dynamic set of the target domain samples, calculating its similarity with all entropy dynamics in the entropy dynamic set of the general domain samples, and taking the maximum value to obtain the inter-domain similarity; and obtaining the corresponding entropy dynamic alignment reward based on the comparison results of the intra-domain similarity and the inter-domain similarity.

[0017] Alternatively, the entropy dynamic alignment reward can be obtained according to the following formula:

[0018] Where, if the i-th trajectory Entropy dynamics Inter-domain similarity Greater than intra-domain similarity At that time, entropy dynamically aligns rewards If it is 1, otherwise it is the entropy dynamic alignment reward. It is 0.

[0019] To achieve the above objectives, a second aspect of the present invention proposes a large language model reasoning method based on dynamic alignment of hybrid domain entropy, comprising the following steps: obtaining a problem to be reasoned; inputting the problem to be reasoned into a trained large language model to obtain a corresponding reasoning result, wherein the trained large language model is trained using the training method described in any of the first aspects. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a large language model training method based on dynamic alignment of hybrid domain entropy according to an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the selective filtering of general domain data according to an embodiment of the present invention; Figure 3 This is a schematic diagram of hybrid training based on entropy dynamic alignment according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the large language model inference method based on dynamic alignment of hybrid domain entropy according to an embodiment of the present invention. Detailed Implementation

[0021] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0022] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0023] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0024] Figure 1This is a flowchart illustrating a large language model training method based on dynamic alignment of hybrid domain entropy according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method for training a large language model based on dynamic alignment of hybrid domain entropy in this embodiment of the invention includes the following steps: S101, obtain the general domain dataset and the target domain dataset.

[0025] As an example, a large-scale general-domain dataset is obtained. (e.g., CommonsenseQA), and a very small target domain dataset. (For example, medical question-and-answer data containing only 32 samples).

[0026] S102, for each general domain sample in the general domain dataset, use the policy model to be trained to generate multiple inference trajectories corresponding to it.

[0027] In other words, such as Figure 2 As shown, for Each sample Use the current policy model to be trained Generate for it Reasoning trajectory .

[0028] S103 calculates the inference uncertainty and exploration diversity corresponding to each general domain sample based on multiple inference trajectories.

[0029] As an example, each general domain sample is calculated according to the following formula. Corresponding inference uncertainty :

[0030]

[0031] in, Indicates an indicator function, This represents the true answer. This indicates the number of inference trajectories. Let represent the i-th inference trajectory. The closer the uncertainty score is to 1, the more uncertain the model's prediction for that sample is, and the higher its value.

[0032] As an example, the exploration diversity corresponding to each general domain sample is calculated based on multiple inference trajectories, including: selecting the top 20% of words with the highest entropy values ​​in each inference trajectory and calculating the average entropy value of these words; averaging the average entropy values ​​of multiple inference trajectories to obtain the exploration diversity score of each general domain sample.

[0033] In other words, for the same sample Each trajectory Select the top 20% of words with the highest entropy values, calculate the average entropy value of these words, and denot it as the average entropy of high-entropy words in this trajectory. Then, for all... The average of this value for each trajectory is used to obtain the sample. Exploration of diversity score The higher the score, the more diverse the exploratory behaviors elicited by the sample.

[0034] S104. Based on the inference uncertainty and exploration diversity, the comprehensive value score of each general domain sample is obtained, and the k general domain samples with the highest scores are selected to obtain the selected general domain dataset.

[0035] As an example, the comprehensive value score of each general domain sample is obtained based on reasoning uncertainty and exploration diversity, including: obtaining the comprehensive value score of each general domain sample based on the product of reasoning uncertainty and exploration diversity.

[0036] Specifically, calculate each general domain sample Overall value score:

[0037] according to For all The samples were sorted in descending order, and the samples with the highest scores were selected. A total of samples constitute a carefully selected general domain dataset. .

[0038] S105, in each iteration of training, a batch of data is sampled from the hybrid dataset for trajectory generation and dynamic entropy extraction to obtain the dynamic entropy set of the target domain samples and the dynamic entropy set of the general domain samples. The hybrid dataset includes a selected general domain dataset and a target domain dataset.

[0039] As an example, a batch of data is sampled from a mixed dataset for trajectory generation and entropy dynamic extraction to obtain the entropy dynamic set of target domain samples and the entropy dynamic set of general domain samples. This includes: generating multiple corresponding trajectories for each sample in the batch using the current strategy; calculating the word probability distribution entropy of each trajectory at each generation time step to obtain the entropy dynamic; and obtaining the entropy dynamic set of target domain samples and the entropy dynamic set of general domain samples based on the entropy dynamic corresponding to each trajectory.

[0040] As a specific example, such as Figure 3 As shown, in each training iteration, from the mixed dataset Sample a batch of data. For each sample in the batch... Use the current strategy Generate multiple trajectories. For each generated trajectory... Calculate its value at each generation time step Entropy of lexical probability distribution The entropy dynamics of this trajectory Defined as this sequence of entropy values: Collect the dynamic entropy sets from samples in the target domain. and the dynamic set of entropy from samples from the general domain .

[0041] S106: Calculate the entropy dynamic alignment reward based on the dynamic entropy set of the target domain samples and the dynamic entropy set of the general domain samples, and update the policy model parameters until the preset training conditions are met to obtain a well-trained large language model.

[0042] As one embodiment, the entropy dynamic alignment reward is calculated based on the entropy dynamic set of the target domain samples and the entropy dynamic set of the general domain samples. This includes: for each entropy dynamic in the entropy dynamic set of the target domain samples, calculating its similarity with other entropy dynamics within the domain, and taking the maximum value to obtain the intra-domain similarity; for each entropy dynamic in the entropy dynamic set of the target domain samples, calculating its similarity with all entropy dynamics in the entropy dynamic set of the general domain samples, and taking the maximum value to obtain the inter-domain similarity; and obtaining the corresponding entropy dynamic alignment reward based on the comparison results of the intra-domain similarity and the inter-domain similarity.

[0043] As an example, the entropy dynamic alignment reward is obtained according to the following formula:

[0044] Where, if the i-th trajectory Entropy dynamics Inter-domain similarity Greater than intra-domain similarity At that time, entropy dynamically aligns rewards If it is 1, otherwise it is the entropy dynamic alignment reward. It is 0.

[0045] As a specific example, for Each entropy dynamic (Corresponding to a target domain trajectory): a. Calculate intra-domain similarity: Calculate and All other entropy dynamics The similarity is taken as the maximum value:

[0046] in, This is the similarity function.

[0047] b. Calculate inter-domain similarity: Calculate and All entropy dynamics The similarity is taken as the maximum value:

[0048] c. Allocate EDA rewards: Based on the comparison results, assign rewards to the trajectory. Assign a binary reward signal:

[0049] This reward encourages the exploration patterns of target domain trajectories to align with richer general domain patterns.

[0050] For each trajectory Its final reward Rewarded by standard verifiable accuracy Together with the aforementioned EDA rewards, they constitute:

[0051] in, If and only if the trajectory The final answer is correct; otherwise, it is 0.

[0052] The optimization objective of the strategy model is to maximize the expected reward, and the corresponding loss function is:

[0053] Iteratively update the policy model parameters using gradient descent algorithms (such as AdamW). Continue until the model converges.

[0054] It should be noted that the HEAL framework proposed in this application can significantly improve the inference performance of large language models trained using RLVR in low-resource scenarios. Its technical effectiveness has been verified through extensive experiments on multiple domains and multiple models. (1) Effectively alleviates the entropy collapse problem of low-resource target domain RLVR: Under the "few sample" setting with only 32 target domain samples, compared with the baseline using only few target domain data, HEAL significantly alleviates the entropy value drop in the target domain during the training process, which confirms that HEAL effectively alleviates the entropy collapse and restores the model's exploration ability.

[0055] (2) Significantly improves the performance of low-resource RLVR: Under the "few-sample" setting using only 32 target domain samples, the model trained using the HEAL framework significantly outperformed the baseline trained using only few target domain data on multiple benchmark tests in the four domains of medicine, physics, code, and mathematics. For example, for the Qwen3-1.7B-Base model, in two benchmark tests in the code domain (HumanEval+ and LiveCodeBench v5), HEAL improved the average performance from 34.36% to 41.28%, a relative improvement of over 20%. This confirms that HEAL can systematically improve the model's exploration ability to obtain accurate inference paths.

[0056] (3) Comparing with or even surpassing high-resource training with minimal data: Experimental results show that the model trained using the HEAL framework with only 32 target domain samples and 384 general domain data can match or even surpass the performance of the model trained with 1000 target domain samples using "full sample" training. For example, in the physics benchmark tests (C-Eval-phy and WebInstruct-phy), HEAL achieved an average performance of 46.99%, which is better than the baseline trained with full sample data (44.57%); in the code domain, HEAL (53.42%) also outperformed the baseline trained with full sample data (51.00%). This proves that HEAL can greatly reduce the dependence on expensive target domain labeled data and achieve the effect of "high performance with low resources".

[0057] (4) Validation of the effectiveness of the framework components: Comparative experiments show that while simple mixing of data from different domains is helpful to some extent, the complete HEAL framework, which combines data selection and EDA rewards, can bring more consistent and significant performance gains. This verifies the necessity and synergistic effect of this application.

[0058] In summary, the large language model training method based on dynamic alignment of hybrid domain entropy according to embodiments of the present invention addresses the fundamental limitation of insufficient target domain data in low-resource scenarios by introducing easily accessible general domain data (such as commonsense reasoning data) as a supplementary source for exploration behavior. Unlike simple hybridization, this application proposes a dual-criteria value assessment and selection strategy. Specifically, for each general domain sample, multiple reasoning trajectories are generated by the model, and its reasoning uncertainty (based on the average accuracy of the reasoning trajectory) and exploration diversity (based on the average entropy of high-entropy tokens in the trajectory) are calculated. The product of these two factors is used as the comprehensive value score of the sample, and only the K samples with the highest value are selected for training. This mechanism ensures that the introduced general data provides diverse basic reasoning patterns to broaden the exploration space of the strategy while avoiding the computational overhead caused by introducing a large amount of low-value data, achieving efficient and guided exploration enhancement. Furthermore, to more accurately transfer the exploration diversity of the general domain to the target domain, this application proposes an Entropy Dynamics Alignment (EDA) reward mechanism. This mechanism goes beyond simple regularization of entropy values, guiding the policy model's exploration behavior by modeling and comparing trajectory-level entropy dynamics. First, a sequence of token entropy values ​​changing over time is constructed for each generated inference trajectory, finely capturing the magnitude and variation patterns of token entropy. During training, trajectory entropy dynamic sets are collected for both the target domain and the general domain. For the entropy dynamics of a target domain trajectory, its maximum similarity with other trajectories within the target domain (intra-domain similarity) and its maximum similarity with all trajectories in the general domain (inter-domain similarity) are calculated. When the inter-domain similarity is higher than the intra-domain similarity, it indicates that the exploration pattern of the target domain trajectory is more consistent with the diversity pattern of the general domain, thus granting a positive EDA reward. This reward mechanism uses the exploration pattern of the general domain as an "anchor" to guide the entropy features (including magnitude and variation patterns) of the target domain policy towards the general domain in a controlled and refined manner, thereby systematically mitigating entropy collapse.

[0059] To achieve the above embodiments, this embodiment also provides a large language model reasoning method based on dynamic alignment of hybrid domain entropy.

[0060] Figure 4 This is a flowchart illustrating a large language model inference method based on dynamic alignment of hybrid domain entropy according to an embodiment of the present invention, as shown below. Figure 4 As shown, the method includes the following steps: S201, Obtain the problem to be reasoned.

[0061] S202, input the problem to be reasoned into the trained large language model to obtain the corresponding reasoning result. The trained large language model is trained using the training method described above.

[0062] In summary, this application effectively addresses the entropy collapse and exploration limitations inherent in existing RLVR technologies when training in low-resource target domains through a value-based selective hybrid training mechanism using general domain data and a reward-guided mechanism based on cross-domain entropy dynamic alignment. Using only a small number of target domain samples, this framework significantly improves the inference performance of large language models, effectively mitigating entropy collapse, restoring the model's exploration capabilities, and achieving performance comparable to or even surpassing models trained with a large number of target domain samples with minimal target domain data. Furthermore, this framework reduces reliance on expensive target domain labeled data, making it widely applicable in low-resource scenarios such as medical diagnostic assistants, professional domain code generation tools, and science education question-answering systems, enabling the training of large language models with stronger inference capabilities and robustness at a lower data labeling cost.

[0063] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0065] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0067] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0068] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0069] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0070] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0071] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0072] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0073] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0074] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for training a large language model based on dynamic alignment of hybrid domain entropy, characterized in that, Includes the following steps: Obtain the general domain dataset and the target domain dataset; For each general domain sample in the general domain dataset, the policy model to be trained is used to generate multiple inference trajectories corresponding to it. Calculate the reasoning uncertainty and exploration diversity corresponding to each general domain sample based on the multiple reasoning trajectories; Based on the inference uncertainty and the exploration diversity, a comprehensive value score is obtained for each general domain sample, and the k general domain samples with the highest scores are selected to obtain a curated general domain dataset. In each iteration of training, a batch of data is sampled from the hybrid dataset for trajectory generation and dynamic entropy extraction to obtain the dynamic entropy set of the target domain samples and the dynamic entropy set of the general domain samples. The hybrid dataset includes a selected general domain dataset and a target domain dataset. The entropy dynamic alignment reward is calculated based on the dynamic entropy set of the target domain samples and the dynamic entropy set of the general domain samples, and the policy model parameters are updated until the preset training conditions are met to obtain a well-trained large language model.

2. The method for training a large language model based on dynamic alignment of hybrid domain entropy as described in claim 1, characterized in that, Calculate the inference uncertainty for each general domain sample x using the following formula. : in, Indicates an indicator function, This represents the true answer. This indicates the number of inference trajectories. Let represent the i-th inference trajectory. The closer the uncertainty score is to 1, the more uncertain the model's prediction for that sample is, and the higher its value.

3. The method for training a large language model based on dynamic alignment of hybrid domain entropy as described in claim 1, characterized in that, The exploration diversity corresponding to each general domain sample is calculated based on the multiple inference trajectories, including: Select the top 20% of words with the highest entropy values ​​in each reasoning trajectory and calculate the average entropy value of these words; The average entropy values ​​of the multiple inference trajectories are averaged to obtain the exploration diversity score for each general domain sample.

4. The method for training a large language model based on dynamic alignment of hybrid domain entropy as described in claim 1, characterized in that, Based on the inference uncertainty and the exploration diversity, a comprehensive value score is obtained for each general domain sample, including: The comprehensive value score for each general domain sample is obtained by multiplying the inference uncertainty and the exploration diversity.

5. The method for training a large language model based on dynamic alignment of hybrid domain entropy as described in claim 1, characterized in that, A batch of data is sampled from a mixed dataset for trajectory generation and dynamic entropy extraction to obtain a dynamic entropy set of the target domain samples and a dynamic entropy set of the general domain samples, including: For each sample in the batch, generate multiple corresponding trajectories using the current strategy; For each trajectory, calculate the entropy of its lexical probability distribution at each generation time step to obtain the entropy dynamics; Based on the entropy dynamics corresponding to each trajectory, the dynamic entropy set of the target domain samples and the dynamic entropy set of the general domain samples are obtained.

6. The method for training a large language model based on dynamic alignment of hybrid domain entropy as described in claim 1, characterized in that, The entropy dynamic alignment reward is calculated based on the dynamic entropy set of the target domain samples and the dynamic entropy set of the general domain samples, including: For each entropy dynamic in the set of entropy dynamics of the target domain samples, calculate its similarity with other entropy dynamics in the domain, and take the maximum value to obtain the intra-domain similarity. For each entropy dynamic in the entropy dynamic set of the target domain sample, calculate its similarity with all entropy dynamics in the entropy dynamic set of the general domain sample, and take the maximum value to obtain the inter-domain similarity. The corresponding entropy dynamic alignment reward is obtained based on the comparison results of intra-domain similarity and inter-domain similarity.

7. The method for training a large language model based on dynamic alignment of hybrid domain entropy as described in claim 6, characterized in that, The entropy dynamic alignment reward is obtained using the following formula: Where, if the i-th trajectory Entropy dynamics Inter-domain similarity Greater than intra-domain similarity At that time, entropy dynamically aligns rewards If it is 1, otherwise it is the entropy dynamic alignment reward. It is 0.

8. A large language model inference method based on dynamic alignment of hybrid domain entropy, characterized in that, Includes the following steps: Obtain the problem to be reasoned; The problem to be reasoned is input into a trained large language model to obtain the corresponding reasoning result, wherein the trained large language model is trained using the training method described in any one of claims 1-7.

Images