A dynamic optimization large language model training method and system based on cognitive region modeling

Abstract

The application provides a dynamic optimization large language model training method and system based on cognitive region modeling, the method comprising: taking training question data as input of a large language model to be trained to obtain output training answers; calculating answer accuracy and answer stability of the training answers of the large language model; dividing the training question data and the training answers according to the answer accuracy to obtain cognitive region division results; the cognitive region division results comprising knowledge mastery region data, knowledge ambiguity region data and knowledge illusion region data; constructing a joint loss function fusing cross-entropy loss and contrastive learning loss, and dynamically adjusting a total weight of the contrastive learning loss according to the answer stability; constructing a training set according to the cognitive region division results, and training the large language model through the joint loss function.

Description

Technical Field

[0001] This invention relates to the field of LLM model training technology, and in particular to a method for dynamically optimizing large language model training based on cognitive region modeling. Background Technology

[0002] In recent years, Large Language Models (LLMs) have been widely used in question answering, reasoning, knowledge base retrieval, and intelligent customer service. However, in practical deployments, models are prone to the "hallucination" problem, which means that in the absence of real knowledge support, they output seemingly reasonable but actually incorrect answers, or fabricate content when faced with unanswerable questions, seriously affecting the reliability, credibility, and security of the system.

[0003] To address the issue of "illusion" that easily occurs in large language models, existing technologies have proposed the following types of solutions: 1) Supervised Fine-Tuning (SFT) is employed. High-quality question-and-answer data is manually labeled to supervise and fine-tune the pre-trained model, enabling the model to learn correct answers.

[0004] The drawback of using supervised fine-tuning is that it is difficult to cover situations where the model "does not know but still answers", and it cannot explicitly model the cognitive uncertainty of the model.

[0005] 2) Employ contrastive or preference-based learning methods. For example, construct contrastive samples of "correct answers vs. incorrect answers," introduce "I don't know" as a rejection option, and divide the question into several regions based on the accuracy of multiple samplings of the model, applying different training strategies to different regions. Typically, this method includes the following steps: ① Sampling the same question multiple times; ② Statistically calculating the proportion of correct answers; ③ Dividing the question into several regions based on a preset threshold; ④ Constructing contrastive samples for different regions and training accordingly.

[0006] The concept of contrastive learning is to learn discriminative representations by bringing "similar samples" closer in the representation space and by increasing the distance between "dissimilar samples".

[0007] Existing methods employing contrastive learning or preference learning, while mitigating the "illusion" problem prevalent in large language models to some extent, still suffer from the following drawbacks: ① Regions within a region are treated as homogeneous. Existing techniques typically divide a problem into a limited number of regions based solely on an accuracy threshold, assuming that samples within the same region possess the same learning characteristics. However, in practice, it has been found that even within the same region, significant differences remain in the model's response quality, stability, and error types; even with "correct answers," high-quality and low-quality expressions may coexist; and stable correct paths and stable incorrect paths may exist simultaneously for the same problem. ② Contrastive learning hyperparameters are statically set. Existing solutions typically use fixed contrastive learning hyperparameters (such as temperature parameters, loss weights, etc.), failing to adaptively adjust according to the model's current cognitive state. This leads to insufficient training constraints in knowledge boundary regions and over-training in knowledge-stable regions, easily causing overfitting or performance degradation. ③ Inability to characterize continuous changes in knowledge boundaries. Existing techniques treat the model's cognitive state as a discrete classification problem, lacking the ability to characterize the continuous evolutionary process from ignorance → fuzzy knowledge → stable knowledge. Summary of the Invention

[0008] In view of this, embodiments of the present invention provide a method and system for dynamically optimizing large language model training based on cognitive region modeling, in order to eliminate or improve one or more defects existing in the prior art.

[0009] One aspect of the present invention provides a method for dynamically optimizing the training of a large language model based on cognitive region modeling, the method comprising the following steps: The training question data is used as input to the large language model to be trained, and the training answer is output. Calculate the response accuracy and response stability of the trained responses of the large language model; The training question data and training answers are divided into cognitive regions based on the accuracy of the answers. The results of the cognitive region division include data from knowledge mastery regions, data from knowledge ambiguity regions, and data from knowledge illusion regions. Each data point includes a training question and a corresponding training answer. Each data point is divided into positive training answer samples and negative training answer samples based on the training answer. Construct a joint loss function that integrates cross-entropy loss and contrastive learning loss, and dynamically adjust the total weight of contrastive learning loss based on the stability of the response; A training set is constructed based on the results of cognitive region segmentation, and the large language model is trained using the joint loss function.

[0010] In some embodiments of the present invention, the step of calculating the response accuracy and response stability of the output results of the large language model includes: The large language model to be trained is sampled multiple times, and the correctness of all the sampled results is determined by some or all of the following methods: matching the output results precisely with the knowledge base according to the set rules to determine whether the answer is correct; determining whether the answer is correct through an external large model API; and obtaining the results of manual annotation to determine whether the answer is correct. The variance is calculated based on the accuracy of the responses. This variance represents the stability of the responses, and the formula is as follows: ; Among them, the Indicates the accuracy rate of the answer. Indicates the stability of the answer. The larger the value, the more unstable it is.

[0011] In some embodiments of the present invention, the step of dividing the training question data and training answers into cognitive regions based on the answer accuracy includes: The training question data and training answers are divided into three regions based on the set first accuracy threshold and the set second accuracy threshold: knowledge mastery region data, knowledge ambiguity region data, and knowledge illusion region data; wherein the first accuracy threshold is greater than the second accuracy threshold.

[0012] In some embodiments of the present invention, after the step of dividing the training question data and training answers into cognitive regions based on the answer accuracy, the method further includes: constructing positive training answer samples and negative training answer samples for comparative learning based on the cognitive region division; wherein, for knowledge-mastered regions and knowledge-ambiguous regions, the focus is on the knowledge known to the large model and improving the generation quality of the large model, and for knowledge-illusion regions, the honesty of the large model is improved.

[0013] In some embodiments of the present invention, the step of constructing a joint loss function that integrates cross-entropy loss and contrastive learning loss includes: ; in, Represents cross-entropy loss, Indicates the contrast learning loss. This indicates that the total weight of the contrastive learning loss is dynamically adjusted based on the stability of the responses; The formula for the total weight of the contrastive learning loss is as follows: ; Based on weights, This is the preset instability amplification factor. It is data The accuracy variance.

[0014] In some embodiments of the present invention, the formula for the contrastive learning loss function is expressed as follows: ; in, yes (Anchor answer) The hidden state of the large model. It's a hyperparameter representing the temperature used for comparative learning. This indicates that the cosine similarity between the two is calculated. yes The hidden layer state of the original sample under the problem; yes The hidden layer state of a positive answer to a given question; yes The hidden layer state of the negative sample (positive answer) under the question.

[0015] In some embodiments of the present invention, the method further includes: constructing a validation set and a test set, using the validation set to validate the trained large language model, and using the test set to test the large language model.

[0016] Corresponding to the above methods, the present invention also provides a method and system for training a dynamically optimized large language model based on cognitive region modeling, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of any of the methods described in the above embodiments.

[0017] In accordance with the above methods, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the method as described in any of the above embodiments.

[0018] Corresponding to the above methods, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

[0019] The proposed method for training a dynamic optimization large language model based on cognitive region modeling involves generating and sampling the same question multiple times. Based on statistical indicators such as answer accuracy and answer stability, the cognitive state of the model is characterized, enabling the model to distinguish its level of knowledge about the question. This helps reduce erroneous generation behavior on unknown or uncertain questions.

[0020] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0021] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1 This is a flowchart of a method for dynamically optimizing large language model training in one embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0024] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0025] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0026] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0027] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0028] To address the common problems in practical applications of existing large language models, such as hallucinatory responses, unclear knowledge boundaries, forced generation of answers for uncertain questions, and static settings of contrastive learning parameters, this invention aims to propose a dynamic optimization training method for large language models based on cognitive region modeling. This method is used to control the quality of large language models and to improve their security by combining dynamic preference optimization methods.

[0029] Figure 1 This is a flowchart of a method for dynamically optimizing a large language model training according to an embodiment of the present invention. The method includes the following steps: Step S110: Use the training question data as input to the large language model to be trained, and obtain the output training answer.

[0030] Step S120: Calculate the response accuracy and response stability of the training responses of the large language model.

[0031] Step S130: Divide the training question data and training answers into cognitive regions based on the accuracy of the answers; the results of the cognitive region division include knowledge mastery region data, knowledge ambiguity region data and knowledge illusion region data, wherein each data includes a training question and a corresponding training answer, and each data is divided into positive training answer samples and negative training answer samples according to the training answer.

[0032] Step S140: Construct a joint loss function that integrates cross-entropy loss and contrastive learning loss, and dynamically adjust the total weight of contrastive learning loss based on the stability of the response.

[0033] Step S150: Construct a training set based on the results of cognitive region segmentation, and train the large language model using the joint loss function.

[0034] The proposed method for training a dynamic optimization large language model based on cognitive region modeling involves generating and sampling the same question multiple times. Based on statistical indicators such as answer accuracy and answer stability, the cognitive state of the model is characterized, enabling the model to distinguish its level of knowledge about the question. This helps reduce erroneous generation behavior on unknown or uncertain questions.

[0035] In some embodiments of the present invention, the steps of calculating the answer accuracy and answer stability of the output results of the large language model include: sampling the large language model to be trained multiple times, and using some or all of the following methods to determine the correctness of all sampled results, including: accurately matching the output results with the knowledge base according to the set rules to determine whether the answer is correct; determining whether the answer is correct through an external large model API; and obtaining the result of manual annotation to determine whether it is correct.

[0036] The variance is calculated based on the accuracy of the responses. This variance represents the stability of the responses, and the formula is as follows: ; Among them, the Indicates the accuracy rate of the answer. Indicates the stability of the answer. The larger the value, the more unstable it is.

[0037] By employing this embodiment of the invention, the total weight of the contrastive learning loss can be dynamically adjusted through answer stability, thereby indirectly and dynamically adjusting the joint loss function. This allows for different training emphases to be adopted for different answer stability conditions, which helps to obtain more accurate large language model training results.

[0038] In some embodiments of the present invention, the step of dividing the training question data and training answers into cognitive regions based on the answer accuracy includes: dividing the training question data and training answers into knowledge mastery region data, knowledge ambiguity region data and knowledge illusion region data according to a set first accuracy threshold and a set second accuracy threshold; wherein the first accuracy threshold is greater than the second accuracy threshold.

[0039] By adopting this embodiment of the invention, a knowledge region division mechanism based on multiple sampling statistics is introduced. By generating and sampling the same question multiple times, and based on the answer accuracy and answer instability indicators, the cognitive state of the model regarding the question is divided into the knowledge mastery region, the knowledge ambiguity region, and the knowledge illusion region. This explicitly characterizes the differences in the model's "knowing / vaguely knowing / not knowing" states from the training data level, and can provide a more representative training data foundation for subsequent differentiated training.

[0040] In some embodiments of the present invention, after the step of dividing the training question data and training answers into cognitive regions based on the answer accuracy, the method further includes: constructing positive training answer samples and negative training answer samples for comparative learning based on the cognitive region division; wherein, for knowledge mastery regions and knowledge ambiguity regions, the focus is on the knowledge known by the large model and improving the generation quality of the large model, and for knowledge illusion regions, the honesty of the large model is improved.

[0041] By adopting this embodiment of the invention, a contrastive learning mechanism with "don't know" or refusal to answer as positive samples can be introduced into the knowledge illusion region. This guides the model to prioritize not answering or explicitly expressing uncertainty when there is a lack of reliable knowledge support, thereby effectively reducing the risk of model illusion, improving the reliability and security of the system, and enhancing the honesty and security of large language models in uncertain knowledge scenarios.

[0042] In some embodiments of the present invention, the step of constructing a joint loss function that integrates cross-entropy loss and contrastive learning loss includes: ; in, Represents cross-entropy loss, Indicates the contrast learning loss. This indicates that the total weight of the comparative learning loss is dynamically adjusted based on the stability of the responses.

[0043] The formula for the total weight of the contrastive learning loss is as follows: ; Based on weights, This is the preset instability amplification factor. It is data The accuracy variance.

[0044] By employing this embodiment of the invention, it is possible to strengthen the contrastive constraints for cognitive instability problems and reduce the contrastive model constraints for stable problems, thereby flexibly adjusting the emphasis of contrastive learning.

[0045] The formula for the contrastive learning loss function is as follows: ; in, yes (Anchor answer) The hidden state of the large model. It's a hyperparameter representing the temperature used for comparative learning. This indicates that the cosine similarity between the two is calculated. yes The hidden layer state of the original sample under the problem; yes The hidden layer state of a positive answer to a given question; yes The hidden layer state of the negative sample (positive answer) under the question.

[0046] By employing this embodiment of the invention, the temperature parameter and loss weight in the contrastive learning loss function can be dynamically adjusted based on the cognitive region where the sample is located and its boundary position. This allows the training intensity to adapt to changes in the model's cognitive state, avoiding excessive intervention in already acquired knowledge or insufficient constraints on boundary knowledge. This helps to overcome the training imbalance problem caused by static hyperparameter settings in contrastive learning training.

[0047] In some embodiments of the present invention, the method further includes: constructing a validation set and a test set, using the validation set to validate the trained large language model, and using the test set to test the large language model.

[0048] For example, the dataset can be split into a 7:2:1 ratio (70% training data, 20% validation data, and 10% test data). The batch size can be set to 64, and the epoch can be set to 2.

[0049] In a specific embodiment of the present invention, the method for training a dynamic optimization large language model based on cognitive region modeling can be divided into the following steps: (1) acquiring training question data; (2) generating and sampling multiple times for each question; (3) evaluating the correctness of each generated answer; (4) calculating the answer accuracy and answer stability index corresponding to the question; (5) dividing the question into knowledge regions based on statistical indicators; (6) constructing positive and negative contrast learning samples according to the knowledge region; (7) dynamically adjusting the contrast learning parameters based on the region boundary; (8) dynamically adjusting the λ of the joint loss function based on the region stability; (9) training the large language model based on the joint loss function.

[0050] In another embodiment of the present invention, the dynamic optimization method for training large language models based on cognitive region modeling proposed in this invention is explained from a principle perspective as follows: (1) Selecting an open-source large model The original large model needs to be an open source large model (such as the Qwen, ChatGLM series large models).

[0051] (2) Calculate the accuracy rate For each problem x, perform K sampling operations using the currently selected open-source model: ; In one embodiment of the present invention, the value of K is selected as 10.

[0052] Correctness of answer: The above K results are evaluated using at least one of the following methods: ① exact matching of rules and knowledge base; ② external large model API (such as ChatGPT-5); ③ manual annotation.

[0053] The accuracy rate is calculated using the following formula: ; in I (y i (Correct) represents an indicator function, when y i The value is 1 if the prediction is correct, and 0 otherwise.

[0054] This formula calculates the model's response accuracy across all K samples.

[0055] (3) Regional division Based on the accuracy rate (Acc) of the answers, each question is divided into regions, and two thresholds are set.

[0056] Knowledge Acquisition Area: They believe that the large model has a better grasp of this problem.

[0057] Areas of ambiguity in knowledge: They believed that large models generally lacked a grasp of this problem and were prone to giving incorrect answers.

[0058] Knowledge Illusion Zone: They believed that the large model had a poor grasp of this problem and frequently gave incorrect answers.

[0059] (4) Supervising fine-tuning of SFT Supervised Fine-Tuning (SFT) is a technique that further optimizes model performance by introducing labeled data on top of a large-scale pre-trained model. Its core function is to enable the model to better adapt to the needs of specific tasks or domains through supervised learning, thereby improving the accuracy, generalization ability, and task adaptability of predictions.

[0060] SFT's cross-entropy loss function: in It's a certain question. This is the output length corresponding to this problem. yes The input to the model is the output. yes The kth output. yes The cross-entropy loss.

[0061] (5) Comparative learning Contrastive learning learns discriminative representations by bringing "similar samples" closer together in the representation space and widening the distance between "dissimilar samples".

[0062] Loss function for contrastive learning: in yes (Anchor answer) is the hidden layer state of the large model (the output of the last layer, with shape [batch_size, sequence_length, hidden_size], which is a vector representation of each token position).

[0063] It is cosine similarity; It is a hyperparameter representing the temperature during comparative learning; yes The hidden layer state of the original sample under the problem; yes The hidden layer state of a positive answer to a given question; yes The hidden layer state of the negative sample (positive answer) under the question.

[0064] (6) The loss of this scheme The SFT method combines cross-entropy loss and contrastive learning loss. The aim is to improve the model's ability to accurately characterize knowledge boundaries through multi-objective optimization, while also enhancing the reliability and generalization of the generated results.

[0065] Where λ is a hyperparameter, cross-entropy loss (such as the standard cross-entropy in SFT) directly optimizes the matching degree between the model's predicted probability and the true label, ensuring the model's basic classification or generation ability based on known knowledge. Contrastive learning explicitly models the knowledge boundary by bringing positive sample pairs (such as the correct answers known to the model) closer and widening the distance between negative sample pairs (such as wrong answers or unknown questions).

[0066] In another specific embodiment of the present invention, the solution is described below from a specific implementation perspective (including detailed examples of prompt words): (1) Organize industry knowledge Data organized according to usage scenarios { Question: "Please answer the question related to financial compliance. What is the definition of insider trading according to the Securities Law?" "Answer": "According to the Securities Law of the People's Republic of China, insider trading refers to the act of an insider using such information to engage in securities trading activities, or disclosing such information to others, thereby causing unfairness in the market." } This invention does not limit its specific implementation method and can adopt existing mature solutions.

[0067] (2) Selection of open source large model Based on the balance between the company's business needs, data characteristics, resource capabilities, and model capabilities, a suitable open-source large model is selected. This invention does not limit the specific implementation method and can adopt existing mature solutions.

[0068] (3) Calculate the accuracy and variance 1) Calculation accuracy: Input each question of all the data in (1) into the selected open source model in sequence, and 10 results will be generated for each question.

[0069] Through the following methods: ① Precise matching based on rules and knowledge base ② External large model APIs (such as ChatGPT-5) ③ Manual annotation The accuracy of each result is calculated (this invention does not limit the specific implementation method and existing mature solutions can be used). For example: { Question: "Which is the largest stock exchange in China?" {"answer":"Shanghai Stock Exchange", "Truth or False": True}, {"answer":"Shanghai Stock Exchange", "Truth or False": True}, {"answer":"Shenzhen Stock Exchange", "Truth or False": False}, ... (There are ten answers in total; this is just an example) } Based on the accuracy of the above 10 results, we can obtain the accuracy of the large model for each question: in I (y i (Correct) represents an indicator function, when y i The value is 1 if the prediction is correct, and 0 otherwise.

[0070] 2) Calculate the variance (a measure of the instability of the response), expressed by the formula: ; Where p is Acc(x).

[0071] (4) Divide the area Based on the accuracy of the 10 results for each sentence above, all data are partitioned. Each question is then divided into regions based on Acc, with two thresholds set (first accuracy threshold). Second accuracy threshold ).

[0072] Knowledge Acquisition Area: They believe that the large model has a better grasp of this problem.

[0073] Areas of ambiguity in knowledge: They believe that large models generally lack a grasp of this problem and are prone to answering incorrectly. Knowledge Illusion Zone: They believe that the large model has a poor grasp of this problem and frequently answers incorrectly. This invention selects =0.8, =0.4. That is, if the large model answers question A correctly at least 8 out of 10 times, then the large model is considered to have a good grasp of this knowledge. The same analysis can be applied to other areas.

[0074] Therefore, the dataset for each region can be obtained. Knowledge mastery area data: .

[0075] Data on fuzzy knowledge regions: .

[0076] Data on the knowledge illusion region: .

[0077] Each sentence in each of the above regions has an accuracy rate. Therefore, the average accuracy rate for each region can be calculated.

[0078] Average accuracy of knowledge mastery data in the region: .

[0079] Average accuracy of data in knowledge-fuzzy regions: .

[0080] ⑦ Average accuracy of data in the knowledge illusion region: .

[0081] (5) Constructing positive and negative samples based on region-based contrastive learning For each data point, there is a training question and a training answer. In the contrastive learning method, positive training answers are designated as positive answers, and negative training answers are designated as negative answers. The following describes the construction of datasets for different regions: Knowledge mastery area and knowledge ambiguity area: 1) Focus on the known knowledge of large models anchor answer: the original answer positive answer: The correct answer generated Negative answer: "This question is beyond my understanding; I don't know." 2) Improve the generation quality of large models anchor answer: the original answer positive answer: The correct answer generated negative answer: The generated incorrect answer Knowledge Illusion Zone: 1) Improve the honesty of large models Anchor answer: "This question is beyond my understanding; I don't know." Positive answer: "This question is beyond my understanding; I don't know." negative answer: The generated incorrect answer (6) Temperature function based on region boundary The loss function for known contrastive learning is: ; This invention dynamically adjusts the hyperparameters in the loss function based on data from different regions. The specific method is as follows: in: τ is a hyperparameter representing the temperature at which the contrastive learning occurs; The accuracy of the current data x The average accuracy of the region to which the current data x belongs. The width of the accuracy interval for the region to which the current data x belongs, such as the knowledge ambiguity region. for - and These are preset upper and lower limits. This invention sets... =0.05 and =0.01 Technical effects: ① Near the regional boundary → lower temperature → stronger contrast; ② Located in the center of the region → higher temperature → more gradual contrast.

[0082] (7) Dynamic adjustment In addition to temperature parameters, this invention also dynamically adjusts the contrast loss weights.

[0083] Define the total weight of the contrastive loss as in, Based on weights, As an instability amplification factor, It is the accuracy variance of data x.

[0084] Technical effects: To address the problem of cognitive instability, strengthen the contrast constraint.

[0085] For stability issues, reduce the constraints of the comparison model.

[0086] (8) Model training Based on the above data construction and loss function construction methods, the dataset is divided into a 7:2:1 ratio (70% training data, 20% validation data, and 10% test data). The batch size is set to 64, and the epoch is set to 2.

[0087] By employing the specific embodiments described above, the technical effects proposed in this invention can be achieved through the partitioning of training data and the configuration of the training process. Furthermore, the above embodiments do not represent limitations on specific implementation methods; existing mature solutions can be adopted according to specific scenarios.

[0088] Key technical aspects of this invention include, but are not limited to: (1) Knowledge region division mechanism based on multiple sampling statistics This invention generates and samples the same question multiple times, and based on statistical indicators such as answer accuracy and answer instability (variance), divides the model's cognitive state of the question into a knowledge mastery region, a knowledge ambiguity region, and a knowledge illusion region. It explicitly characterizes the model's differences in "knowing / vaguely knowing / not knowing" at the training data level, providing a foundation for subsequent differentiated training.

[0089] (2) Modeling continuous cognitive states within a region, rather than discrete homogeneous processing. Unlike existing technologies that only roughly divide samples into finite discrete regions and treat them as homogeneous samples, this invention further utilizes multi-dimensional indicators such as the accuracy of a single sample, the average accuracy of the region, and the stability of responses to continuously characterize the cognitive state of samples within a region, thus solving the problem of not being able to model the differences in response quality and stability within the same region.

[0090] (3) Dynamic contrastive learning temperature regulation mechanism based on regional boundary location This invention proposes a temperature function based on the relative positional relationship between sample accuracy and the average accuracy of its region. The temperature parameter in the contrastive learning loss function is not a fixed constant, but dynamically determined according to the relative positional relationship between sample accuracy and the average accuracy of its region. This enables differentiated contrast intensity control, allowing the temperature parameter in the contrastive learning loss function to dynamically change with the sample's position in the knowledge region, thereby achieving: ① Enhance contrast constraints as you approach the cognitive boundary; ② When located at the center of the region, reduce the contrast intensity to avoid overfitting.

[0091] (4) Combine the dynamic adjustment mechanism of contrast loss weights based on the instability of the response This invention not only dynamically adjusts temperature parameters but also introduces a contrastive loss weight adjustment mechanism based on the variance of response accuracy. This strengthens constraints when the model faces cognitively unstable problems and reduces intervention intensity when facing cognitively stable problems, thereby improving the robustness and stability of the model from the training objective level. The weight ratio of contrastive learning loss in the overall training objective is dynamically adjusted based on the variance of response accuracy or equivalent instability indicators obtained from multiple samplings.

[0092] (5) Regionalized positive and negative sample construction strategy oriented towards "honesty" To address different knowledge regions, this invention designs a differentiated method for constructing positive and negative samples for contrastive learning. Particularly in the knowledge illusion region, "I don't know" answers are introduced as positive samples into the training objective, explicitly guiding the model to choose to refuse or give an uncertain answer in uncertain knowledge scenarios, fundamentally suppressing model illusions. Different anchor, positive, and negative sample construction strategies are employed in different cognitive regions. In the knowledge illusion region, "I don't know" or "refuse to answer" answers are used as positive samples in contrastive learning training to suppress the model's error generation under unknown knowledge.

[0093] Corresponding to the above method, the present invention also provides a dynamic optimization large language model training system based on cognitive region modeling. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0094] Optionally, the system can be divided into any combination of functional units such as data processing module, region division module, parameter dynamic adjustment module, and model training module, as well as their equivalent implementations.

[0095] Corresponding to the methods described above, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the method as described in any of the above embodiments. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0096] Corresponding to the above methods, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

[0097] In summary, the present invention can achieve the following beneficial technical effects: (1) By generating samples of the same question multiple times and based on statistical indicators such as answer accuracy and answer stability, the cognitive state of the model is characterized, enabling the model to distinguish its knowledge of the question, which helps to reduce erroneous generation behavior on unknown or uncertain questions.

[0098] (2) By introducing multi-dimensional indicators such as single sample accuracy, regional average accuracy and response instability on the basis of knowledge region division, continuous modeling of samples within the region is carried out, avoiding treating samples of different quality and stability in the same region as the same training object. This can overcome the defects of coarse-grained cognitive region and homogenization within the region in the existing technology.

[0099] (3) Compared with the existing technology that simply regards the cognitive state of the model as a discrete category, this application describes the continuous change process of the model from "don't know" to "fuzzy know" and then to "stable know", which helps to describe the knowledge boundary position of the model more precisely and can effectively describe the continuous evolution process of the model's knowledge boundary.

[0100] (4) By dynamically adjusting the temperature parameter and loss weight in the contrastive learning loss function based on the cognitive region where the sample is located and its boundary position, the training intensity can adapt to the cognitive state of the model, which can avoid the problem of excessive intervention in the knowledge already mastered or insufficient constraint on the boundary knowledge, and helps to overcome the training imbalance problem caused by the static setting of hyperparameters in contrastive learning training.

[0101] (5) By combining supervised fine-tuning and dynamic comparative learning, the model can improve its error correction ability in fuzzy knowledge areas while maintaining the stability of the knowledge it has mastered. This will achieve a comprehensive improvement in the output quality, stability and security of the model, and help to comprehensively improve the overall output quality and generalization stability of the large language model.

[0102] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0103] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0104] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0105] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for dynamically optimizing large language model training based on cognitive region modeling, characterized in that, The method includes the following steps: The training question data is used as input to the large language model to be trained, and the training answer is output. Calculate the response accuracy and response stability of the trained responses of the large language model; The training question data and training answers are divided into cognitive regions based on the accuracy of the answers. The results of the cognitive region division include data from knowledge mastery regions, data from knowledge ambiguity regions, and data from knowledge illusion regions. Each data point includes a training question and a corresponding training answer. Each data point is divided into positive training answer samples and negative training answer samples based on the training answer. Construct a joint loss function that integrates cross-entropy loss and contrastive learning loss, and dynamically adjust the total weight of contrastive learning loss based on the stability of the response; A training set is constructed based on the results of cognitive region segmentation, and the large language model is trained using the joint loss function.

2. The method according to claim 1, characterized in that, The steps for calculating the response accuracy and response stability of the output results of the large language model include: The large language model to be trained is sampled multiple times, and the correctness of all the sampled results is determined by some or all of the following methods: matching the output results precisely with the knowledge base according to the set rules to determine whether the answer is correct; determining whether the answer is correct through an external large model API; and obtaining the results of manual annotation to determine whether the answer is correct. The variance is calculated based on the accuracy of the responses. This variance represents the stability of the responses, and the formula is as follows: ； Among them, the Indicates the accuracy rate of the answer. Indicates the stability of the answer. The larger the value, the more unstable it is.

3. The method according to claim 1, characterized in that, The steps of dividing the training question data and training answers into cognitive regions based on the accuracy of the answers include: The training question data and training answers are divided into three regions based on the set first accuracy threshold and the set second accuracy threshold: knowledge mastery region data, knowledge ambiguity region data, and knowledge illusion region data; wherein the first accuracy threshold is greater than the second accuracy threshold.

4. The method according to claim 1, characterized in that, After the step of dividing the training question data and training answers into cognitive regions based on the accuracy of the answers, the method further includes: constructing positive training answer samples and negative training answer samples for contrastive learning based on the cognitive region division; wherein, for knowledge-mastered regions and knowledge-ambiguous regions, the focus should be on the knowledge known by the large model and the generation quality of the large model should be improved, and for knowledge-illusion regions, the honesty of the large model should be improved.

5. The method according to claim 1, characterized in that, The steps for constructing a joint loss function that integrates cross-entropy loss and contrastive learning loss include: ； in, Represents cross-entropy loss, Indicates the contrast learning loss. This indicates that the total weight of the contrastive learning loss is dynamically adjusted based on the stability of the responses; The formula for the total weight of the contrastive learning loss is as follows: ； Based on weights, This is the preset instability amplification factor. It is data The accuracy variance.

6. The method according to claim 5, characterized in that, The formula for the contrastive learning loss function is as follows: ； in, yes (Anchor answer) The hidden state of the large model. It's a hyperparameter representing the temperature used for comparative learning. This indicates that the cosine similarity between the two is calculated. yes The hidden layer state of the original sample under the problem; yes The hidden layer state of a positive answer to a given question; yes The hidden layer state of the negative sample (positive answer) under the question.

7. The method according to claim 1, characterized in that, The method further includes: constructing a validation set and a test set, using the validation set to validate the trained large language model, and using the test set to test the large language model.

8. A dynamic optimization large language model training system based on cognitive region modeling, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images