Method and device for selecting training samples in a model based on value perception

Abstract

The application discloses a value perception-based model mid-term training sample selection method and device, and relates to obtaining a candidate training sample set of a large language model and a vertical field question and answer system in a mid-term training stage. The value perception reasoning is performed on the candidate sample set through the parameters and the intermediate state of the current large language model, and the in-domain value representation of the sample is constructed. According to the potential training value and the multi-dimensional features of the sample, the dynamic weighting algorithm is adopted to allocate weights in combination with the mid-term training demand of the vertical field question and answer system, and the high-value sample is selected for incremental training. The model performance evolution result is obtained through the update of the model parameters and the change of the intermediate state, and the sample selection strategy is dynamically optimized. The method can improve the question and answer accuracy and matching efficiency of the large language model in the vertical field, and is widely applied to professional problem retrieval matching, accurate answer and field knowledge output scenes.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence and natural language processing technology, and in particular relates to a method and apparatus for selecting mid-term training samples for a value-perception-based model. Background Technology

[0002] With the widespread application of large language models in natural language processing, their value in vertical domain question answering systems and retrieval question answering systems is becoming increasingly prominent. By pre-training on general corpora and fine-tuning with domain data, large language models can complete complex question answering and information retrieval tasks. However, vertical domains are typically characterized by high specialization, uneven knowledge distribution, and prominent long-tail problems, which makes model training and continuous optimization face high cost and efficiency bottlenecks. After the model training enters the mid-stage, the overall performance of the model gradually stabilizes, but there are still shortcomings in specific professional problems, long-tail knowledge points, and complex reasoning scenarios. At this point, continuing to train with large-scale, uniformly sampled training data often introduces a large number of redundant samples that contribute little to improving the model's capabilities, not only increasing training overhead but also making it difficult to effectively improve the model's performance in key application scenarios.

[0003] To improve training efficiency, active learning methods are used to select more valuable samples for model training from candidate data. However, existing active learning methods mostly rely on static indicators such as model prediction uncertainty or sample diversity, making it difficult to accurately reflect the true contribution of samples to the improvement of model capabilities during the mid-term training of large language models. Their effectiveness is limited, especially when the model has a large number of parameters and complex training dynamics. In recent years, Test-Time Training (TTT) technology has developed a self-supervised signal by utilizing samples themselves during the inference phase, enabling the model to have a certain degree of adaptability during testing. This indicates that the intermediate states during model inference contain important behavioral and uncertainty information. However, existing TTT technology is mainly used for model adaptation during the testing phase and has not yet formed a sample value evaluation and selection mechanism suitable for the mid-term training of large language models, making it difficult to directly serve the training optimization needs of vertical domain question answering and retrieval question answering systems.

[0004] Therefore, how to accurately evaluate the potential training value of candidate samples by combining the current state of the model and the characteristics of the samples during the mid-term training stage of a large language model, and thereby achieve efficient sample selection to improve the training efficiency and performance of vertical domain question answering and retrieval question answering systems, still needs further research and solutions. Summary of the Invention

[0005] The purpose of this invention is to provide a value-aware model mid-term training sample selection method and apparatus, which mines the potential training value of candidate samples in the mid-term training stage of a large language model, and thereby achieves efficient sample selection to improve the training efficiency and performance of vertical domain question answering and retrieval question answering systems.

[0006] According to a first aspect of the embodiments of this application, a method for selecting mid-term training samples for a model based on value perception is provided, comprising:

[0007] Obtain a set of candidate training samples for large language models in the mid-training stage, as well as vertical domain question answering systems or retrieval question answering systems.

[0008] Based on the parameters and intermediate states of the current large language model, value-aware reasoning is performed on the candidate training sample set to construct a domain-specific sample value representation.

[0009] Based on the value representation of samples in the domain, the potential training value of candidate training samples in improving model capabilities in the vertical domain is evaluated, and the multi-dimensional value features of each candidate sample are quantified.

[0010] Based on the key training requirements of vertical domain question answering systems or retrieval question answering systems in the middle stage, and according to the potential training value and multi-dimensional value characteristics, a dynamic weighting algorithm is used to allocate weights and select high-value samples as middle stage training samples.

[0011] The large language model is incrementally trained using the intermediate training samples, and the model parameters and intermediate state changes of the model during training are updated.

[0012] Based on the model parameters and the intermediate state changes of the model during training, the evolution results of model performance changes are obtained, and the sample value evaluation strategy is dynamically updated, thereby adaptively optimizing the sample selection results during multiple rounds of intermediate training.

[0013] According to a second aspect of the application embodiments, a value-aware model mid-term training sample selection device is provided, comprising:

[0014] The acquisition module is used to acquire a set of candidate training samples for large language models in the mid-training stage, as well as vertical domain question answering systems or retrieval question answering systems.

[0015] The construction module is used to perform value-aware reasoning on the candidate training sample set based on the parameters and intermediate states of the current large language model, and to construct a domain-specific sample value representation.

[0016] The quantization module is used to evaluate the potential training value of candidate training samples in improving model capabilities in the vertical domain based on the value representation of samples in the domain, and to quantify the multi-dimensional value features of each candidate sample.

[0017] The mid-term training sample selection module is used to combine the key mid-term training requirements of vertical domain question answering systems or retrieval question answering systems, and select high-value samples as mid-term training samples by using a dynamic weighting algorithm to allocate weights based on the potential training value and multi-dimensional value features.

[0018] The model update module is used to incrementally train the large language model using the intermediate training samples and update the model parameters and the intermediate state changes of the model during training.

[0019] The sample selection module is used to obtain the evolution results of model performance changes based on the model parameters and the intermediate state changes of the model during training, and to dynamically update the sample value evaluation strategy, thereby adaptively optimizing the sample selection results during multiple rounds of intermediate training.

[0020] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising:

[0021] One or more processors;

[0022] Memory, used to store one or more programs;

[0023] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0024] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0025] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects:

[0026] Performing value-aware reasoning on the candidate training sample set can enhance the potential training value of the candidate training samples in improving the model's capabilities within the vertical domain; using a dynamic weighting algorithm to allocate weights and selecting high-value samples as intermediate training samples can enhance the robustness of the model; using incremental training and updating the model parameters and the changes in the model's intermediate states during training to dynamically update the sample value evaluation strategy enables adaptive optimization of sample selection results during multiple rounds of intermediate training, thereby constructing a high-quality, highly diverse training dataset that meets the task requirements, significantly improving the performance of the large language model in downstream tasks. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a value-perception-based method for selecting mid-term training samples for a model, according to an exemplary embodiment.

[0028] Figure 2This is a block diagram illustrating a value-aware model mid-term training sample selection device according to an exemplary embodiment.

[0029] Figure 3 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0031] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0032] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0033] Figure 1 This is a flowchart illustrating a value-perception-based method for selecting mid-term training samples in a model, according to an exemplary embodiment. This case uses a large language model in the legal field and a legal question-answering system or retrieval question-answering system as data sources, but is not limited to these. (Reference) Figure 1 The method for selecting mid-term training samples for a value-perception-based model provided in this embodiment of the invention may include:

[0034] S1: Obtain a set of candidate training samples for a large language model in the mid-training stage, as well as a vertical domain question answering system or a retrieval question answering system; this step includes the following sub-steps:

[0035] S11: Obtain a large language model in the question-answering domain that has completed the initial pre-training and intermediate training phases;

[0036] Specifically, firstly, a model that has completed its initial pre-training and intermediate training phases is obtained from a pre-training platform for a large-scale legal language model. This model has been pre-trained using a large amount of general data and has begun intermediate training in a specific legal domain, demonstrating some question-answering capabilities. While the model typically possesses basic language comprehension abilities in its initial stages, further optimization is still needed to adapt to the complex requirements of this vertical domain.

[0037] S12: Monitor the performance trends of large language models on the validation sets of vertical domain question answering systems or retrieval question answering systems;

[0038] Specifically, by monitoring the performance changes of the legal-oriented large language model (hereinafter referred to as the model) on the validation set of legal domain question answering system or retrieval question answering system, recording begins when the overall performance of the model tends to stabilize and the convergence speed slows down.

[0039] S13: When the overall performance of the model enters a stage of slow convergence but local capabilities continue to change, the model is determined to be in the mid-training stage.

[0040] Specifically, by monitoring the model's performance changes on the validation set, when the overall model performance tends to stabilize and the convergence speed slows down, it can be determined that the model has entered the mid-training stage. At this point, the model's capabilities in the legal domain have stabilized, but there will still be local variations and room for optimization in specific tasks or certain vertical domains.

[0041] S14: Collect candidate training samples from the question-answering system's question-answering logs, user queries and retrieval results from the question-answering system, synthetic question-answer pairs constructed from the question-answering system's document library, and candidate samples generated by the model but not yet used for training.

[0042] Specifically, by analyzing question-and-answer logs and user query logs, data containing user queries, system responses, and retrieval results is extracted from the question-and-answer system and the retrieval question-and-answer system. Candidate samples not yet used for training are collected, including synthetic question-and-answer pairs and candidate training samples not used during model generation. D :

[0043]

[0044] in, The first candidate training sample is denoted as the... i Sample questions, The first candidate training sample is denoted as the... i Sample answer, I This represents the total number of current candidate training samples.

[0045] These samples can provide potentially valuable training data to improve the model's ability in a specific domain.

[0046] S2: Based on the parameters and intermediate states of the current large language model, perform value-aware reasoning on the candidate training sample set to construct a domain-specific sample value representation. This step includes the following sub-steps:

[0047] S21: Input the candidate training samples into the current large language model and perform question-answering reasoning or retrieval question-answering reasoning;

[0048] Specifically, the candidate training samples collected in step S14 are input into the current large language model for question-answering reasoning or retrieval question-answering reasoning. Specifically, for each candidate sample, the model needs to perform a task-related reasoning process (such as answering questions, performing information retrieval, etc.) and record the reasoning results for each sample to evaluate its performance in a specific task. Through the reasoning process, the model can generate answers related to the target question for each sample, thereby evaluating the value of the sample.

[0049] S22: While keeping the main parameters of the model frozen or partially frozen, record the output of the model and the intermediate states during the inference process. The intermediate states refer to the hidden state representations, attention weight distributions, trainable intermediate adaptation parameters, and fast weights or temporary state variables used to assist training in the Transformer layer.

[0050] Specifically, while keeping the main parameters of the large language model frozen or partially frozen, during inference operations, in addition to recording the final model output, it is also necessary to record each intermediate state of the model during the inference process. These intermediate states include:

[0051] Hidden state representation in Transformer layers: In the Transformer architecture, the model generates hidden states (i.e., the output representation of each layer) during the computation of each layer. These hidden states reflect the model's understanding of the input and are crucial intermediate information during training. Recording these hidden states is essential for subsequent analysis of the model's responses to different inputs and its learning process.

[0052] Attention Weight Distribution: In Transformer models, the attention mechanism is used to determine the weights of different parts of the input sequence. Recording the weight distribution of each attention head can help understand the input regions the model focuses on during inference and reveal the model's decision-making process. For example, which words have higher attention weights when answering questions, and whether the model is able to focus on the correct context.

[0053] Trainable intermediate adaptation parameters: While freezing the main parameters, some trainable intermediate adaptation parameters can continue to participate in the training process. These parameters are typically used to fine-tune the model's adaptability to a specific task. Recording these adaptation parameters can help further optimize the model, making it more accurate in a specific domain.

[0054] Fast weights or temporary state variables: Fast weights are parameters that are dynamically adjusted during training, quickly adjusting the model's responsiveness after each training update. Temporary state variables, on the other hand, are variables used only temporarily during specific training steps or inference processes, such as activation values ​​and gradient values.

[0055] Recording these variables helps capture the details of changes within the model and provides a basis for further fine-tuning or optimization.

[0056] S23: Without updating the model's main parameters, only trainable constraints are introduced to the intermediate states, and the intermediate states are repeatedly reconstructed and learned to enhance the model's intermediate states.

[0057] Specifically, for candidate training samples D Feature extraction is performed, represented as a feature vector. :

[0058]

[0059] in Characteristics of legal terminology As the core semantic feature, As a characteristic of problem complexity, Adapt features to business scenarios.

[0060] The preprocessed candidate samples are input into the current large language model to perform retrieval-enhanced question-answering inference. During the inference process, the intermediate state information of the model in the Transformer network is recorded, including:

[0061] Will Represented as the first Items of data, hidden state is represented as We can obtain:

[0062]

[0063] in Indicates the first The forward computation process of the layer Transformer, This indicates that the user has a query question. This represents the retrieval context document content corresponding to the query.

[0064] Without updating the model's master parameters, a lightweight intermediate state training mechanism is introduced. A sub-supervised objective is constructed based on a masked autoencoder, and the intermediate states are repeatedly reconstructed and learned.

[0065] Mask Reconstruction Loss L Defined as:

[0066]

[0067] in This represents the intermediate state after reconstruction.

[0068] This method can improve the model's performance on specific tasks or datasets without updating the model's main parameters, while maintaining the advantages of large model pre-training and avoiding overfitting or loss of generality.

[0069] S24: Construct a self-supervised or weakly supervised objective function based on candidate samples, and use the objective function to perform one or more rounds of lightweight updates on the intermediate state, and record the change magnitude of the intermediate state before and after the update;

[0070] Specifically, based on candidate samples, self-supervised or weakly supervised objective functions are designed to guide the model to learn effectively without fully labeled data. Self-supervised objective functions typically utilize input data, such as contextual information, to construct pre-training tasks; for example, they learn data representations by predicting a portion of the input data. Weakly supervised objective functions, on the other hand, can utilize partially labeled information or external signals to constrain model training.

[0071] Self-supervised objective function The form can be:

[0072]

[0073] Where I This represents the total number of current candidate training samples.

[0074] Using the objective function described above, one or more lightweight updates are performed on the intermediate states of the model. These updates only adjust the representation of the intermediate states (such as hidden layer states, attention weights, etc.) without affecting the model's main parameters. These updates are performed through backpropagation, updating the adaptation module of the intermediate states and optimizing the representation of the intermediate states.

[0075] The update formula for intermediate states is:

[0076]

[0077] in, This is an intermediate state before the update. This is the updated intermediate state. The learning rate is set to 0.01 in this case. The gradient represents the intermediate state.

[0078] After each update, the magnitude of the change in the intermediate states is recorded, and the update amount of each intermediate state is measured. By recording these changes, the progress of the model in each training round and the optimization effect of the intermediate states can be evaluated.

[0079] S25: Based on the changes in the model output and the intermediate states of the inference process before and after the update, extract the value representation of the samples in the domain.

[0080] Specifically, after completing the lightweight update, the model's output and intermediate state changes during the inference process are recorded. For example, the model's predictions before and after the update are compared. and hidden state representation Output results before and after the update .

[0081]

[0082] Based on the magnitude of changes in the model output and intermediate states, the value representation of each sample within the vertical domain is extracted. These representations reflect the contribution of each sample to improving the model's capabilities. For example, a larger magnitude of change may indicate that the sample has high value for model optimization, while a smaller magnitude of change may indicate that the sample has low training utility.

[0083] Sample value representation The calculation can be formalized as:

[0084]

[0085] Here, 𝑓 represents the value of a sample calculated based on the magnitude of change in the model output and intermediate states. This value representation is used in subsequent steps to select high-value samples as training data.

[0086] These steps allow us to extract the value of each candidate sample during model training and help the model select efficient and critical training samples for further optimization.

[0087] S3: Based on the value representation of samples within the aforementioned domain, evaluate the potential training value of candidate training samples in improving model capabilities within the vertical domain, and quantify the multi-dimensional value features of each candidate sample; this step includes the following sub-steps:

[0088] S31: Determine the representational meaning of at least three core assessment dimensions within each domain;

[0089] Specifically, the matching degree between the domain-adapted value representation samples and the vertical domain knowledge system corresponding to the current parameters of the model, and the core business scenarios; the information density of the information-scarce value representation samples containing new domain scenarios and scarce knowledge that the model has not fully learned; and the quantitative value of the professional semantic similarity between the redundancy avoidance value representation samples and the trained samples.

[0090] Domain Adaptation Value This is used to measure the degree of matching between the current sample and the knowledge system of the vertical domain:

[0091]

[0092] in Representation of domain knowledge prototypes Represented as The hidden state of the layer i Indicated as the current number i 10 candidate training samples.

[0093] The scarcity value of professional information This is used to measure the knowledge density in the samples that the model has not yet fully learned, representing the magnitude of change in intermediate states before and after self-supervised training.

[0094]

[0095] in, Represented as t Intermediate state at time 1 i Candidate training samples in l The hidden state of the layer.

[0096] Redundancy avoidance value , used to measure the semantic difference between a sample and the trained samples, is defined as:

[0097]

[0098] in, Represented as the first j The candidate training samples in the th l The hidden state when layered.

[0099] S32: Based on the value representation within the domain, verify the meaning of the representation of the core evaluation dimensions within the domain, predict the potential contribution of the sample to the improvement of the model's capabilities during formal training, so as to obtain a value perception evaluation system within the domain.

[0100] Specifically, by combining the domain-specific value representation and core evaluation dimensions of the samples, the potential contribution of candidate training samples to improving model capabilities is predicted. This process involves the following steps:

[0101] Verify the correlation between the evaluation dimensions: Verify whether each dimension can effectively reflect the contribution of the samples to the improvement of the model's capabilities. For example, is the uncertainty of the updated model proportional to the complexity of the samples, or can the samples trigger significant updates to the intermediate states of the model?

[0102] Constructing an evaluation system: Using the above analysis, we construct an evaluation system based on the value of samples within the domain. This system integrates various evaluation dimensions and can quantitatively evaluate the potential contribution of each candidate sample.

[0103] S33: Based on the value perception evaluation system, assign potential training value to candidate training samples;

[0104] Specifically, regarding sample value representation Preliminary weighted summation yields training value Using a constructed domain-specific value-perceived evaluation system, a potential training value is assigned to each candidate training sample. This value can be quantified based on the following formula:

[0105]

[0106] in, It is the first i The weights of the evaluation dimensions for each candidate training sample. For example, a sample has high potential training value if it excels in driving intermediate state updates and reducing model uncertainty.

[0107] S34: Based on the potential training value of the training samples, the multi-dimensional value features of the samples are quantified. The multi-dimensional value features include the update dimension of the intermediate state of the model triggered by the candidate training samples, the change dimension of the model inference consistency after the update, the change dimension of the model uncertainty before and after the update, and the dimension of the immediate impact on the accuracy of question answering in the vertical domain after the update.

[0108] Specifically, by quantifying these multi-dimensional value characteristics, we can comprehensively evaluate the specific contribution of each candidate sample to improving the model's capabilities, thereby providing a precise basis for sample selection.

[0109] S4: Combining the key training requirements of vertical domain question answering systems or retrieval question answering systems in the mid-term, and based on the potential training value and multi-dimensional value features, a dynamic weighting algorithm is used to allocate weights and select high-value samples as mid-term training samples; this step includes the following sub-steps:

[0110] S41: Based on the potential training value and multi-dimensional value features, a dynamic weighted algorithm is used to allocate weights and rank the candidate samples;

[0111] Specifically, based on the high-value samples, the comprehensive value score of each candidate sample is further calculated, and the screening threshold is dynamically adjusted according to the sample score distribution to select samples with scores exceeding the threshold. At the same time, the idempotency test is introduced as a training technique to evaluate the stability of sample value, ensuring that the selected samples have continuous training value, and finally forming the core training sample set for active learning.

[0112] The multidimensional values ​​of each candidate sample are weighted and fused to obtain a comprehensive value score. :

[0113]

[0114] in, The weight parameters are dynamically adjusted based on mid-term training needs, and the selection threshold is dynamically determined based on the sample score distribution. In this case, the score is 90, and the following criteria are met:

[0115]

[0116] S42: Based on the sorted samples, the samples are screened, and samples with value scores higher than the dynamic threshold are selected under the training budget constraint.

[0117] S43: Integrate all samples above the dynamic threshold to construct a high-value sample subset for mid-term training, which will serve as mid-term training samples;

[0118] Specifically, high-value samples are integrated: all candidate training samples with scores higher than the dynamic threshold are integrated into a high-value sample subset. This subset contains the high-value samples selected in the aforementioned screening process, ensuring that these samples can contribute to improving the model's capabilities to the greatest extent.

[0119] Constructing an intermediate training sample set: These high-value samples are used as intermediate training samples for further training of the large language model. Since these samples have a significant potential contribution to the model, using them for training can effectively improve the model's performance, especially in question-answering tasks in vertical domains.

[0120] Ensure sample diversity and representativeness: When constructing a high-value sample subset, it is essential to guarantee the diversity and representativeness of the samples to avoid the model focusing solely on training data from certain specific domains. Sample diversity can be optimized through domain coverage, task type, etc., ensuring that the model performs well across various tasks and scenarios.

[0121] By integrating and filtering high-value samples, a subset of high-value samples is constructed for mid-term training, thereby optimizing the training effect of the model and improving the accuracy and matching efficiency of question answering in vertical fields.

[0122] S5: Incrementally train the large language model using the intermediate training samples, and update the model parameters and the intermediate state changes of the model during training; this step includes the following sub-steps:

[0123] S51: During training, the overall value of the samples is prioritized, and parameter optimization is driven by high-value samples. Only local core parameters of the model are updated in a lightweight manner.

[0124] Specifically, attention weights for sample values Adjustments are made within the Transformer structure for the first... The self-attention calculation process for a layer is as follows:

[0125]

[0126] in , These represent the query vector and the key vector, respectively. This represents the dimension of the key vector.

[0127] In this example, a comprehensive sample value score is introduced. Attention weights Adjustments were made to obtain the second attention weight after optimizing the overall value score. :

[0128]

[0129] in, Indicates the first The overall value score of each training sample. This is the adjustment coefficient.

[0130] Through the above process, high-value samples are given higher weight in the attention allocation process, driving the model to pay more attention to samples containing high information density and key knowledge structures during the mid-term training process.

[0131] S52: During training, the model utilizes an in-model attention mechanism and a multilayer perceptron in the processing layer to optimize sample feature capture, enabling the model to focus on high-value features.

[0132] Specifically, in the feedforward network following the Transformer, a multilayer perceptron is used to model the intermediate representation:

[0133]

[0134] In this model, MLP is represented as a multilayer perceptron model. During the mid-term training phase, only the parameters related to high-value samples in the MLP are updated. For example, the gradients corresponding to high-value samples are given higher update weights, and the gradients of low-value samples are decayed or frozen.

[0135] Set update rules Represented as:

[0136]

[0137] in, This is represented as the gradient of the trainable parameters in the MLP. This represents the loss during sample training. This indicates that the updated weights can be freely adjusted.

[0138] To reduce computational overhead and the risk of model parameter shift, this example employs a lightweight parameter update strategy, specifically including:

[0139] Freeze the underlying general semantic parameters of the large model language, update only some attention parameters in the high-level Transformer, update only the parameters of the multilayer perceptron module, update the trainable intermediate adaptation parameters and fast weights, and control the number of update steps in the mid-term training of a single round to avoid overfitting high-value samples.

[0140] S53: Repeat S52 to obtain the updated training model parameters and the intermediate state changes of the samples during the training process;

[0141] S6: Based on the model parameters and the intermediate state changes of the model during training, obtain the evolution results of model performance changes, and dynamically update the sample value evaluation strategy to adaptively optimize the sample selection results during multiple rounds of intermediate training; this step includes the following sub-steps:

[0142] S61: After completing a round of intermediate training, based on the updated training model parameters and the intermediate state changes of samples during training, the performance changes of the model in vertical domain question answering or retrieval question answering tasks are obtained.

[0143] Specifically, the intermediate state evolution analysis records the intermediate state representations of high-value samples in the Transformer layer, comparing the Transformer before and after training. l The intermediate states in the layer are represented as follows:

[0144]

[0145] Define the intermediate state evolution index as :

[0146]

[0147] Intermediate state evolution index Used to characterize the extent to which samples affect the internal representation structure of the model during actual training.

[0148] S62: Establish a correlation between the performance changes and the intermediate states during sample training to obtain the correlation factor;

[0149] Specifically, by combining the performance changes of the model on the validation set, a training effect feedback signal is constructed:

[0150]

[0151] in This indicates the increase in question-and-answer accuracy. This indicates the increase in retrieval recall rate. This is the balance coefficient.

[0152] Feedback signal R A mapping relationship is established between the sample intermediate state evolution index and the relationship factor, which is used to evaluate the accuracy of sample value prediction.

[0153] S63: Based on the aforementioned correlation factors, the sample value assessment strategy is updated in reverse, making subsequent sample selection more biased towards intermediate state sensitive samples;

[0154] Specifically, based on the feedback results, the weight parameters in the sample comprehensive value scoring formula are updated:

[0155]

[0156] At the same time, dynamically adjust according to changes in the sample score distribution. The selection threshold makes subsequent training rounds more inclined to select samples that produce significant positive feedback in actual training.

[0157] S64: Retain the training adaptation rules and parameter optimization experience of high-value samples to provide data support for subsequent sample value assessment and model training;

[0158] S65: Repeat S61-S64, collect model feedback data after each iteration, dynamically optimize the dimensional weights and sample selection thresholds of the evaluation system until the convergence condition is met, and complete the accurate selection of mid-term training samples and efficient model optimization.

[0159] Specifically, the above-mentioned intermediate training, feedback update, and sample selection process is repeated, accumulating the training adaptation patterns and parameter optimization experience of high-value samples round by round, until the model performance changes meet the preset convergence conditions, completing the accurate selection of intermediate training samples and efficient model optimization, and finally based on the second attention weight. Based on the feedback results, the weight parameters in the comprehensive value scoring formula are updated to optimize the comprehensive value score. The weight parameters in the comprehensive value scoring formula are combined to ultimately output the optimized mid-term training sample set. :

[0160]

[0161] Figure 2 This is a block diagram illustrating a value-aware mid-term training sample selection device for a model, according to an exemplary embodiment. (Reference) Figure 2 The value-aware model mid-term training sample selection device provided in this embodiment of the invention may include:

[0162] Module 1 is used to acquire a set of candidate training samples for large language models in the mid-training stage and vertical domain question answering systems or retrieval question answering systems.

[0163] Module 2 is used to perform value-aware reasoning on the candidate training sample set based on the parameters and intermediate states of the current large language model, and to construct a domain-specific sample value representation.

[0164] Quantization module 3 is used to evaluate the potential training value of candidate training samples in improving model capabilities in the vertical domain based on the value representation of samples in the domain, and to quantify the multi-dimensional value features of each candidate sample.

[0165] The mid-term training sample selection module 4 is used to combine the mid-term training focus requirements of the vertical domain question answering system or retrieval question answering system, and based on the potential training value and multi-dimensional value features, use a dynamic weighting algorithm to allocate weights and select high-value samples as mid-term training samples.

[0166] Model update module 5 is used to incrementally train the large language model using the intermediate training samples and update the model parameters and the intermediate state changes of the model during training.

[0167] The sample selection module 6 is used to obtain the evolution results of model performance changes based on the model parameters and the intermediate state changes of the model during training, and to dynamically update the sample value evaluation strategy, thereby adaptively optimizing the sample selection results during multiple rounds of intermediate training.

[0168] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the value-aware model mid-term training sample selection method described above. Figure 3 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is a mid-term training sample selection device based on value perception provided in an embodiment of the present invention. (Except for...) Figure 3In addition to the processor and memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0169] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the value-aware model mid-term training sample selection method described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of a wind turbine, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0170] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0171] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for selecting mid-term training samples for a model based on value perception, characterized in that, include: Obtain a set of candidate training samples for large language models in the mid-training stage, as well as vertical domain question answering systems or retrieval question answering systems. Based on the parameters and intermediate states of the current large language model, value-aware reasoning is performed on the candidate training sample set to construct a domain-specific sample value representation. Based on the value representation of samples in the domain, the potential training value of candidate training samples in improving model capabilities in the vertical domain is evaluated, and the multi-dimensional value features of each candidate sample are quantified. Based on the key training requirements of vertical domain question answering systems or retrieval question answering systems in the middle stage, and according to the potential training value and multi-dimensional value characteristics, a dynamic weighting algorithm is used to allocate weights and select high-value samples as middle stage training samples. The large language model is incrementally trained using the intermediate training samples, and the model parameters and intermediate state changes of the model during training are updated. Based on the model parameters and the intermediate state changes of the model during training, the evolution results of model performance changes are obtained, and the sample value evaluation strategy is dynamically updated, thereby adaptively optimizing the sample selection results during multiple rounds of intermediate training. Specifically, performing value-aware reasoning on the candidate training sample set to construct a domain-specific sample value representation includes: Input the candidate training samples into the current large language model to perform question-answering reasoning or retrieval question-answering reasoning. While keeping the main parameters of the model frozen or partially frozen, record the output of the model and the intermediate states during the inference process. The intermediate states refer to the hidden state representations, attention weight distributions, trainable intermediate adaptation parameters, and fast weights or temporary state variables used to assist training in the Transformer layer. Without updating the model's main parameters, trainable constraints are introduced only to the intermediate states, and the intermediate states are repeatedly reconstructed and learned to enhance the model's intermediate states. A self-supervised or weakly supervised objective function is constructed based on candidate samples. The objective function is used to perform one or more rounds of lightweight updates on the intermediate state, and the change magnitude of the intermediate state before and after the update is recorded. Based on the changes in the model output and intermediate states of the inference process before and after the update, the value representation of samples in the domain is extracted.

2. The method according to claim 1, characterized in that, Obtain a set of candidate training samples for large language models in the mid-training stage, as well as vertical domain question answering systems or retrieval question answering systems, including: Obtain a large language model in the question-answering domain that has completed the initial pre-training and intermediate training phases; Monitor the performance trends of large language models on the validation sets of vertical domain question answering systems or retrieval question answering systems; When the overall performance of the model enters a stage of slow convergence but local capabilities continue to change, the model is determined to be in the mid-training stage. Candidate training samples are collected from the question-and-answer logs of the question-and-answer system, user queries and retrieval results from the question-and-answer system, synthetic question-and-answer pairs constructed from the question-and-answer system's document library, and candidate samples generated by the model but not yet used for training.

3. The method according to claim 1, characterized in that, Based on the value representation of samples within the aforementioned domain, the potential training value of candidate training samples in improving model capabilities within the vertical domain is evaluated, and the multi-dimensional value features of each candidate sample are quantified, including: Determine the representational meaning of at least three core evaluation dimensions within each domain; Based on the value representation within the domain, the meaning of the representation of the core evaluation dimensions within the domain is verified, and the potential contribution of the sample to the improvement of the model's capabilities during formal training is predicted, so as to obtain a value perception evaluation system within the domain. Based on the aforementioned value perception evaluation system, potential training value is assigned to candidate training samples; Based on the potential training value of the training samples, the multi-dimensional value features of the samples are quantified. These multi-dimensional value features include the update dimension of the intermediate state of the model triggered by the candidate training samples, the change dimension of the consistency of model inference after the update, the change dimension of the uncertainty of the model before and after the update, and the dimension of the immediate impact of the update on the accuracy of question answering in the vertical domain.

4. The method according to claim 1, characterized in that, Based on the key training requirements of vertical domain question answering systems or retrieval question answering systems in the mid-term, and according to the potential training value and multi-dimensional value features, a dynamic weighted algorithm is used to allocate weights and select high-value samples as mid-term training samples, including: Based on the potential training value and multi-dimensional value features, a dynamic weighted algorithm is used to allocate weights and rank the candidate samples. Based on the sorted samples, the samples are filtered, and samples with value scores higher than the dynamic threshold are selected under the training budget constraint. Integrate all samples above the dynamic threshold to construct a high-value sample subset for mid-term training, which will serve as the mid-term training samples.

5. The method according to claim 1, characterized in that, The large language model is incrementally trained using the intermediate training samples, and the model parameters and intermediate state changes during training are updated, including: S51: During training, the overall value of the samples is prioritized, and parameter optimization is driven by high-value samples. Only local core parameters of the model are updated in a lightweight manner. S52: During training, the model utilizes an in-model attention mechanism and a multilayer perceptron in the processing layer to optimize sample feature capture, enabling the model to focus on high-value features. S53: Repeat S52 to obtain updated training model parameters and intermediate state changes of samples during training.

6. The method according to claim 1, characterized in that, Based on the model parameters and the intermediate state changes of the model during training, the evolution results of model performance changes are obtained, and the sample value evaluation strategy is dynamically updated to adaptively optimize the sample selection results during multiple rounds of intermediate training, including: S61: After completing a round of mid-term training, based on updating the training model parameters and the intermediate state changes of samples during training, obtain the performance changes of the model in vertical domain question answering or retrieval question answering tasks. S62: Establish a correlation between the performance changes and the intermediate states during sample training to obtain the correlation factor; S63: Based on the aforementioned correlation factors, the sample value assessment strategy is updated in reverse, making subsequent sample selection more biased towards intermediate state sensitive samples; S64: Retain the training adaptation rules and parameter optimization experience of high-value samples to provide data support for subsequent sample value assessment and model training; S65: Repeat S61-S64, collect model feedback data after each iteration, dynamically optimize the dimensional weights and sample selection thresholds of the evaluation system until the convergence condition is met, and complete the accurate selection of training samples and efficient model optimization in the mid-term.

7. A device for selecting mid-term training samples for a model based on value perception, characterized in that, include: The acquisition module is used to acquire a set of candidate training samples for large language models in the mid-training stage, as well as vertical domain question answering systems or retrieval question answering systems. The construction module is used to perform value-aware reasoning on the candidate training sample set based on the parameters and intermediate states of the current large language model, and to construct a domain-specific sample value representation. The quantization module is used to evaluate the potential training value of candidate training samples in improving model capabilities in the vertical domain based on the value representation of samples in the domain, and to quantify the multi-dimensional value features of each candidate sample. The mid-term training sample selection module is used to combine the key mid-term training requirements of vertical domain question answering systems or retrieval question answering systems, and select high-value samples as mid-term training samples by using a dynamic weighting algorithm to allocate weights based on the potential training value and multi-dimensional value features. The model update module is used to incrementally train the large language model using the intermediate training samples and update the model parameters and the intermediate state changes of the model during training. The sample selection module is used to obtain the evolution results of model performance changes based on the model parameters and the intermediate state changes of the model during training, and to dynamically update the sample value evaluation strategy, thereby adaptively optimizing the sample selection results during multiple rounds of intermediate training. Specifically, performing value-aware reasoning on the candidate training sample set to construct a domain-specific sample value representation includes: Input the candidate training samples into the current large language model to perform question-answering reasoning or retrieval question-answering reasoning. While keeping the main parameters of the model frozen or partially frozen, record the output of the model and the intermediate states during the inference process. The intermediate states refer to the hidden state representations, attention weight distributions, trainable intermediate adaptation parameters, and fast weights or temporary state variables used to assist training in the Transformer layer. Without updating the model's main parameters, trainable constraints are introduced only to the intermediate states, and the intermediate states are repeatedly reconstructed and learned to enhance the model's intermediate states. A self-supervised or weakly supervised objective function is constructed based on candidate samples. The objective function is used to perform one or more rounds of lightweight updates on the intermediate state, and the change magnitude of the intermediate state before and after the update is recorded. Based on the changes in the model output and intermediate states of the inference process before and after the update, the value representation of samples in the domain is extracted.

8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-6.

Images