Instruction tuning methods, question-answering methods, and electronic devices for large language models

Abstract

This application relates to the fields of artificial intelligence and natural language processing technology, and discloses a method for instruction tuning, question answering, and electronic device for large language models. The method further includes: acquiring an instruction dataset; determining the sample instruction sequences under different evaluation dimensions and the instruction selection range corresponding to each training step; acquiring the current model parameters of the large language model at the current training step and determining the state feature vector corresponding to the current model parameters; using the state feature vector to determine the sampling weights corresponding to each sample instruction sequence at the current training step; using the sampling weights within the instruction selection range corresponding to the current training step to extract training sample instructions from each sample instruction sequence; adjusting the current model parameters using all training sample instructions to obtain the current model update parameters, until the training process of all training steps is completed in a loop to obtain the target large language model, which can improve the efficiency and performance of model training.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and natural language processing technology, specifically to instruction tuning methods for large language models, question answering methods, and electronic devices. Background Technology

[0002] Currently, instruction tuning for large language models mainly focuses on two aspects: data selection and training order optimization. However, instruction tuning based on data selection is often a one-time static selection, which can only determine data selection but cannot dynamically arrange the training order. This can easily lead to the omission of usable samples later and incur additional data evaluation overhead. In contrast, instruction tuning based on training order optimization relies on a single dimension for judging the difficulty of sample instructions and has a fixed training order. This makes it difficult to adjust in real time according to the model's capabilities and can easily cause learning conflicts in multi-task scenarios. Ultimately, this results in low training efficiency for large language models and makes it difficult for the overall learning performance of large language models to achieve the desired effect. Summary of the Invention

[0003] This application provides a method for instruction tuning, a question-answering method, and an electronic device for large language models, in order to solve the problem of low training efficiency and training effect of large language models.

[0004] Firstly, this application provides a method for instruction tuning of a large language model. The method includes: acquiring an instruction dataset; determining the sample instruction sequences of the instruction dataset under different evaluation dimensions and the instruction selection range corresponding to each training step; acquiring the current model parameters of the large language model at the current training step; determining the state feature vector corresponding to the current model parameters; using the state feature vector to determine the sampling weights corresponding to each sample instruction sequence at the current training step; using the sampling weights within the instruction selection range corresponding to the current training step to extract training sample instructions from each sample instruction sequence; adjusting the current model parameters using all training sample instructions to obtain the current model update parameters, until the training process of all training steps is completed in a loop to obtain the target large language model, which is used to perform question answering tasks.

[0005] The instruction tuning method for a large language model provided in this embodiment determines the state feature vector of the large language model in the current training step based on the current model parameters of the large language model in the current training step; then, the sampling weights corresponding to each sample instruction sequence can be determined based on the state feature vectors; and further, the training sample instructions corresponding to the current training step can be determined relatively accurately based on the sampling weights. This application dynamically and adaptively selects the training sample instructions corresponding to the current training step, eliminating the need for manual data screening, and the instruction optimization order is relatively flexible, which can take into account both existing high-quality sample instructions and the effective exploration of high-value unknown sample instructions. Therefore, by using the training sample instructions selected in each training step to adjust the parameters of the large language model corresponding to that training step, the training efficiency of the large language model can be improved, and the performance of the final large language model can be ensured to be good.

[0006] In one optional implementation, the current model parameters include the sub-model parameters corresponding to the last network layer of the large language model; determining the state feature vector corresponding to the current model parameters includes: using a pre-trained state extraction model to extract features from the sub-model parameters to obtain the state feature vector; wherein, the state extraction model is configured to: extract features from the sub-model parameters to obtain the gradient feature vector, cross-entropy distribution vector, and memory retention rate vector corresponding to the current training step; and concatenate the gradient feature vector, cross-entropy distribution vector, and memory retention rate vector to obtain the state feature vector.

[0007] By extracting gradient feature vectors, cross-entropy distribution vectors, and memory retention rate vectors from the sub-model parameters using a state extraction model, and then concatenating the extracted feature vectors to obtain state feature vectors, we can achieve an accurate evaluation of the capabilities of the large language model in the current training step and provide a reliable basis for the subsequent dynamic generation of sampling weights for each sample instruction sequence.

[0008] In an optional implementation, the method further includes: updating the model parameters of the state extraction model by utilizing the performance difference of the large language model before and after the update in the current training step to obtain a new state extraction model; wherein the new state extraction model is used to extract the state feature vector corresponding to the updated parameters of the current model in the next training step.

[0009] By leveraging the performance difference of the large language model before and after the current training step update, the model parameters of the state extraction model are updated. This allows the new state extraction model to extract the state feature vector corresponding to the next training step more accurately.

[0010] In one optional implementation, the state extraction model is updated by utilizing the performance difference before and after the large language model update in the current training step to obtain a new state extraction model. This includes: constructing a state correction value using the loss value before and after the large language model update in the current training step; using the state correction value as a supervision signal and constructing a confidence constraint using the neural confidence upper bound to update the model parameters of the state extraction model to obtain a new state extraction model.

[0011] By constructing a state correction value using the loss values ​​before and after updating the large language model, the learning gain and forgetting loss of the large language model at each training step can be accurately quantified. Using this state correction value as a supervision signal, combined with the neural confidence upper bound constraint, the state extraction model is updated, realizing the adaptive iterative optimization of the state extraction model. This ensures that the state extraction model updated in the current training step can better extract the state feature vector of the large language model in the next training step, and thus more accurately extract the sampling weights corresponding to the next training step.

[0012] In one optional implementation, the sampling weights corresponding to the current training step are determined using the state feature vector, including: inputting the state feature vector into a pre-trained weight determination model, and the weight determination model performing Dirichlet sampling on the state feature vector to obtain the sampling weights corresponding to each sample instruction sequence.

[0013] By using the weight-based model to perform Dirichlet sampling on the state feature vector, the sampling weights of each sample instruction sequence can be adaptively adjusted according to the learning state of the large language model at different training steps, enabling adaptive emphase sampling of high-value sample instruction sequences.

[0014] In one optional implementation, determining the sequence of sample instructions in the instruction dataset under different evaluation dimensions includes: for each evaluation dimension, evaluating the difficulty of each sample instruction in the instruction dataset according to the difficulty evaluation strategy under the evaluation dimension, and obtaining the difficulty evaluation index corresponding to each sample instruction; sorting all the sample instructions from easy to difficult according to the difficulty evaluation index to obtain the sequence of sample instructions under the evaluation dimension, wherein one evaluation dimension corresponds to one difficulty evaluation strategy.

[0015] By setting different difficulty assessment strategies for different assessment dimensions, and then assessing the difficulty of each sample instruction in the instruction dataset according to the difficulty assessment strategy under the corresponding assessment dimension, the difficulty assessment of each sample instruction can be performed more accurately. Furthermore, based on the difficulty assessment index corresponding to each sample instruction, an orderly and progressive training logic can be formed, which can further improve the rationality of instruction tuning for large language models.

[0016] In one optional implementation, when the difficulty assessment strategy is an interaction trajectory assessment strategy, the difficulty of each sample instruction in the instruction dataset is assessed according to the difficulty assessment strategy under the assessment dimension to obtain the difficulty assessment index corresponding to each sample instruction. This includes: for each sample instruction, obtaining the belief evolution entropy, causal reasoning topological complexity, and trajectory flow alignment deviation corresponding to the sample instruction; and performing weighted fusion of the belief evolution entropy, causal reasoning topological complexity, and trajectory flow alignment deviation to obtain the interaction trajectory difficulty assessment index corresponding to the sample instruction.

[0017] By comprehensively evaluating sample instructions using three different indicators—belief evolution entropy, causal reasoning topological complexity, and trajectory manifold alignment deviation—the true difficulty of the interaction trajectory can be comprehensively and objectively characterized. By weighted fusion, the interaction trajectory difficulty evaluation indicators corresponding to each sample instruction are obtained. This not only preserves the effective information of each indicator but also normalizes the difficulty of the sample instructions. Subsequently, the interaction trajectory difficulty evaluation indicators can be used to sort the sample instructions more accurately, further ensuring that the sample instruction sequence corresponding to the obtained interaction trajectory evaluation strategy is more accurate.

[0018] In an optional implementation, the method further includes: after obtaining the current model update parameters, and before determining the state feature vector corresponding to the next training step, using the current model update parameters to re-evaluate the difficulty of sample instructions that have not participated in training in the target sample instruction sequence, to obtain a new sample instruction sequence, the new sample instruction sequence being used to extract the training sample instructions corresponding to the next training step; wherein, the target sample instruction sequence is a sample instruction sequence obtained by difficulty evaluation based on a capability-aware difficulty evaluation strategy.

[0019] By using a large language model with updated parameters (i.e., the large language model obtained after the current training step), the difficulty of sample instructions that have not participated in training in the target sample instruction sequence can be reassessed. This ensures that the order of sample instructions that have not yet participated in training in the new sample instruction sequence can better adapt to the current capability state of the large language model, and further improves the instruction tuning efficiency and effect of the large language model.

[0020] Secondly, this application provides a question-answering method, which includes: receiving a question text to be answered; and processing the question text to be answered using a target large language model of the first aspect or any embodiment thereof to obtain a target answer.

[0021] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the instruction tuning method of the large language model described in the first aspect or any corresponding embodiment, or the question-answering method described in the second aspect.

[0022] Fourthly, this application provides a computer-readable storage medium storing computer instructions, which are used to cause a computer to execute the instruction tuning method of the large language model of the first aspect or any corresponding embodiment thereof, or the question-and-answer method of the second aspect.

[0023] Fifthly, this application provides a computer program product, including computer instructions, which are used to cause a computer to execute the instruction tuning method of the large language model of the first aspect or any corresponding embodiment described above, or the question-and-answer method of the second aspect described above. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this application, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a schematic flowchart of the first method for instruction tuning of a large language model according to an embodiment of this application. Figure 2 This is a second flowchart illustrating the instruction tuning method for a large language model according to an embodiment of this application. Figure 3 This is a schematic diagram of the third method for instruction tuning of a large language model according to an embodiment of this application. Figure 4 This is a structural block diagram of an instruction tuning apparatus for a large language model according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0028] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0029] Currently, instruction tuning for large language models mainly focuses on two aspects: data selection and training order optimization. Data selection-based instruction tuning primarily improves training effectiveness by selecting higher-quality, more representative, or more suitable subsets of data. Training order optimization-based instruction tuning mainly improves the training process and enhances model training performance by designing the input order of training sample instructions, allowing the large language model to learn progressively according to a certain level of difficulty.

[0030] However, large language model instruction tuning methods based on data screening are mostly one-time static screening methods. They can only determine data selection but cannot dynamically arrange the training order, which easily leads to the omission of usable samples later and generates additional data evaluation overhead. In contrast, large language model instruction tuning methods based on training order optimization have a single dimension for judging the difficulty of sample instructions and a fixed training order. They are difficult to adjust in real time with the model's capabilities and are prone to learning conflicts in multi-task scenarios. Ultimately, this results in low training efficiency of large language models and makes it difficult for the overall learning performance of large language models to achieve the desired effect.

[0031] Based on this, this application evaluates the difficulty of sample instructions in the instruction dataset from multiple evaluation dimensions, including static text statistics, dynamic model feedback, environmental state transition, and information topology compression, to obtain the sample instruction sequence under each evaluation dimension. Subsequently, during the training process, the extraction process of training sample instructions in each training step (also known as the selection process) is modeled as a partially observable Markov decision process. A context-aware multi-armed slot machine based on neural confidence upper bound is used as the state extraction model to perceive the capability state and gradient conflict of the large language model in real time, and output the sampling weights corresponding to each sample instruction sequence in each training step. The training sample instructions corresponding to each training step are then extracted according to the sampling weights. Finally, the parameters of the large language model in each training step are updated according to the training sample instructions corresponding to each training step until the training process of all training steps is completed in a loop, resulting in the final instruction-optimized target large language model, which can be used to perform question answering tasks.

[0032] The large language model instruction tuning method proposed in this application can overcome the problem that instruction tuning based on data screening mainly focuses on data screening and lacks dynamic organization of the training process. It can also overcome the problem that instruction tuning based on training order optimization relies heavily on static difficulty indicators, has a fixed training path, and is difficult to adaptively adjust according to changes in model capabilities. Therefore, it can improve the training efficiency in the large language model instruction tuning process and the overall performance of the final large language model.

[0033] According to an embodiment of this application, an embodiment of an instruction tuning method for a large language model is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] This embodiment provides a method for instruction tuning of a large language model, which can be used in electronic devices such as servers or computers. Figure 1 This is a flowchart of an instruction tuning method for a large language model according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain the instruction dataset and determine the sample instruction sequences of the instruction dataset under different evaluation dimensions and the instruction selection range corresponding to each training step.

[0035] The instruction dataset can be a dataset used for instruction tuning of large language models. It can include multiple sample instructions for large language models. These sample instructions can include, but are not limited to, regular question-and-answer instructions, multi-step logical reasoning instructions, and high-order instructions for dynamic interaction between the agent and the external environment.

[0036] The evaluation dimension here can be a dimension that evaluates the difficulty of sample instructions. Each evaluation dimension can correspond to a difficulty evaluation strategy. Through this difficulty evaluation strategy, the difficulty of multiple sample instructions in the instruction dataset can be evaluated to obtain a difficulty evaluation index. Then, based on the difficulty evaluation index, the multiple sample instructions can be sorted from easy to difficult, thus obtaining a sequence of sample instructions corresponding to the evaluation dimension.

[0037] For example, the data length difficulty index of each sample instruction can be calculated, and all sample instructions can be sorted by the data length difficulty index to obtain the sample instruction sequence under this evaluation dimension.

[0038] For example, the text vocabulary diversity difficulty index of each sample instruction can be calculated, and all sample instructions can be sorted by the text vocabulary diversity difficulty index to obtain the sample instruction sequence under this evaluation dimension.

[0039] In some cases, to complete the training process corresponding to a training step, it is necessary to first extract the training sample instructions for that training step. During the extraction of training sample instructions, the instruction selection range for each training step's instruction sequence can be determined first; then, training sample instructions falling within the instruction selection range are extracted from each instruction sequence. Within the same training step, the instruction selection ranges for each instruction sequence can be the same or different.

[0040] As a concrete example, training steps The corresponding instruction selection range can be represented as Furthermore It can be determined in the following way: ; in, This is used to represent the upper bound of the instruction selection range corresponding to the t-th training step. , T is used to represent the total number of training steps. It is used to represent the lower bound of the instruction selection range corresponding to the t-th training step, and it is also the upper bound of the instruction selection range corresponding to the (t-1)-th training step.

[0041] As a specific example, let's set the total number of training steps to 10. Therefore, the instruction selection range for the first training step is [0, 0.01]; the instruction selection range for the second training step is [0.01, 0.4473]; and so on, without further explanation.

[0042] Step S102: Obtain the current model parameters of the large language model in the current training step, and determine the state feature vector corresponding to the current model parameters.

[0043] Here, the current model parameters of the large language model corresponding to each training step can be the model parameters updated in the previous training step. For example, if the current training step is the first training step in the training process, the current model parameters of the large language model in the current training step can be the model parameters of the initial large language model; if the current training step is the second training step in the training process, the current model parameters of the large language model in the current training step can be the model parameters obtained after the first training step update, and so on.

[0044] State feature vectors can be used to characterize the current capability state of a large language model at the current training step. As a concrete example, a pooling network layer can be connected after the last network layer of the large language model; and this pooling network layer can be used to perform average pooling on the sub-model parameters corresponding to the last network layer of the large language model at the current training step to obtain a state feature vector with a fixed evaluation dimension.

[0045] Step S103: Using the state feature vector, determine the sampling weights corresponding to each sample instruction sequence in the current training step.

[0046] The sampling weights here can be the weights corresponding to each sample instruction sequence in the current training step. As a specific example, if the total number of sample instruction sequences is 5, then the sampling weights corresponding to the current training step can be represented as [0.1, 0.3, 0.1, 0.3, 0.2]. This can be used to indicate that the sampling weight corresponding to the first sample instruction sequence is 0.1, the sampling weight corresponding to the second sample instruction sequence is 0.3, and so on.

[0047] In some cases, the state feature vector can be used as the query feature, and learnable key features and value features can be configured for each sample instruction sequence. Finally, the attention score corresponding to each sample instruction sequence is calculated through the attention mechanism, and then the attention score is normalized to obtain the sampling weight corresponding to the current training step.

[0048] In other cases, the state feature vector can be input into a fully connected linear layer to obtain the original scores corresponding to each sample instruction sequence; then the original scores can be normalized by the softmax function to obtain the sampling weights corresponding to the current training step.

[0049] Step S104: Using the sampling weights, extract training sample instructions from each sample instruction sequence within the instruction selection range corresponding to the current training step.

[0050] Here, the total number of training sample instructions can be preset; then, the number of samples to be extracted for each sample instruction sequence can be determined by combining the sampling weights corresponding to the current training step; subsequently, within the instruction selection range corresponding to the current training step for each sample instruction sequence, the corresponding number of sample instructions can be extracted as the training sample instructions corresponding to the current training step.

[0051] As a specific example, let the current training step be the first training step in the training process. The instruction selection range corresponding to each sample instruction sequence is [0, 0.01], the sampling weight is [0.1, 0.3, 0.1, 0.3, 0.2], and the total number of training samples is 10. Therefore, one sample instruction with a difficulty evaluation index in the range of [0, 0.01] is randomly selected from the first sample instruction sequence as a training sample instruction, and two sample instructions with a difficulty evaluation index in the range of [0, 0.01] are randomly selected from the second sample instruction sequence as training sample instructions.

[0052] Step S105: Adjust the current model parameters using all training sample instructions to obtain the current model update parameters, until the training process of all training steps is completed in a loop, and the target large language model is obtained. The target large language model is used to perform question answering tasks.

[0053] Here, all training sample instructions corresponding to the current training step can be input into the large language model corresponding to the current training step to obtain the prediction result output by the large language model corresponding to the current training step; then, the loss value of the large language model corresponding to the current training step can be calculated based on the prediction result and the true label result corresponding to each training sample instruction; finally, the current model parameters corresponding to the large language model of the current training step can be adjusted using the loss value to obtain the current model update parameters, thereby obtaining the large language model trained by the current training step.

[0054] Furthermore, the specific process of adjusting the current model parameters corresponding to the large language model in the current training step using the loss value to obtain the current model update parameters is as follows: backpropagation can be performed based on the calculated loss value to solve for the gradient information of the large language model parameters, and the gradient can be constrained and corrected through the gradient projection strategy to remove invalid gradient noise and control the parameter update magnitude; based on the corrected effective gradient, gradient descent iterative update is performed on the large language model parameters to complete the parameter optimization of the large language model in the current training step, and the current model update parameters in the current training step are obtained.

[0055] Furthermore, the current model update parameters obtained here are also the current model parameters for the large language model in the next training step. Therefore, the processing flow of each of the aforementioned training steps is executed repeatedly, iteratively updating the model parameters round by round, until all training steps are completed or the preset training termination condition is met, such as the model performance reaching the optimal level. At this point, the loop stops, and the current model update parameters obtained in the last loop are used as the final model parameters of the large language model, thus obtaining the final target large language model.

[0056] The instruction tuning method for a large language model provided in this embodiment determines the state feature vector of the large language model in the current training step based on the current model parameters of the large language model in the current training step; then, the sampling weights corresponding to each sample instruction sequence can be determined based on the state feature vectors; and further, the training sample instructions corresponding to the current training step can be determined relatively accurately based on the sampling weights. This application dynamically and adaptively selects the training sample instructions corresponding to the current training step, eliminating the need for manual data screening, and the instruction optimization order is relatively flexible, which can take into account both existing high-quality sample instructions and the effective exploration of high-value unknown sample instructions. Therefore, by using the training sample instructions selected in each training step to adjust the parameters of the large language model corresponding to that training step, the training efficiency of the large language model can be improved, and the performance of the final large language model can be ensured to be good.

[0057] This embodiment provides a method for instruction tuning of a large language model, which can be used in electronic devices such as servers or computers. Figure 2 This is a flowchart of an instruction tuning method for a large language model according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain the instruction dataset and determine the sample instruction sequences under different evaluation dimensions, as well as the instruction selection range corresponding to each training step. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0058] Step S202: Obtain the current model parameters of the large language model in the current training step, and determine the state feature vector corresponding to the current model parameters.

[0059] In some cases, features can be directly extracted from the current model parameters of the large language model at the current training step to obtain the state feature vector corresponding to the current model parameters. Since the current model parameters are all the model parameters of the large language model, this allows for an accurate assessment of the true capability state of the large language model at the current training step.

[0060] In other cases, in order to reduce the amount of computation and improve the efficiency of instruction tuning for large language models, feature extraction can be performed on only a portion of the parameters of the current model in the current training step, thereby obtaining the state feature vector of the large language model in the current training step.

[0061] Specifically, the current model parameters corresponding to the current training step may include the sub-model parameters corresponding to the last network layer of the large language model. Step S202 above includes: Step S2021: Use the pre-trained state extraction model to extract features from the sub-model parameters to obtain a state feature vector; wherein, the state extraction model is configured to: extract features from the sub-model parameters to obtain the gradient feature vector, cross-entropy distribution vector, and memory retention rate vector corresponding to the current training step; and concatenate the gradient feature vector, cross-entropy distribution vector, and memory retention rate vector to obtain the state feature vector.

[0062] The gradient feature vector can be used to characterize the current convergence state and parameter update trend of the large language model in the current training step. Specifically, it can be the moving average gradient feature corresponding to the sub-model parameters. This gradient feature vector can be obtained by accumulating the gradients of multiple historical training steps and smoothing them. It can stably reflect the update direction, gradient magnitude, and convergence speed of the current model parameters of the large language model in the current training step, accurately reflect the current learning iteration status of the large language model, and thus effectively avoid the interference of single-step gradient noise on state judgment.

[0063] The cross-entropy distribution vector can be used to characterize the fit and cognitive uncertainty of a large language model to the training sample instructions corresponding to the previous training step. This cross-entropy distribution vector is obtained by calculating and normalizing the predicted cross-entropy values ​​of the large language model for various difficulty levels and training sample instructions. It can intuitively reflect the differences in mastery, cognitive bias, and fitting blind spots of the large language model for different tasks in the current training step, demonstrating the adaptability of the large language model to various training tasks in the current training step.

[0064] The memory retention rate vector can be used to characterize the retention of historical basic task capabilities by a large language model. This memory retention rate vector can be obtained by calculating the performance retention rate of the large language model on historically trained basic instruction tasks. It can quantify whether the large language model has experienced capability forgetting or performance degradation during iterative updates, and can accurately characterize the retention degree of historical capabilities of the large language model, providing a state basis for subsequent balancing of training on new and old tasks and avoiding catastrophic forgetting.

[0065] In some cases, the state extraction model can be the context-aware multi-armed slot machine model shown above. As a specific example, in the... In each training step, the sub-model parameters can be input into the state extraction model, which can extract the state feature vector of the large language model in the current training step. To comprehensively characterize the capability state of a large language model at the current training step, the state feature vector... It can be composed of three cascaded parts: ; in, This is used to represent the gradient feature vector corresponding to the t-th training step, characterizing the convergence trend of the large language model. This is used to represent the cross-entropy distribution vector corresponding to the t-th training step. This is used to represent the memory retention rate vector corresponding to the t-th training step.

[0066] By extracting gradient feature vectors, cross-entropy distribution vectors, and memory retention rate vectors from the sub-model parameters using a state extraction model, and then concatenating the extracted feature vectors to obtain state feature vectors, we can achieve an accurate evaluation of the capabilities of the large language model in the current training step and provide a reliable basis for the subsequent dynamic generation of sampling weights for each sample instruction sequence.

[0067] In some alternative implementations, the method further includes: Step a1: Utilize the performance difference of the large language model before and after the update in the current training step to update the model parameters of the state extraction model and obtain a new state extraction model; wherein, the new state extraction model is used to extract the state feature vector corresponding to the updated parameters of the current model in the next training step.

[0068] The performance difference here can be seen as the difference in the large language model's sample prediction accuracy, loss value, and task fitting effect before and after the parameter update in the same training step.

[0069] By leveraging the performance difference of the large language model before and after the current training step update, the model parameters of the state extraction model are updated. This allows the new state extraction model to more accurately extract the state feature vector corresponding to the next training step. Furthermore, it should be understood that the updated parameters of the current model for the current training step are the same as the current model parameters for the next training step.

[0070] In some alternative implementations, step a1 above includes: Step a11: Construct the state correction value using the loss values ​​before and after the large language model update in the current training step.

[0071] Step a12: Use the state correction as a supervision signal and construct confidence constraints using the neural confidence upper bound to update the model parameters of the state extraction model and obtain a new state extraction model.

[0072] As a concrete example, the state correction can be calculated based on the difference in performance parameters of the large language model before and after the current training step update. This, combined with a neural network and a confidence boundary strategy, enables iterative parameter updates for the state extraction model. The specific process is as follows: First, it can be done through Calculate the state correction for the current training step, where, Used to represent the state correction amount corresponding to the t-th training step. This is used to represent the loss value corresponding to the instruction of the i-th training sample at training step t. Used to represent the weight corresponding to the instruction of the i-th training sample in the t-th training step. This is used to represent the baseline loss value of the large language model on a global reserved set, such as a general instruction set, after the update at the t-th training step. This represents the baseline loss of the large language model on the global holdout set before the update at the t-th training step, which is also the baseline loss of the large language model on the global holdout set after the update at the (t-1)-th training step. The value is used to represent the forgetting penalty coefficient, and n represents the total number of training sample instructions at the t-th training step.

[0073] Secondly, the deep neural network in the state extraction model can be used to process the state feature vector and sampling weights corresponding to the current training step to output the expected state correction; at the same time, the feature gradient and covariance matrix are combined to construct the confidence boundary of action uncertainty, thereby balancing the model's exploration and utilization capabilities. Solving for the optimal action and refining the state evaluation logic, among which, Used to represent feature gradients This is used to represent the covariance matrix corresponding to the t-th training step. Used to represent the exploration coefficient Used to represent the desired state correction amount of the deep neural network output.

[0074] Finally, based on the state correction values ​​obtained from the quantization and the state evaluation results corresponding to the optimal action, the model parameters of the state extraction model are updated iteratively in reverse order to obtain the updated state extraction model, i.e., the new state extraction model. The updated state extraction model can more accurately extract the state feature vectors corresponding to the real-time model parameters of the large language model in the next training step.

[0075] By constructing a state correction value using the loss values ​​before and after updating the large language model, the learning gain and forgetting loss of the large language model at each training step can be accurately quantified. Using this state correction value as a supervision signal, combined with the neural confidence upper bound constraint, the state extraction model is updated, realizing the adaptive iterative optimization of the state extraction model. This ensures that the state extraction model updated in the current training step can better extract the state feature vector of the large language model in the next training step, and thus more accurately extract the sampling weights corresponding to the next training step.

[0076] Step S203: Using the state feature vector, determine the sampling weights corresponding to each sample instruction sequence in the current training step.

[0077] Specifically, step S203 includes: Step S2031: Input the state feature vector into the pre-trained weight determination model. The weight determination model performs Dirichlet sampling on the state feature vector to obtain the sampling weights corresponding to each sample instruction sequence.

[0078] In some cases, the weight determination model can be a Dirichlet policy network, and further, the Dirichlet policy network can be a submodule in the state extraction model.

[0079] In addition, Dirichlet sampling can be a probability sampling operation based on the Dirichlet distribution, commonly used to generate a set of multidimensional probability values ​​that sum to 1 and take values ​​between 0 and 1.

[0080] Here, the state feature vector corresponding to the t-th training step is input into the pre-trained weight determination model. The weight determination model can fit the Dirichlet distribution parameters that fit the current model state based on the received state feature vector and construct a multidimensional Dirichlet probability distribution that fits the current training scenario. Then, based on the multidimensional Dirichlet probability distribution constructed in real time, probability sampling is performed to randomly generate weight probability vectors that meet the distribution constraints. Finally, each weight probability vector is normalized so that the sum of the sampled weights of all sample instruction sequences is 1, thereby obtaining the effective sampled weights corresponding to each sample instruction sequence in the current training step.

[0081] By using the weight-based model to perform Dirichlet sampling on the state feature vector, the sampling weights of each sample instruction sequence can be adaptively adjusted according to the learning state of the large language model at different training steps, enabling adaptive emphase sampling of high-value sample instruction sequences.

[0082] Step S204: Using the sampling weights, extract training sample instructions from each sample instruction sequence within the instruction selection range corresponding to the current training step. For details, please refer to [link to details]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0083] Step S205: Adjust the current model parameters using all training sample instructions to obtain the updated model parameters. Repeat this process until all training steps are completed, resulting in the target large language model. This target large language model is used to perform the question-answering task. For details, please refer to [link to details]. Figure 1 Step S105 of the illustrated embodiment will not be described again here.

[0084] The instruction tuning method for a large language model provided in this embodiment can accurately extract the state feature vector at each training step through a dynamically updated state extraction model. Then, the state feature vector is input into a weight determination model, which can output the sampling weights corresponding to each sample instruction sequence that fits each training step. Based on the sampling weights, the training sample instructions that fit each training step can be determined. By using the training sample instructions for each training step, the large language model can be trained better for that training step. This can improve the efficiency of instruction tuning for the large language model and ensure that the overall performance of the final target large language model is better.

[0085] This embodiment provides a method for instruction tuning of a large language model, which can be used in electronic devices such as servers or computers. Figure 3 This is a flowchart of an instruction tuning method for a large language model according to an embodiment of this application, such as... Figure 3 As shown, the process includes the following steps: Step S301: Obtain the instruction dataset and determine the sample instruction sequences of the instruction dataset under different evaluation dimensions and the instruction selection range corresponding to each training step.

[0086] Specifically, step S301 includes: Step S3011: Obtain the instruction dataset. Please refer to the previous text for details, which will not be repeated here.

[0087] Step S3012: For each evaluation dimension, perform difficulty evaluation on each sample instruction in the instruction dataset according to the difficulty evaluation strategy under the evaluation dimension, and obtain the difficulty evaluation index corresponding to each sample instruction.

[0088] In the data length difficulty assessment dimension, the sum of the total number of input tokens and the total number of output tokens for each sample instruction can be calculated as the data length difficulty index corresponding to that sample instruction.

[0089] As a concrete example, the data length difficulty assessment metric can be expressed as: ;in, This is used to indicate the difficulty index of the data length corresponding to the sample instruction. Used to indicate the total number of input tokens corresponding to the sample instruction. This is used to represent the total number of output tokens corresponding to a sample instruction. The smaller the data length difficulty assessment index of a sample instruction, the easier the sample instruction is considered; conversely, the larger the data length difficulty assessment index of a sample instruction, the more difficult the sample instruction is considered.

[0090] In the dimension of assessing the difficulty of textual lexical diversity, the Measure of Textual Lexical Diversity (MTLD) can be used to measure the richness of words in each sample instruction, i.e., the textual lexical diversity difficulty assessment index. Specifically, the textual lexical diversity difficulty assessment index for each sample instruction can be defined as: the average length of the sequential token list in the sample instruction, under the condition of satisfying a given type-token ratio threshold. The smaller the textual lexical diversity difficulty assessment index for a sample instruction, the worse the textual lexical diversity is considered, and the easier the sample instruction is considered; conversely, the larger the textual lexical diversity difficulty assessment index for a sample instruction, the better the textual lexical diversity is considered, and the more difficult the sample instruction is considered.

[0091] In the dimension of data loss difficulty assessment, the loss predicted by the initial large language model (i.e., the large language model before instruction tuning) on ​​each sample instruction can be used as the data loss difficulty index for the corresponding sample instruction; whereby the data loss difficulty index can be expressed as: ;in, Used to indicate the length of the output text sequence Model parameters used to represent the initial large language model. This is used to output the t-th token in the text sequence. The data loss difficulty metric measures the difficulty of the initial large language model in solving the sample instruction. The larger the data loss difficulty metric, the greater the prediction bias of the initial large language model on the sample instruction, and the more difficult the sample is for the initial large language model. Conversely, the smaller the data loss difficulty metric, the smaller the prediction bias of the initial large language model on the sample instruction, and the easier the sample is for the initial large language model.

[0092] In the dimension of perceived competence assessment, a scoring model can be used. Output a capability-aware difficulty index for each sample instruction. (Scoring model) It can receive the data features and model state features corresponding to each sample instruction, process these features, and output the capability-aware difficulty index corresponding to that sample instruction. The data features corresponding to the sample instruction are the data features extracted from the initial large language model (without instruction tuning) for that sample instruction. Simultaneously, the model state features corresponding to this sample instruction can be obtained by inputting the trainable embedding matrix initialized by Kaiming as the initial semantic fragment into the large language model that has not yet been updated at the t-th training step. Among them, the scoring model R incorporates data features. With model state features To create hidden features, the parts are spliced ​​together. Then, a multi-layer perceptron (MLP) is used to map the hidden features into a capability-aware difficulty assessment metric. In other words, the sample instruction sequence under the capability-aware assessment dimension changes dynamically as the large language model is gradually updated.

[0093] As a specific example, the training process for the rating model R can be as follows: First, the training dataset... Randomly shuffle and divide equally Data subsets Then, in the scoring model The In the training round, data pairs were selected. Using a subset of data Fine-tuning of the large language model in round i-1: , where n represents the total number of data subsets corresponding to the training dataset. That is, in the 0th round of training the scoring model, the parameters of the large language model corresponding to that round are the parameters of the initial large language model. In the training round, the data subset The samples in the model and the model parameters of the current large language model Form training instances and assign them a difficulty label of 0; data subset The samples in the dataset are assigned a difficulty label of 1. For the input samples... Using the initial large language model parameters Extracting data features Simultaneously, a trainable embedding matrix initialized by Kaiming is used as the initial semantic fragment and input into the current large language model. The model state features are obtained from the process. Data features With model state features By splicing the pieces together, hidden features are formed. Then, a multilayer perceptron (MLP) is used to map the hidden features into a capability-aware difficulty evaluation index. Finally, the predicted capability-aware difficulty index and the constructed labels are used to calculate the cross-entropy loss value, which can be specifically expressed as: .

[0094] In some optional implementations, when the difficulty assessment strategy is an interaction trajectory assessment strategy, step S3012 includes: Step b1: For each sample instruction, obtain the belief evolution entropy, causal reasoning topological complexity, and trajectory flow alignment deviation corresponding to the sample instruction.

[0095] Step b2 involves weighted fusion of belief evolution entropy, causal reasoning topological complexity, and trajectory popular alignment deviation to obtain the interaction trajectory difficulty assessment index corresponding to the sample instruction.

[0096] For a line containing Sample instructions for the trajectory of an agent with an interaction step size. ,in, Used to represent the agent's local environmental observation at step t. Used to represent the agent's internal reasoning during the t-th interaction. This is used to represent the action performed by the agent at step t of the interaction. Specifically, it represents the difficulty index of the interaction trajectory. It can be jointly constructed from three parts: belief evolution entropy, causal reasoning topological complexity, and trajectory flow alignment bias, which can be specifically expressed as: ; Among them, belief evolution entropy can be used to output the current action of a large language model that has not undergone instruction tuning, given historical observations and action sequences. Conditional information entropy This term measures the uncertainty noise in environmental state transitions. A higher belief evolution entropy indicates more ambiguous environmental feedback, and a greater difficulty in decision-making exploration for the agent. Causal inference topological complexity. This allows us to parse the internal inference chain of a large language model that has not undergone instruction tuning into a directed acyclic graph, i.e., a causal inference graph. Then, the algebraic connectivity and maximum cyclic rank of the causal reasoning graph are extracted to obtain the topological complexity of the causal reasoning. The more backtracking steps and the denser the conditional branches, the deeper the representation of cognitive logic. Trajectory manifold alignment deviation. It can be introduced - Order Wasserstein distance, and sample trajectory distribution using a metric model. Distribution of expert demonstration trajectories The optimal transport cost on the joint high-dimensional state-action manifold. The larger the alignment deviation of this trajectory manifold, the lower the task's fault tolerance. Small decision offsets can lead to extremely large final reward sparsity, characterizing the extreme difficulty of long-range credit allocation.

[0097] By comprehensively evaluating sample instructions using three different indicators—belief evolution entropy, causal reasoning topological complexity, and trajectory manifold alignment deviation—the true difficulty of the interaction trajectory can be comprehensively and objectively characterized. By weighted fusion, the interaction trajectory difficulty evaluation indicators corresponding to each sample instruction are obtained. This not only preserves the effective information of each indicator but also normalizes the difficulty of the sample instructions. Subsequently, the interaction trajectory difficulty evaluation indicators can be used to sort the sample instructions more accurately, further ensuring that the sample instruction sequence corresponding to the obtained interaction trajectory evaluation strategy is more accurate.

[0098] Step S3013: Sort all sample instructions from easy to difficult according to the difficulty assessment index to obtain the sample instruction sequence under the assessment dimension; where each assessment dimension corresponds to a difficulty assessment strategy.

[0099] Within the same evaluation dimension, the difficulty evaluation indicators corresponding to each sample instruction can be used to sort all sample instructions from easy to difficult, thus obtaining the sample instruction sequence for that evaluation dimension. Furthermore, as mentioned earlier, each evaluation dimension corresponds to a difficulty evaluation strategy, and a sample instruction sequence can be obtained through a single difficulty evaluation strategy.

[0100] Step S3014: Determine the instruction selection range corresponding to different training steps. Please refer to the previous text for details, which will not be repeated here.

[0101] Step S302: Obtain the current model parameters of the large language model in the current training step, and determine the state feature vector corresponding to the current model parameters. For details, please refer to [link to relevant documentation]. Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0102] Step S303: Using the state feature vector, determine the sampling weights corresponding to each sample instruction sequence in the current training step. For details, please refer to [link to relevant documentation]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0103] Step S304: Using the sampling weights, extract training sample instructions from each sample instruction sequence within the instruction selection range corresponding to the current training step. For details, please refer to [link to details]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0104] Step S305: Adjust the current model parameters using all training sample instructions to obtain the updated model parameters. Repeat this process until all training steps are completed, resulting in the target large language model. This target large language model is used to perform question-answering tasks. For details, please refer to [link to details]. Figure 1 Step S105 of the illustrated embodiment will not be described again here.

[0105] In some alternative implementations, the method further includes: Step c1: After obtaining the current model update parameters, and before determining the state feature vector corresponding to the next training step, the current model update parameters are used to re-evaluate the difficulty of the sample instructions that did not participate in the training in the target sample instruction sequence, so as to obtain a new sample instruction sequence. The new sample instruction sequence is used to extract the training sample instructions corresponding to the next training step. The target sample instruction sequence is the sample instruction sequence obtained by difficulty evaluation based on the capability-aware difficulty evaluation strategy.

[0106] This can be achieved by using a large language model with the current model update parameters (i.e., the large language model obtained after the current training step update) to re-evaluate the difficulty of sample instructions that have not participated in training in the target sample instruction sequence. This can ensure that the sorting order of each sample instruction that has not yet participated in training in the new sample instruction sequence can better adapt to the current capability state of the large language model, and further improve the instruction tuning efficiency and instruction tuning effect of the large language model.

[0107] The instruction tuning method for large language models provided in this embodiment sets different difficulty assessment strategies for different evaluation dimensions, and then performs difficulty assessment on each sample instruction in the instruction dataset according to the difficulty assessment strategy under the corresponding evaluation dimension. This can accurately assess the difficulty of each sample instruction, and then form an orderly and progressive training logic based on the difficulty assessment index corresponding to each sample instruction, which can further improve the rationality of instruction tuning for large language models.

[0108] This embodiment provides a question-and-answer method that can be used in electronic devices such as servers or computers. The method includes the following steps: Step d1: Receive the text of the question to be answered.

[0109] After obtaining the final target large language model using the instruction tuning method described above, the target large language model can be deployed on electronic devices such as servers or computers. Simultaneously, the target large language model can also be configured with a corresponding interactive interface. In this way, users can send unanswered question texts (i.e., interactive instructions) to the target large language model through the interactive interface, and the target large language model can receive the unanswered question texts.

[0110] Step d2: Use the target large language model to process the text of the question to be answered and obtain the target answer.

[0111] After receiving the text of the question to be answered sent by the user, the target large language model can perform text cleaning, word segmentation, and semantic feature parsing on the text of the question to be answered, accurately extracting the core semantic information, query intent, key constraints, and other question features from the text of the question to be answered. Then, combined with the existing prior knowledge and task reasoning ability of the target large language model, it performs logical deduction, content association, and answer organization optimization on the parsed question features, adaptively matches the optimal response logic and completes text generation, thereby outputting the target answer that is semantically complete, logically coherent, and fits the user's query needs.

[0112] The question-answering method provided in this embodiment relies on a target large language model with optimized instructions to process the text of the question to be answered. The target large language model can efficiently parse the semantics of the question and output the target answer, thus ensuring the real-time nature of the question-answering interaction and the accuracy and logic of the response content.

[0113] This embodiment also provides an instruction tuning apparatus for a large language model, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0114] This embodiment provides an instruction tuning device for a large language model, such as... Figure 4 As shown, it includes: The first acquisition module 401 is used to acquire the instruction dataset and determine the sample instruction sequences of the instruction dataset under different evaluation dimensions and the instruction selection range corresponding to each training step.

[0115] The second acquisition module 402 is used to acquire the current model parameters of the large language model in the current training step and determine the state feature vector corresponding to the current model parameters.

[0116] The weight determination module 403 is used to determine the sampling weights corresponding to each sample instruction sequence in the current training step using the state feature vector.

[0117] The sample extraction module 404 is used to extract training sample instructions from each sample instruction sequence within the instruction selection range corresponding to the current training step by utilizing the sampling weights.

[0118] The parameter adjustment module 405 is used to adjust the current model parameters using all training sample instructions to obtain the current model update parameters, until the training process of all training steps is completed in a loop, and the target large language model is obtained. The target large language model is used to perform question answering tasks.

[0119] In some optional implementations, the current model parameters include the sub-model parameters corresponding to the last network layer of the large language model; the second acquisition module 402 is used to extract features from the sub-model parameters using a pre-trained state extraction model to obtain a state feature vector; wherein, the state extraction model is configured to: extract features from the sub-model parameters to obtain the gradient feature vector, cross-entropy distribution vector, and memory retention rate vector corresponding to the current training step; and concatenate the gradient feature vector, cross-entropy distribution vector, and memory retention rate vector to obtain the state feature vector.

[0120] In some optional implementations, the apparatus further includes: a model update module, used to update the model parameters of the state extraction model by utilizing the performance difference of the large language model before and after the update in the current training step, to obtain a new state extraction model; wherein the new state extraction model is used to extract the state feature vector corresponding to the updated parameters of the current model in the next training step.

[0121] In some optional implementations, the model update module is also used to construct a state correction value using the loss value of the large language model before and after the current training step; use the state correction value as a supervision signal and construct confidence constraints using the neural confidence upper bound to update the model parameters of the state extraction model and obtain a new state extraction model.

[0122] In some optional implementations, the weight determination module 403 is used to input the state feature vector into a pre-trained weight determination model, which performs Dirichlet sampling on the state feature vector to obtain the sampling weights corresponding to each sample instruction sequence.

[0123] In some optional implementations, the first acquisition module 401 is used to perform difficulty assessment on each sample instruction in the instruction dataset according to the difficulty assessment strategy under the assessment dimension for each assessment dimension, and obtain the difficulty assessment index corresponding to each sample instruction; sort all the sample instructions from easy to difficult according to the difficulty assessment index to obtain the sample instruction sequence under the assessment dimension; wherein, one assessment dimension corresponds to one difficulty assessment strategy.

[0124] In some optional implementations, the first acquisition module 401 is used to acquire, for each sample instruction, the belief evolution entropy, causal reasoning topological complexity, and trajectory flow alignment deviation corresponding to the sample instruction; and to perform weighted fusion of the belief evolution entropy, causal reasoning topological complexity, and trajectory flow alignment deviation to obtain the interaction trajectory difficulty evaluation index corresponding to the sample instruction.

[0125] In some alternative embodiments, the device further includes: The sample order update module is used to re-evaluate the difficulty of sample instructions that have not participated in training in the target sample instruction sequence after obtaining the current model update parameters and before determining the state feature vector corresponding to the next training step, so as to obtain a new sample instruction sequence. The new sample instruction sequence is used to extract the training sample instructions corresponding to the next training step. The target sample instruction sequence is the sample instruction sequence obtained by difficulty evaluation based on the capability-aware difficulty evaluation strategy.

[0126] The instruction tuning apparatus for large language models provided in this application can execute the instruction tuning method for large language models provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the execution method. Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0127] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0128] The following is a detailed reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0129] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0130] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 509, or installed from memory 508, or installed from ROM 502. When the computer program is executed by processor 501, it performs the functions defined in the instruction tuning method for a large language model according to embodiments of this application.

[0131] Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0132] This application also provides a computer-readable storage medium. The methods described above according to this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the instruction tuning method for the large language model shown in the above embodiments is implemented.

[0133] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0134] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method for instruction tuning in a large language model, characterized in that, The method includes: Obtain the instruction dataset and determine the sample instruction sequences of the instruction dataset under different evaluation dimensions and the instruction selection range corresponding to each training step; Obtain the current model parameters of the large language model in the current training step, and determine the state feature vector corresponding to the current model parameters; Using the state feature vector, determine the sampling weights corresponding to each sample instruction sequence in the current training step; Using the sampling weights, training sample instructions are extracted from each of the sample instruction sequences within the instruction selection range corresponding to the current training step; The current model parameters are adjusted using all the training sample instructions to obtain the current model update parameters. This process is repeated until all training steps are completed to obtain the target large language model, which is used to perform question answering tasks.

2. The method according to claim 1, characterized in that, The current model parameters include the sub-model parameters corresponding to the last network layer of the large language model; determining the state feature vector corresponding to the current model parameters includes: The sub-model parameters are feature extracted using a pre-trained state extraction model to obtain the state feature vector; The state extraction model is configured to: extract features from the sub-model parameters to obtain the gradient feature vector, cross-entropy distribution vector, and memory retention rate vector corresponding to the current training step; and concatenate the gradient feature vector, the cross-entropy distribution vector, and the memory retention rate vector to obtain the state feature vector.

3. The method according to claim 2, characterized in that, The method further includes: By utilizing the performance difference of the large language model before and after the update in the current training step, the model parameters of the state extraction model are updated to obtain a new state extraction model; The new state extraction model is used to extract the state feature vector corresponding to the current model update parameters in the next training step.

4. The method according to claim 3, characterized in that, The step of updating the state extraction model by utilizing the performance difference of the large language model before and after the update in the current training step to obtain a new state extraction model includes: Using the loss values ​​of the large language model before and after the update in the current training step, a state correction value is constructed; The state correction amount is used as a supervision signal, and confidence constraints are constructed using the neural confidence upper limit. The model parameters of the state extraction model are updated to obtain the new state extraction model.

5. The method according to claim 1, characterized in that, The step of determining the sampling weights corresponding to each sample instruction sequence in the current training step using the state feature vector includes: The state feature vector is input into a pre-trained weight determination model, which performs Dirichlet sampling on the state feature vector to obtain the sampling weights corresponding to each sample instruction sequence.

6. The method according to claim 1, characterized in that, Determining the sample instruction sequences of the instruction dataset under different evaluation dimensions includes: For each evaluation dimension, the difficulty of each sample instruction in the instruction dataset is evaluated according to the difficulty evaluation strategy under the evaluation dimension, so as to obtain the difficulty evaluation index corresponding to each sample instruction. All the sample instructions are sorted from easy to difficult according to the difficulty assessment index to obtain the sample instruction sequence under the assessment dimension, wherein each assessment dimension corresponds to a difficulty assessment strategy.

7. The method according to claim 6, characterized in that, When the difficulty assessment strategy is an interaction trajectory assessment strategy, the step of assessing the difficulty of each sample instruction in the instruction dataset according to the difficulty assessment strategy under the assessment dimension to obtain the difficulty assessment index corresponding to each sample instruction includes: For each of the sample instructions, obtain the belief evolution entropy, causal reasoning topological complexity, and trajectory flow alignment deviation corresponding to the sample instruction; The belief evolution entropy, the causal reasoning topological complexity, and the trajectory popular alignment deviation are weighted and fused to obtain the interaction trajectory difficulty evaluation index corresponding to the sample instruction.

8. The method according to any one of claims 1 to 7, characterized in that, After obtaining the current model update parameters and before determining the state feature vector corresponding to the next training step, the method further includes: Using the current model update parameters, the difficulty of sample instructions that did not participate in training in the target sample instruction sequence is re-evaluated to obtain a new sample instruction sequence, which is used to extract the training sample instructions corresponding to the next training step. The target sample instruction sequence is a sample instruction sequence obtained by difficulty assessment based on a capability-aware difficulty assessment strategy.

9. A question-and-answer method, characterized in that, The method includes: Receive the text of the question to be answered; Using the target large language model described in any one of claims 1 to 8, the text of the question to be answered is processed to obtain the target answer.

10. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the instruction tuning method for a large language model as described in any one of claims 1 to 8, or the question-answering method as described in claim 9.

Images