Question answering methods and devices for fine-tuning large models for ensemble learning

By combining ensemble learning and reinforcement learning, an ensemble reward model is constructed, which solves the problems of insufficient knowledge and illusion phenomena in LLMs in specific domains, and achieves accurate alignment with human preferences and high performance in multimodal environments.

CN118761459BActive Publication Date: 2026-11-17BEIJING UNIV OF TECH
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202410740134.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-07
Publication Date
2026-11-17
Estimated Expiration
2044-06-07

AI Technical Summary

Technical Problem

LLMs have insufficient knowledge coverage in specific domains, may produce illusions when generating text, and traditional fine-tuning methods rely on a large amount of labeled data which is difficult to obtain, especially in multimodal environments where they are costly and difficult to adapt to the diversity of human preferences and needs.

Method used

An ensemble learning strategy is used to train a reward model, which is then constructed. Reinforcement learning techniques are combined to fine-tune LLMs, and human feedback is used to optimize model behavior. The model is continuously adjusted to adapt to human preferences through multimodal datasets and feedback databases.

Benefits of technology

It improves the performance and applicability of LLMs in specific domains, reduces hallucinations, enhances the model's sensitivity and accuracy to human feedback, adapts to changes in multimodal environments, and improves user experience and satisfaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118761459B_ABST
    Figure CN118761459B_ABST
Patent Text Reader

Abstract

This invention provides a question-answering method and apparatus for fine-tuning large language models using ensemble learning. The method includes: pre-training an LLM (Large Language Model) to obtain a baseline LLM; training a reward model based on an ensemble learning strategy to obtain an ensemble reward model; fine-tuning the baseline LLM based on the reward score output by the ensemble reward model using a reinforcement learning strategy to obtain a target LLM; and inputting a user's question into the target LLM to obtain the answer to the question. This invention achieves precise alignment with human preferences and improves the performance of LLMs in specific domains by integrating a reward model based on an ensemble learning strategy and reinforcement learning techniques based on human feedback.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a question-answering method and apparatus for fine-tuning large models for ensemble learning. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs), such as GPT-4, have made significant progress in the field of Natural Language Processing (NLP). These models, by learning from large amounts of text data, are able to generate coherent and logically sound text and demonstrate excellent performance in various language tasks. However, despite their outstanding general performance, LLMs still face some challenges in their application in specific domains.

[0003] First, LLMs are typically trained on large-scale, general-purpose datasets, which can lead to insufficient knowledge coverage in specific domains. For example, in industrial settings, models may need to understand specific industry terminology, operating procedures, and safety regulations, which may not be common in general-purpose datasets. Furthermore, LLMs may exhibit a so-called "illusion" phenomenon when generating text, producing information that is inconsistent with or inaccurate with reality, which is unacceptable in domains requiring high accuracy.

[0004] To address these issues and improve the applicability and practicality of LLMs in specific domains, targeted fine-tuning is necessary. The fine-tuning process aims to enable the model to generate text that meets the needs of a specific domain while reducing inaccurate or irrelevant output. However, traditional fine-tuning methods often rely on large amounts of labeled data, which may be difficult to obtain in some domains, especially in multimodal environments where the annotation cost for image and video content is higher.

[0005] Furthermore, human preferences and needs are diverse and change over time and with environmental changes. Therefore, a method is needed that can continuously adapt to and accurately align with human preferences to improve the user experience and satisfaction of the model in practical applications. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, this invention provides a question-answering method and apparatus for fine-tuning large models in ensemble learning.

[0007] This invention provides a question-answering method for fine-tuning large models oriented towards ensemble learning, comprising:

[0008] A baseline LLM model is obtained by pre-training the LLM model.

[0009] An ensemble reward model is obtained by training the reward model using an ensemble learning strategy.

[0010] The target LLM model is obtained by fine-tuning the baseline LLM model based on the reward score output by the integrated reward model using a reinforcement learning strategy.

[0011] Input the user's question into the target LLMs model and obtain the answer to the question.

[0012] According to the present invention, a question-answering method for fine-tuning a large model based on ensemble learning is provided, wherein training a reward model based on an ensemble learning strategy to obtain an ensemble reward model includes:

[0013] The first data is input into the benchmark LLMs model multiple times to obtain the answers to the first data output by the benchmark LLMs model. A first question-answer pair is generated based on the first data and the answers to the first data.

[0014] Obtain the human revision results for each answer in the first data, and generate a second question-answer pair from the first data and the revision results of each answer;

[0015] The first question and answer pair is sorted according to its quality, and the second question and answer pair is sorted according to its quality. The second question and answer pair is sorted before the first question and answer pair. The first question and answer and the second question and answer are then stored in the feedback database according to the sorting results.

[0016] The reward model is trained using the feedback database based on the ensemble learning strategy to obtain the ensemble reward model.

[0017] According to the question-answering method for large model fine-tuning based on ensemble learning provided by the present invention, the step of training the reward model using the feedback database based on the ensemble learning strategy to obtain an ensemble reward model includes:

[0018] The data in the feedback database is divided into a training set and a validation set;

[0019] Multiple reward models are trained using different subsets of data from the training set and / or different hyperparameters;

[0020] The trained reward models are validated using the validation set to obtain the performance evaluation function values ​​of each reward model on the validation set.

[0021] The multiple reward models are fused based on the performance index evaluation function values ​​of each reward model on the validation set to obtain an integrated reward model.

[0022] According to the question-answering method for large model fine-tuning ensemble learning provided by the present invention, the weights of the ensemble reward model are obtained by the following formula:

[0023]

[0024] in, The weights of the integrated reward model, For the i-th reward model RM i The weights, N is the number of reward models, P f (RM i Let be the performance evaluation function value of the i-th reward model on the validation set. Let be the performance evaluation function value of the i-th reward model on the first question-answer pair in the validation set. Let α be the performance evaluation function value of the i-th reward model on the second question-answer pair in the validation set, and let α be a preset adjustment coefficient. Greater than the α is positive when α is positive, otherwise it is negative.

[0025] According to the present invention, a question-answering method for fine-tuning large models oriented towards ensemble learning includes training multiple reward models using different subsets of data and / or different hyperparameters in the training set, comprising:

[0026] Input different subsets of data from the training set and / or different hyperparameters into each reward model;

[0027] Based on whether the reward scores of the training set output by each reward model are Markovian, determine the preference probability of each reward model;

[0028] The loss function of each reward model is determined based on the cross-entropy loss between the preference probability of each reward model on the training set and the preset preference label.

[0029] The weights of each reward model are adjusted according to the loss function of each reward model.

[0030] According to the present invention, a question answering method for fine-tuning a large model for ensemble learning is provided, wherein the reward model is a multilayer perceptron (MLP), a convolutional neural network (CNN), or a Transformer model.

[0031] According to the present invention, a question-answering method for fine-tuning large models oriented to ensemble learning, wherein the target LLM model is obtained by fine-tuning the baseline LLM model based on the reward score output by the ensemble reward model according to the reinforcement learning strategy, the method includes:

[0032] The second data is input into the benchmark LLMs model and the target LLMs model respectively to obtain the answer of the second data output by the benchmark LLMs model and the target LLMs model;

[0033] The answers to the second data output by the benchmark LLMs model and the target LLMs model are respectively input into the policy model to obtain the penalty term between the outputs of the policy model;

[0034] The answer to the second data output by the target LLMs model is input into the ensemble reward model to obtain the reward score output by the ensemble reward model;

[0035] The loss function of the target LLMs model is determined based on the penalty term and the reward score.

[0036] The weights of the target LLMs model are adjusted according to the loss function of the target LLMs model.

[0037] According to the question-answering method for large model fine-tuning ensemble learning provided by the present invention, after inputting the user's question into the target LLMs model and obtaining the answer to the question, the method further includes:

[0038] Obtain user evaluation and modification of the answers to the questions;

[0039] The evaluation and modifications are used as a new dataset to fine-tune the target LLMs model again.

[0040] This invention also provides a question-answering device for fine-tuning large models in ensemble learning, comprising:

[0041] The first pre-training module is used to pre-train the LLMs model to obtain the baseline LLMs model;

[0042] The second pre-training module is used to train the reward model based on the ensemble learning strategy to obtain the ensemble reward model;

[0043] The fine-tuning module is used to fine-tune the baseline LLMs model based on the reward score output by the ensemble reward model according to the reinforcement learning strategy, so as to obtain the target LLMs model.

[0044] The generation module is used to input the user's question into the target LLMs model and obtain the answer to the question.

[0045] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a question-answering method for fine-tuning a large model for ensemble learning as described above.

[0046] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a question-answering method for large model fine-tuning ensemble learning as described above.

[0047] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a question-answering method for large model fine-tuning oriented ensemble learning as described above.

[0048] This invention provides a question-answering method and apparatus for fine-tuning large models using ensemble learning. It pre-trains LLMs on a large dataset to learn rich linguistic and visual representations. By applying an ensemble learning reward model (ELRM) strategy to fine-tune multiple reward models, a robust ensemble reward model is constructed, enabling effective evaluation of multimodal inputs. Reinforcement learning from human feedback (RLHF) is used to fine-tune the pre-trained LLMs, optimizing their behavior and reducing hallucinations, while improving the model's sensitivity and accuracy to human feedback, thus enhancing the performance and reliability of LLMs in multimodal environments. Finally, by fusing the reward model based on ensemble learning and reinforcement learning based on human feedback, precise alignment with human preferences is achieved, improving the performance of LLMs in specific domains. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is one of the flowcharts of the question-answering method under large model fine-tuning for ensemble learning provided by the present invention;

[0051] Figure 2 This is the second schematic diagram of the question-answering method under large model fine-tuning for ensemble learning provided by the present invention;

[0052] Figure 3 This is one of the flowcharts illustrating the data acquisition and preprocessing process in the question-answering method for large model fine-tuning oriented towards ensemble learning provided by this invention;

[0053] Figure 4 This is the second schematic diagram of the data acquisition and preprocessing process in the question-answering method for large model fine-tuning oriented to ensemble learning provided by this invention;

[0054] Figure 5 This is a schematic diagram of the reward model ensemble learning process in the question answering method for large model fine-tuning ensemble learning provided by this invention;

[0055] Figure 6 This is a flowchart illustrating the training process of the reward model in the question-answering method for large-scale model fine-tuning oriented to ensemble learning provided by this invention.

[0056] Figure 7 This is a schematic diagram of the reward model structure in the question-answering method for large-scale model fine-tuning oriented to ensemble learning provided by the present invention;

[0057] Figure 8 This is a schematic diagram of the RLHF fine-tuning process in the question-answering method for large model fine-tuning oriented to ensemble learning provided by this invention;

[0058] Figure 9 This is a schematic diagram of the terminal interactive feedback process in the question-answering method for fine-tuning large models for ensemble learning provided by this invention.

[0059] Figure 10 This is a schematic diagram of the question-answering device for fine-tuning large models for ensemble learning provided by the present invention.

[0060] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0062] The following is combined Figure 1 This invention describes a question-answering method for fine-tuning large models in ensemble learning, comprising:

[0063] Step 101: Pre-train the LLMs model to obtain the baseline LLMs model;

[0064] LLMs models can be pre-trained on large-scale, general datasets to learn rich linguistic and visual representations.

[0065] The dataset can be multimodal, including text, images, audio, and video. The dataset is preprocessed to suit the model's input requirements.

[0066] Multimodal data is input into an LLM model to perform a question-answering generation task. The parameters of the LLM model are adjusted based on the error between the generated and actual answers to obtain a baseline LLM model. The goal is to enable the LLM model to generate correct text output based on given image and text prompts.

[0067] Step 102: Train the reward model based on the ensemble learning strategy to obtain the ensemble reward model;

[0068] In multimodal large model fine-tuning methods, training the reward model is a crucial step, involving the application of ensemble learning strategies and the construction of the reward model. This step aims to enhance the performance of LLMs on multimodal tasks by integrating the predictions of multiple reward models.

[0069] Multiple reward models can be obtained by training the same reward model with different hyperparameters. Similarly, multiple reward models can be obtained by training the same reward model with different datasets. The parameters of multiple reward models can be weighted and combined to form an ensemble reward model.

[0070] Step 103: Based on the reinforcement learning strategy, fine-tune the baseline LLMs model according to the reward score output by the ensemble reward model to obtain the target LLMs model;

[0071] By using human feedback-based reinforcement learning (RLHF) techniques and reward scores from an ensemble reward model, benchmark LLMs models were fine-tuned to optimize their behavior and reduce hallucinations, while improving the model’s sensitivity and accuracy to human feedback.

[0072] Step 104: Input the user's question into the target LLMs model to obtain the answer to the question.

[0073] User questions can be multimodal data, including images, text prompts, and other possible modalities such as video. The user's question is input as a prompt into the target LLM model, which then generates the answer.

[0074] Human preferences and needs are diverse and change over time and with environmental changes. Therefore, a method is needed that can continuously adapt to and accurately align with human preferences to improve the user experience and satisfaction of models in practical applications.

[0075] In practical applications, feedback can be collected through user interaction to further fine-tune the target LLM model to adapt to constantly changing user experiences and preferences. A complete flowchart is shown below. Figure 2 As shown.

[0076] This embodiment uses a large-scale dataset to pre-train LLMs to learn rich linguistic and visual representations; it then fine-tunes multiple reward models by applying an ensemble learning strategy to construct a robust ensemble reward model, enabling effective evaluation of multimodal inputs; it utilizes RLHF technology to fine-tune the pre-trained LLMs to optimize their behavior and reduce hallucinations, while improving the model's sensitivity and accuracy to human feedback, thus enhancing the performance and reliability of LLMs in multimodal environments; finally, by fusing the reward model based on the ensemble learning strategy and reinforcement learning techniques based on human feedback, it achieves precise alignment with human preferences, improving the performance of LLMs in specific domains.

[0077] Based on the above embodiments, the training of the reward model based on the ensemble learning strategy to obtain the ensemble reward model in this embodiment includes:

[0078] The first data is input into the benchmark LLMs model multiple times to obtain the answers to the first data output by the benchmark LLMs model. A first question-answer pair is generated based on the first data and the answers to the first data.

[0079] Obtain the human revision results for each answer in the first data, and generate a second question-answer pair from the first data and the revision results of each answer;

[0080] The first question and answer pair is sorted according to its quality, and the second question and answer pair is sorted according to its quality. The second question and answer pair is sorted before the first question and answer pair. The first question and answer and the second question and answer are then stored in the feedback database according to the sorting results.

[0081] The reward model is trained using the feedback database based on the ensemble learning strategy to obtain the ensemble reward model.

[0082] The first data can be a general multimodal data resource, including images, text prompts, and other possible modalities such as video.

[0083] like Figure 3As shown, in the multimodal large language models (MM-LLMs) fine-tuning method provided in this embodiment, data acquisition and preprocessing are the foundation and key steps of the entire process, directly affecting the effect of subsequent model training and the performance of the final model. The specific steps include:

[0084] a. Multimodal Input Processing: This involves collecting and integrating multimodal data resources to enhance the model's generative capabilities, enabling it to generate corresponding text output based on the input multimodal information. Specifically, this includes the following aspects:

[0085] Data collection: Collect image and text data from various sources to ensure data diversity and coverage, encompassing different scenarios and themes.

[0086] Data integration: The collected images are paired with relevant text prompts to form an input dataset. For example, an image showing a field in spring can be paired with a text prompt describing the beauty of spring.

[0087] Question-answer pair generation: The integrated data is input n times into MM-LLMs to perform the question-answer generation task. The model generates a series of question-answer pairs based on the input image and text prompts, labeled as M1 to Mn, forming the generated dataset.

[0088] b. Manual Quality Control: The generated question-and-answer pairs may contain inaccurate or irrelevant information. Therefore, quality assessment and revision by domain experts are necessary. This includes the following aspects:

[0089] Quality assessment: Domain experts are invited to evaluate the generated question-and-answer pairs and select them based on criteria such as accuracy, relevance, and fluency.

[0090] Revision and optimization: Revision of non-standard question-and-answer pairs, optimization of language expression, and ensuring the accuracy and relevance of information.

[0091] Creating an artificial dataset: Question-answer pairs that have been reviewed and revised by experts are relabeled as H1 to Hn, forming a high-quality artificial dataset.

[0092] c. Dataset Ranking and Optimization: To establish a dataset prioritization system centered on human evaluation, generated datasets and manually generated datasets are prioritized. This includes the following aspects:

[0093] Ranking Mechanism: The initially generated dataset and the artificial dataset are prioritized based on human preferences, ensuring that the artificial dataset takes precedence over the generated dataset in the ranking process; that is, the ranking positions of H1 to Hn are higher than those of M1 to Mn. This ranking strategy aims to establish a dataset priority based on human evaluation, but it does not involve assigning specific scores to individual question-answer pairs; it only ensures that the overall order of the feedback database reflects human preferences and quality standards.

[0094] Optimize the dataset: By sorting, the overall order of the dataset reflects human preferences and quality standards, laying a solid data foundation for subsequent model training and fine-tuning.

[0095] from Figure 4 The flowchart clearly shows the entire data process, from collection and integration to question-and-answer pair generation, manual quality control, and dataset sorting. Each step has a clear goal and operation to ensure the quality and applicability of the dataset. The flowchart illustrates the various stages of data acquisition and preprocessing from left to right, with corresponding operations and outputs for each stage. Data begins with multimodal input processing on the left, passes through manual quality control in the middle, and finally reaches dataset sorting and optimization on the right.

[0096] In this embodiment, data acquisition and preprocessing play a crucial role in the MM-LLMs fine-tuning method. They not only improve the practicality and quality of the dataset but also provide important guarantees for the training efficiency and output accuracy of the model, thereby ensuring the high performance of the model in practical applications.

[0097] Based on the above embodiments, the step of training the reward model using the feedback database based on the ensemble learning strategy to obtain the ensemble reward model in this embodiment includes:

[0098] The data in the feedback database is divided into a training set and a validation set;

[0099] Multiple reward models are trained using different subsets of data from the training set and / or different hyperparameters;

[0100] The trained reward models are validated using the validation set to obtain the performance evaluation function values ​​of each reward model on the validation set.

[0101] The multiple reward models are fused based on the performance index evaluation function values ​​of each reward model on the validation set to obtain an integrated reward model.

[0102] The feedback database includes generated datasets and artificial datasets. The artificial datasets further include training and validation sets, with the validation set used to evaluate the performance of multiple reward models.

[0103] The core idea of ​​ensemble learning of reward models is to leverage the diversity of multiple models to reduce reliance on a single model, thereby improving adaptability to distribution changes and robustness to label noise.

[0104] like Figure 5 As shown, multiple reward models (RMs) are pre-trained first, using the same reward model architecture. The diversity between models is increased by pre-training on different subsets of data for the same task or by using different hyperparameter settings, such as learning rate and regularization strength.

[0105] During the fusion phase of the reward models, each fine-tuned reward model independently provides reward predictions for the input samples, and the predicted rewards are weighted and averaged to form the final reward score.

[0106] Train N RMs, denoted as {RM1, RM2, ..., RM3}. N For a given input sample, each RM i A reward score will be output. The weighted average reward is calculated as follows:

[0107]

[0108] in, Given a state-action pair (s) t ,a t The average reward of the model, where N is the number of reward models, and P is the average reward. f (RM i ) represents the performance evaluation function value of the i-th reward model on the validation set.

[0109] The weights are assigned based on the performance of each reward model on the validation set, and are calculated using the softmax function according to performance metrics to ensure that the model ensemble reflects the best performance.

[0110] This embodiment obtains an integrated reward model by effectively training and integrating the reward model, thereby improving its performance and robustness in multimodal tasks. This not only helps to improve the generation quality of LLMs, but also ensures that the generated content better meets human preferences and needs.

[0111] Based on the above embodiments, the weights of the integrated reward model in this embodiment are obtained using the following formula:

[0112]

[0113] in, The weights of the integrated reward model, For the i-th reward model RM iThe weights, N is the number of reward models, P f (RM i Let be the performance evaluation function value of the i-th reward model on the validation set. Let be the performance evaluation function value of the i-th reward model on the first question-answer pair in the validation set. Let α be the performance evaluation function value of the i-th reward model on the second question-answer pair in the validation set, and let α be a preset adjustment coefficient. Greater than the α is positive when α is positive, otherwise it is negative.

[0114] Based on the above embodiments, such as Figure 6 As shown in this embodiment, training multiple reward models using different subsets of data in the training set and / or different hyperparameters includes:

[0115] Input different subsets of data from the training set and / or different hyperparameters into each reward model;

[0116] Based on whether the reward scores of the training set output by each reward model are Markovian, determine the preference probability of each reward model;

[0117] The loss function of each reward model is determined based on the cross-entropy loss between the preference probability of each reward model on the training set and the preset preference label.

[0118] The weights of each reward model are adjusted according to the loss function of each reward model.

[0119] The specific steps for training a reward model include the following:

[0120] Initialize parameters and select a reward model architecture, for example... This is a parameterized function used to determine the state-action pair (s) in the trajectory σ. t ,a t Predict the reward.

[0121] In calculating the preference probability of the reward model, the distinction is made based on whether the reward depends on visited states, i.e., whether it is a Markov model. The formula for calculating the preference probability based on the Markov model is as follows:

[0122]

[0123] Where P[σ1>σ0;ψ] represents the probability that trajectory σ1 is preferred over trajectory σ0 given parameter ψ. It is a parameterized reward function, given in state s t and action a tThe expected reward is given by parameter ψ. t represents the time step. ψ is a parameter of the reward function, adjusted using training data to maximize the accuracy of predictions based on human preferences.

[0124] The preference probability in the non-Markov case is calculated using the following formula:

[0125]

[0126] For non-Markov reward functions, in order to capture time dependence, an importance weight w is introduced as a function of the entire trajectory segment, enabling the preference prediction model to allocate importance within the trajectory segment.

[0127] The loss function of the reward model is:

[0128]

[0129] Among them, L CE (ψ) is the cross-entropy loss function used to optimize the parameter ψ during training. D is a dataset containing multiple preference comparisons (σ0, σ1, y). y is a preference label indicating whether a human evaluator prefers trajectory σ1 over σ0 (y = 1) or vice versa (y = 0), or if there is no difference between the two (y = 0.5). Gradient descent or other optimization algorithms are used to minimize the loss function L. CE (ψ), thereby adjusting the parameter ψ of the reward model.

[0130] After training each reward model, the reward models trained with different parameters are integrated into one model to improve the generalization and robustness of the model.

[0131] Based on the above embodiments, the reward model in this embodiment is an MLP, CNN, or Transformer model.

[0132] Choose the appropriate reward model architecture based on the data type, including MLP, CNN, and Transformer, etc., with specific structures as follows: Figure 7 As shown. The selected reward model is then used with a parameterized function. This indicates that the state-action pair (s) in the trajectory σ is used to determine the state-action pair. t ,a t Predict the reward.

[0133] The MLP model employs a three-layer neural network structure, with each layer containing N hidden units and using the ReLU activation function. Here, N is a power of 2. To ensure the stability of reward learning, an ensemble of K reward models is used, with the output constrained within the range [-1, 1] using the tanh function. The minimum value of K is 3.

[0134] The CNN model improves upon the MLP structure by employing a three-layer CNN architecture, with each layer having a different kernel size and number of filters. Since the tanh activation function can cause gradient vanishing in visual environments, no activation function is used in the output layer.

[0135] The Transformer model employs the Preference Transformer, which includes a Causal Transformer and a Preference Attention Layer. Given a trajectory state-action segment, an embedding is first generated through a linear layer, then input into the Transformer network to generate the output embedding. The preference attention layer receives the hidden embedding of the output and generates the reward and importance weights.

[0136] Based on the above embodiments, such as Figure 8 As shown, in this embodiment, the reinforcement learning-based strategy fine-tunes the baseline LLMs model according to the reward score output by the ensemble reward model to obtain the target LLMs model, including:

[0137] The second data is input into the benchmark LLMs model and the target LLMs model respectively to obtain the answer of the second data output by the benchmark LLMs model and the target LLMs model;

[0138] The answers to the second data output by the benchmark LLMs model and the target LLMs model are respectively input into the policy model to obtain the penalty term between the outputs of the policy model;

[0139] The answer to the second data output by the target LLMs model is input into the ensemble reward model to obtain the reward score output by the ensemble reward model;

[0140] The loss function of the target LLMs model is determined based on the penalty term and the reward score.

[0141] The weights of the target LLMs model are adjusted according to the loss function of the target LLMs model.

[0142] The second set of data can be domain-specific data, such as data from an industrial production environment. Using the RLHF technique, the target LLMs are fine-tuned based on the reward scores provided by the ELRM. The loss function includes reward scores and a KL divergence penalty term to ensure that the model does not deviate too far from the initial policy while pursuing high rewards.

[0143] The loss function for fine-tuning target LLMs using reinforcement learning is shown below:

[0144]

[0145] Where, π INIT It is the initial strategy model, with basic multimodal capabilities. The strategy attempts to generate the most appropriate response for each user query. The model is optimized through reinforcement learning, and the model parameters are represented by the symbol φ. It is a dataset containing all collected images and user prompts, denoted by (I, x), where I is the image and x is the prompt. I and x together constitute the state parameter s. t . s t It is a strategy model The generated output is typically a prediction or action for a given input. In the case of a vision-language model, a t This could be a description of an image, an answer to a question, or any form of text output related to visual content. The penalty term penalizes the deviation of the policy model from the initial policy model, preventing the model from over-optimizing the reward signal and ignoring other potential problems (such as generating unrealistic content). β is a hyperparameter used to control the importance of the KL divergence.

[0146] Based on the above embodiments, this embodiment further includes, after inputting the user's question into the target LLMs model and obtaining the answer to the question:

[0147] Obtain user evaluation and modification of the answers to the questions;

[0148] The evaluation and modifications are used as a new dataset to fine-tune the target LLMs model again.

[0149] Interactive feedback from the terminal is a key element in improving the performance of LLMs after RLHF fine-tuning. Through real-time user interaction and feedback loops, the model is continuously optimized and adjusted. For example... Figure 9 As shown, the following detailed optimization steps are included:

[0150] Interactive Platform Integration and Configuration: Integrate the RLHF-tuned LLMs with an interactive platform (such as Streamlit) to ensure the system can process and display image-cue word pairs for input and output. Configure the interactive platform's user interface to support real-time input submission and output display, while providing necessary user guidance and help documentation.

[0151] Real-time display and user evaluation: Users submit N pairs of image-prompt words through the interactive platform. LLMs process these pairs in real time and generate corresponding outputs. The generated results are displayed instantly on the interactive platform, allowing users to provide preliminary evaluations, such as satisfaction ratings and selective feedback.

[0152] Human feedback: A dedicated feedback module allows users to thoroughly evaluate and modify the model's output. Users can provide specific suggestions for improvement, such as correcting errors, enhancing clarity, or improving relevance.

[0153] Iterative Dataset Updates: User evaluations and modifications are integrated into a new dataset that reflects direct user feedback and optimization suggestions on the model output. The updated dataset is then used to fine-tune the LLMs, aiming for better performance in future generation tasks.

[0154] Model performance monitoring and optimization: Continuously monitor the performance changes of LLMs and analyze the impact of user feedback on model output quality. Based on performance monitoring results and user feedback, adjust fine-tuning strategies and model parameters to achieve continuous model optimization.

[0155] During the fine-tuning of LLM models, benchmark LLMs serve as baseline models, acquiring the basic capabilities to handle industrial tasks through supervised fine-tuning (SFT). The performance level of the benchmark LLMs provides an important reference point for the continuous optimization of the target LLMs. As the performance of the target LLMs gradually improves during training, the best-performing model is selected as the new benchmark LLM. This mechanism ensures that the performance benchmark always remains synchronized with the current best results, thus maintaining its relevance and validity in the evaluation process.

[0156] At the end of each training phase, the text generated by the target LLMs will be submitted to industry experts for detailed evaluation. The experts will provide professional feedback based on the accuracy of the text content, its relevance to industrial practices, and its applicability in practical operation. This feedback will be compiled and incorporated into a feedback database, which is the core of the interactive feedback mechanism.

[0157] The feedback database not only records text samples generated by the model but also includes detailed expert evaluations and improvement suggestions. This data is used to make crucial updates to the reward model, ensuring that it more accurately maps the preferences of human experts and fully considers the specific needs of the industrial sector. In this way, each iteration of the model can more closely reflect the complexity and dynamism of the actual industrial environment, thereby providing users with more accurate, reliable, and efficient artificial intelligence support.

[0158] The large model fine-tuning method provided in this embodiment involves an ELRM method, which, combined with the RLHF method, can improve the performance and adaptability of LLMs in industrial applications. Figure 6 and Figure 9 As shown, this method enhances the multimodal understanding and generative capabilities of LLMs by integrating predictions from multiple reward models.

[0159] In the industrial sector, accurate information understanding and efficient communication are crucial. To improve the performance of LLMs in handling industry-related queries and tasks, this paper details how to apply an ELRM-based RLHF method to enhance the reliability and generalization of LLMs in industrial applications. Figure 5 and Figure 8 As shown. This method enhances the multimodal understanding and generative capabilities of LLM by integrating predictions from multiple reward models. The specific implementation steps are as follows:

[0160] Building a Feedback Database for Industrial Scenarios: In industrial applications, building an accurate and practical feedback database is crucial to ensure that the output of LLM meets the specific needs of the industrial sector. This involves developing a feedback database specifically for the industrial field, consisting of two parts: generated datasets and artificial datasets.

[0161] The generated dataset consists of text samples such as maintenance guides, safety procedure instructions, and equipment maintenance manuals automatically generated by LLMs based on industrial images and operational prompts. The manually generated dataset includes case studies, operational procedure descriptions, and troubleshooting guides carefully reviewed and annotated by experienced industrial engineers and technicians. These samples undergo rigorous quality control to ensure high accuracy and usability. The manually generated dataset is further subdivided into training and validation sets. The training set is used to train and adjust the reward model to accurately evaluate the industrial-related text generated by LLMs; the validation set is used to test and validate the evaluation capabilities of the reward model, ensuring its effectiveness and reliability in real-world industrial applications.

[0162] Building and training baseline and target LLMs: Supervised fine-tuning (SFT) of the LLMs using feedback databases from industrial scenarios further optimizes their performance to learn rich linguistic and visual representations, making them better suited to the specific needs of industrial applications.

[0163] This embodiment significantly improves the performance of LLMs in industrial applications through the ELRM-RLHF method. The model can generate more accurate, relevant, and easily understood text output, better meeting the needs of industrial experts and technicians. Furthermore, the model's adaptability and user-friendliness are enhanced, providing users with more satisfactory artificial intelligence services. Through continuous iterative optimization, the model can maintain high efficiency and reliability in constantly changing industrial environments.

[0164] The question-answering device for large model fine-tuning in ensemble learning provided by the present invention is described below. The question-answering device for large model fine-tuning in ensemble learning described below can be referred to in correspondence with the question-answering method for large model fine-tuning in ensemble learning described above.

[0165] like Figure 10 As shown, the device includes a first pre-training module 1001, a second pre-training module 1002, a fine-tuning module 1003, and a generation module 1004, wherein:

[0166] The first pre-training module 1001 is used to pre-train the LLMs model to obtain the baseline LLMs model;

[0167] The second pre-training module 1002 is used to train the reward model based on the ensemble learning strategy to obtain the ensemble reward model;

[0168] The fine-tuning module 1003 is used to fine-tune the baseline LLMs model based on the reward score output by the ensemble reward model according to the reinforcement learning strategy, so as to obtain the target LLMs model.

[0169] The generation module 1004 is used to input the user's question into the target LLMs model and obtain the answer to the question.

[0170] This embodiment uses a large-scale dataset to pre-train LLMs to learn rich linguistic and visual representations; it constructs a robust ensemble reward model by fine-tuning multiple reward models using the ELRM strategy, enabling effective evaluation of multimodal inputs; it utilizes RLHF technology to fine-tune the pre-trained LLMs to optimize their behavior and reduce hallucinations, while improving the model's sensitivity and accuracy to human feedback, thus enhancing the performance and reliability of LLMs in multimodal environments; and it achieves precise alignment with human preferences by fusing the reward model based on the ensemble learning strategy and reinforcement learning techniques based on human feedback, thereby improving the performance of LLMs in specific domains.

[0171] Figure 11 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 11As shown, the electronic device may include a processor 1110, a communications interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communications interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute a question-answering method for fine-tuning a large model based on ensemble learning. This method includes: pre-training an LLMs model to obtain a baseline LLMs model; training a reward model based on an ensemble learning strategy to obtain an ensemble reward model; fine-tuning the baseline LLMs model based on a reinforcement learning strategy according to the reward score output by the ensemble reward model to obtain a target LLMs model; and inputting the user's question into the target LLMs model to obtain the answer to the question.

[0172] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0173] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the question-answering method for fine-tuning large models for ensemble learning provided by the above methods. The method includes: pre-training an LLM model to obtain a baseline LLM model; training a reward model based on an ensemble learning strategy to obtain an ensemble reward model; fine-tuning the baseline LLM model based on a reinforcement learning strategy according to the reward score output by the ensemble reward model to obtain a target LLM model; and inputting a user's question into the target LLM model to obtain the answer to the question.

[0174] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a question-answering method for fine-tuning large models for ensemble learning provided by the methods described above. This method includes: pre-training an LLM model to obtain a baseline LLM model; training a reward model based on an ensemble learning strategy to obtain an ensemble reward model; fine-tuning the baseline LLM model based on a reinforcement learning strategy according to the reward score output by the ensemble reward model to obtain a target LLM model; and inputting a user's question into the target LLM model to obtain the answer to the question.

[0175] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0176] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for question and answer under large model fine-tuning of integrated learning, characterized in that, include: A baseline LLM model is obtained by pre-training the LLM model. An ensemble reward model is obtained by training the reward model using an ensemble learning strategy. The target LLM model is obtained by fine-tuning the baseline LLM model based on the reward score output by the integrated reward model using a reinforcement learning strategy. Input the user's question into the target LLMs model to obtain the answer to the question; The process of training the reward model based on the ensemble learning strategy to obtain the ensemble reward model includes: The first data is input into the benchmark LLMs model multiple times to obtain the answers to the first data output by the benchmark LLMs model. A first question-answer pair is generated based on the first data and the answers to the first data. Obtain the human revision results for each answer in the first data, and generate a second question-answer pair from the first data and the revision results of each answer; The first question and answer pair is sorted according to its quality, and the second question and answer pair is sorted according to its quality. The second question and answer pair is sorted before the first question and answer pair. The first question and answer and the second question and answer are then stored in the feedback database according to the sorting results. The reward model is trained using the feedback database based on the ensemble learning strategy to obtain the ensemble reward model; The step of training the reward model using the feedback database based on the ensemble learning strategy to obtain the ensemble reward model includes: The data in the feedback database is divided into a training set and a validation set; Multiple reward models are trained using different subsets of data from the training set and / or different hyperparameters; The trained reward models are validated using the validation set to obtain the performance evaluation function values ​​of each reward model on the validation set. The multiple reward models are fused based on the performance index evaluation function values ​​of each reward model on the validation set to obtain an integrated reward model; The step of training multiple reward models using different subsets of data in the training set and / or different hyperparameters includes: Input different subsets of data from the training set and / or different hyperparameters into each reward model; Based on whether the reward scores of the training set output by each reward model are Markovian, determine the preference probability of each reward model; The loss function of each reward model is determined based on the cross-entropy loss between the preference probability of each reward model on the training set and the preset preference label. The weights of each reward model are adjusted according to the loss function of each reward model.

2. The question-answering method for large model fine-tuning oriented towards ensemble learning according to claim 1, characterized in that, The weights of the integrated reward model are obtained using the following formula: ; ; in, The weights of the integrated reward model, For the i-th reward model The weights are given by N, where N is the number of reward models. Let be the performance evaluation function value of the i-th reward model on the validation set. Let be the performance evaluation function value of the i-th reward model on the first question-answer pair in the validation set. Let be the performance evaluation function value of the i-th reward model on the second question-answer pair in the validation set. For the preset adjustment coefficient, in Greater than the hour It is a positive number if it is positive, otherwise it is a negative number.

3. The question-answering method for fine-tuning large models oriented towards ensemble learning according to claim 1 or 2, characterized in that, The reward model is a multilayer perceptron, convolutional neural network, or Transformer model.

4. The question-answering method for large model fine-tuning oriented towards ensemble learning according to claim 1 or 2, characterized in that, The reinforcement learning-based strategy fine-tunes the baseline LLMs model based on the reward score output by the ensemble reward model to obtain the target LLMs model, including: The second data is input into the benchmark LLMs model and the target LLMs model respectively to obtain the answer of the second data output by the benchmark LLMs model and the target LLMs model; The answers to the second data output by the benchmark LLMs model and the target LLMs model are respectively input into the policy model to obtain the penalty term between the outputs of the policy model; The answer to the second data output by the target LLMs model is input into the ensemble reward model to obtain the reward score output by the ensemble reward model; The loss function of the target LLMs model is determined based on the penalty term and the reward score. The weights of the target LLMs model are adjusted according to the loss function of the target LLMs model.

5. The question-answering method for large model fine-tuning oriented towards ensemble learning according to claim 1 or 2, characterized in that, After inputting the user's question into the target LLMs model and obtaining the answer to the question, the process further includes: Obtain user evaluation and modification of the answers to the questions; The evaluation and modifications are used as a new dataset to fine-tune the target LLMs model again.

6. A question-answering device for fine-tuning large models in ensemble learning, characterized in that, The question-answering method applied to large model fine-tuning for ensemble learning as described in any one of claims 1-5 includes: The first pre-training module is used to pre-train the LLMs model to obtain the baseline LLMs model; The second pre-training module is used to train the reward model based on the ensemble learning strategy to obtain the ensemble reward model; The fine-tuning module is used to fine-tune the baseline LLMs model based on the reward score output by the ensemble reward model according to the reinforcement learning strategy, so as to obtain the target LLMs model. The generation module is used to input the user's question into the target LLMs model and obtain the answer to the question.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the question-answering method for large model fine-tuning oriented to ensemble learning as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Question and answer reply method, system and equipment based on human feedback and reinforcement learning

    CN116955576A