Large model optimization method and device and medium

Abstract

The invention provides a large model optimization method and device and a medium, and relates to the technical field of computers, and the method comprises the steps: obtaining a basic large model pedestal which comprises a basic large model and a basic reward model for rewarding the output of the basic large model; forming generalization data by using high-quality samples of a plurality of professional scenes of the target domain, and performing generalization training on the basic large model to obtain a generalization large model of the target domain; using high-quality samples and non-high-quality samples of each professional scene in the target field to form professional data, and performing vertical training on the basic reward model and the generalization large model to obtain a special large model of each professional scene in the target field; and deploying the special large models to each professional scene of the target field for application. The generalization ability of the large model is improved by adopting the thought of meta-learning, the available sample size during vertical training is increased by adding the reward model, and the problem of training imbalance caused by small high-quality sample size is avoided.

Description

Technical Field

[0001] The present disclosure at least relates to the field of computer technology, and in particular, to a large model optimization method, a large model optimization device, and a computer-readable storage medium. Background Art

[0002] At present, the development and application of big models have received widespread attention. Big models are trained with common data sets, can understand and generate human language, provide decision support, and optimize business processes, thus playing an important role in multiple industries. However, although existing big model intelligent assistants have made remarkable achievements in natural language processing, image recognition, data analysis, etc., they still face some challenges and limitations.

[0003] For example, some target fields have multiple professional scenarios. It is expected that a model that can be generalized and applied in the target field can be obtained through training. At the same time, a specialized model for a professional scenario can be obtained based on the generalized model. However, multiple professional scenarios may have data imbalance, that is, some preferred categories have fewer samples. Using traditional model training methods may result in poor performance of the model in these professional scenarios. Therefore, it is necessary to design a mechanism to handle this imbalance to prevent the model from being biased towards categories with large sample sizes. Summary of the invention

[0004] The technical problem to be solved by the present disclosure is to provide a large model optimization method, a large model optimization device and a computer-readable storage medium in view of the above-mentioned shortcomings, so as to solve the problem of training imbalance caused by the small number of high-quality samples in certain professional scenarios.

[0005] In a first aspect, the present disclosure provides a large model optimization method, the method comprising:

[0006] Obtain a basic large model base, the basic large model base including a basic large model and a basic reward model for rewarding the output of the basic large model;

[0007] Use high-quality samples from multiple professional scenarios in the target field to form generalized data, perform generalization training on the basic large model, and obtain a generalized large model in the target field;

[0008] Use high-quality samples and non-high-quality samples of each professional scene in the target field to form professional data, and vertically train the basic reward model and the generalized large model to obtain a specific large model for each professional scene in the target field;

[0009] Deploy the special large models to various professional scenarios in the target field for application.

[0010] Further, a basic large model base is obtained, and the basic large model base includes a basic large model and a basic reward model for rewarding the output of the basic large model, specifically including:

[0011] Acquire general data in a general field, the general data at least including a first user question;

[0012] Using a general large model as a basic large model, inputting a first user question into the basic large model, so that the basic large model outputs a first output answer;

[0013] Inputting the first output answer and the corresponding first user question into the original reward model so that the original reward model outputs a first predicted reward for the first output answer;

[0014] Obtain a first real reward for the first output answer, and obtain a first difference between the first predicted reward and the first real reward;

[0015] Optimize the parameters of the original reward model according to the first difference to obtain a basic reward model.

[0016] Furthermore, the method further comprises:

[0017] Obtain target data for multiple professional scenarios in the target field,

[0018] The target data includes a second user question, a second real answer corresponding to the second user question, and a second real reward in each professional scenario for answering the corresponding second user question using the second real answer.

[0019] The second real reward is obtained by collecting the scores or selection frequencies of users in each professional scenario who use the second real answer to answer the corresponding second user question.

[0020] Furthermore, we use high-quality samples from multiple professional scenarios in the target field to form generalized data, and perform generalization training on the basic large model to obtain a generalized large model in the target field, including:

[0021] From the target data whose second real reward value is higher than the first threshold, obtain a second user question and a corresponding second real answer as generalized data;

[0022] Inputting a second user question in the generalized data into the basic large model so that the basic large model outputs a second output answer;

[0023] The basic large model is optimized according to the loss value between the second output answer and the second true answer to obtain a generalized large model for the target domain.

[0024] Furthermore, professional data is composed of high-quality samples and non-high-quality samples of each professional scene in the target field, and the basic reward model and the generalized large model are vertically trained to obtain a special large model for each professional scene in the target field, including:

[0025] Obtain the target data of each professional scene in the target field as each professional data, and use each professional data to perform the following vertical training:

[0026] Inputting a second user question in the professional data into the generalized large model so that the generalized large model outputs a third output answer;

[0027] Inputting the third output answer and the corresponding second user question into the basic reward model so that the basic reward model outputs a second predicted reward for the third output answer;

[0028] Obtain a third real reward for the third output answer, and obtain a second difference between the second predicted reward and the third real reward;

[0029] Optimize the basic reward model based on the second difference to obtain a specific reward model for each professional scenario in the target field;

[0030] Inputting the third output answer and the corresponding second user question into the special reward model, so that the special reward model outputs a third predicted reward for the third output answer;

[0031] The generalized big model is optimized according to the third prediction reward to obtain a specialized big model for each professional scenario in the target field.

[0032] Further, obtaining a third real reward for the third output answer and obtaining a second difference between the second predicted reward and the third real reward specifically include:

[0033] Obtaining a third real reward for a user's answer to the third output in a professional scenario corresponding to the target field; and / or,

[0034] The similarity between the third output answer and the corresponding second real answer is obtained, and the third real reward for the third output answer is obtained according to the similarity and the corresponding second real reward.

[0035] Further, wherein:

[0036] The basic reward model is optimized according to the following formula:

[0037]

[0038] In the formula, φ represents the parameters of the basic reward model, J represents the number of iterations of the basic reward model optimization, and β represents the learning rate of the basic reward model. represents the gradient of optimizing φ, and L(φ) represents the loss function used to measure the second difference;

[0039] The generalized large model is optimized according to the following formula:

[0040]

[0041] In the formula, θ represents the parameters of the generalized large model, i represents the number of iterations of the generalized large model optimization, and α represents the learning rate of the generalized large model. represents the gradient of optimizing θ, and J(θ,r) represents the objective function for rewarding r based on the third prediction.

[0042] Furthermore, the special large models are deployed to various professional scenarios in the target field for application, including:

[0043] Deploy the special student model and the special student reward model to various professional scenarios in the target field using the first container docker and the second container docker respectively;

[0044] Deploy data collection functions using the third container Docker in each professional scenario in the target field;

[0045] The data collection function collects third-party user questions newly raised by users, third-party real answers output by the special large-scale model according to the third-party user questions, and user ratings or selection frequencies of the third-party real answers;

[0046] A fourth real reward is obtained according to the user's rating or number of selections of the third real answer;

[0047] In response to the fourth real reward value being lower than the second threshold, updating the professional data of the professional scenario according to the third user question, the third real answer, and the third real reward value;

[0048] Using updated professional data, the special student reward model and the special student big model of this professional scenario are vertically trained for the second time to improve the special student big model of this professional scenario.

[0049] In a second aspect, the present disclosure provides a large model optimization device, the device comprising:

[0050] A base module, used to obtain a basic large model base, where the basic large model base includes a basic large model and a basic reward model for rewarding the output of the basic large model;

[0051] The generalization module is connected to the base module and is used to use high-quality samples of multiple professional scenarios in the target field to form generalization data and perform generalization training on the basic large model to obtain a generalized large model in the target field;

[0052] The vertical module is connected to the generalization module and is used to use the high-quality samples and non-high-quality samples of each professional scene in the target field to form professional data, and vertically train the basic reward model and the generalization model to obtain the specific large model of each professional scene in the target field;

[0053] The deployment module is connected to the vertical module and is used to deploy the special large models to various professional scenarios in the target field for application.

[0054] In a third aspect, the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the large model optimization method as described above is implemented.

[0055] The present disclosure provides a large model optimization method, a large model optimization device and a computer-readable storage medium, which adopt the idea of ​​meta-learning to improve the generalization ability of the large model, and increase the amount of available samples for vertical training in professional fields by adding a reward model to the large model, thereby avoiding the training imbalance problem caused by the small amount of high-quality samples in certain professional scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of a large model optimization method according to an embodiment of the present disclosure;

[0057] Figure 2 It is a structural schematic diagram of a large model optimization device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0059] It should be understood that the specific embodiments and drawings described herein are only used to explain the present disclosure rather than to limit the present disclosure.

[0060] It can be understood that, in the absence of conflict, the various embodiments of the present disclosure and the various features in the embodiments can be combined with each other.

[0061] It can be understood that, for the convenience of description, the drawings of the present disclosure only show parts related to the present disclosure, while parts irrelevant to the present disclosure are not shown in the drawings.

[0062] It can be understood that each module or unit involved in the embodiments of the present disclosure may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.

[0063] It will be understood that, in the absence of conflict, the functions and steps marked in the flowcharts and block diagrams of the present disclosure may occur in an order different from that marked in the drawings.

[0064] It is understood that the flowcharts and block diagrams of the present disclosure illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present disclosure. Each box in the flowchart or block diagram may represent a module, unit, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or may be implemented by a combination of hardware and computer instructions.

[0065] It can be understood that the modules and units involved in the embodiments of the present disclosure can be implemented by software or hardware, for example, the modules and units can be located in a processor.

[0066] Embodiment 1:

[0067] like Figure 1 As shown, the present disclosure provides a large model optimization method, the method comprising:

[0068] S1. Obtain a basic large model base, where the basic large model base includes a basic large model and a basic reward model for rewarding the output of the basic large model;

[0069] S2. Use high-quality samples from multiple professional scenarios in the target field to form generalized data, and perform generalization training on the basic large model to obtain a generalized large model in the target field;

[0070] S3. Use high-quality samples and non-high-quality samples of each professional scene in the target field to form professional data, and vertically train the basic reward model and the generalized large model to obtain a specific large model for each professional scene in the target field;

[0071] S4. Deploy the special large models to various professional scenarios in the target field for application.

[0072] In this embodiment, the method adopts the idea of ​​meta-learning to improve the generalization ability of the large model. By adding a reward model to the large model, the amount of available samples for vertical training in professional fields is increased, thereby avoiding the training imbalance problem caused by the small number of high-quality samples in certain professional scenarios.

[0073] Its features are: building a basic big model base. The big model base capability refers to all the capabilities that users need to manage when interacting with the system. Here, the big model will serve as the lowest-level service of the system; algorithm improvement based on feedback optimization to tune the big model itself; generalization training based on data set feature fusion to improve the generalization ability of the big model; vertical training of internal data to improve the professionalism of the big model, and designing a reward model to handle discrete feedback; the entire process is included in the intranet environment during deployment, and all mirror services must be deployed on the company's server.

[0074] Specifically, the current focus is on optimizing the performance of the model on the training distribution, with the model's own preference learning. However, in practical applications, the model often needs to operate on unseen distributions. In addition, obtaining high-quality human feedback (i.e., high-quality samples) is time-consuming and expensive, which limits the improvement of the model's adaptability in new fields.

[0075] In view of this, this embodiment proposes a large model optimization method, based on which an optimized large model office intelligent assistant can be developed to comprehensively solve the following problems: the model is not universal enough, and its adaptability and accuracy to specific fields are limited; the computational cost and maintenance cost of model training and reasoning are high; and the model does not take privacy and security into consideration enough.

[0076] That is, this embodiment proposes an improved large-model intelligent assistant technology, which not only improves the efficiency and applicability of the model, but also enhances the application capability and security of the model in specific fields by introducing a new model architecture, optimization algorithm, quantization function block, training module, and data processing module.

[0077] More specifically, this embodiment introduces a feedback mechanism based on meta-learning and reward algorithms to improve preference learning, aiming to significantly improve the system's feedback mechanism for users. This system pays special attention to optimizing the feedback algorithm between the target domain and the source data domain, thereby enhancing the generalization ability of the model.

[0078] We optimize the feedback algorithm for large model training, mainly for the reward model, and continuously update it according to the performance of the model during the training process to ensure that the model can adapt to new user preferences and feedback in a timely manner; we adopt an adaptive sampling strategy to integrate feedback from different users or scenarios to establish a comprehensive reward model, aiming to capture a wider range of preference features, rather than relying solely on single user feedback. In addition, we have carried out quantitative engineering based on the resource issues of large models. After the model is trained, a large model file will be generated. The file can be reduced by quantitative branch reduction, which greatly improves the computing power of GPU or NPU; based on the underlying large model architecture, Docker (a group of platform as a service products, based on the virtualization technology at the operating system level, the software and its dependencies are packaged as containers) is used for separation. In actual engineering applications, the trained model capability is an application point, but the model capability cannot be directly integrated and managed. It needs to be deployed through Docker images to improve the model iteration capability and uniformly manage all services with Docker. For example, the model capability side is deployed with Docker, the application capability is deployed with Docker, and the data capability is deployed with Docker. Any module will not affect other parts when it is re-iterated. In this way, the model can better adapt to different usage scenarios and data types, thereby providing a more accurate and personalized user experience, saving resources while improving the work efficiency of internal R&D personnel.

[0079] In one embodiment, S1, obtaining a basic large model base, the basic large model base includes a basic large model and a basic reward model for rewarding the output of the basic large model, specifically including:

[0080] Acquire general data in a general field, the general data at least including a first user question;

[0081] Using a general large model as a basic large model, inputting a first user question into the basic large model, so that the basic large model outputs a first output answer;

[0082] Inputting the first output answer and the corresponding first user question into the original reward model so that the original reward model outputs a first predicted reward for the first output answer;

[0083] Obtain a first real reward for the first output answer, and obtain a first difference between the first predicted reward and the first real reward;

[0084] Optimize the parameters of the original reward model according to the first difference to obtain a basic reward model.

[0085] In this embodiment, the functions of the exclusive model can be integrated based on the existing framework, different language models (LM models) can be used to integrate the reward function, and the model can be evaluated and fine-tuned, allowing various types of LM models to be tested and evaluated in the industry to determine which model is most suitable for a specific application scenario. Based on the reward function, the LM model is used for evaluation training, and on this basis, the reward function is integrated with a certain LM model to fine-tune the expected large model. For example, local tuning and test deployment are performed based on the llama3 8b large model to prove the feasibility of the solution of this embodiment. After selecting a general LM model, the reward function can be preliminarily debugged (the original reward model is optimized) to simplify the reward function tuning in subsequent professional scenarios in the target field.

[0086] Specifically, a reward function is given as an example. For the discounted accumulation of future rewards and regularization of strategy uncertainty, the following reward function is designed:

[0087]

[0088] Discount factor γ: controls the degree of discount between current rewards and future rewards. Its value is usually 0<γ≤1. It determines the importance of the impact of the strategy in future time steps in the current reward.

[0089] Future time step T: represents the cumulative time step of future rewards. In reinforcement learning, a longer T will consider more future impacts, while a shorter T will tend to make decisions at the current moment.

[0090] Discount factor λ t : represents the discount weight at the tth time step, usually 0<λ t ≤1, as the number of time steps increases, the impact of future rewards on current decisions gradually weakens;

[0091] Logarithmic difference term (logπ θ (y t ∣x t )-logπ θ (y′ t ∣x t )): indicates action y t and the suboptimal action y′ t Differences in preferences among strategies;

[0092] Uncertainty regularization term a·Var(π θ (y|x)): By introducing the variance of the action selection probability of the strategy in state x, we can prevent the strategy from being too uncertain or overfitting the actions in certain states. A large variance will make the strategy unstable. Therefore, the role of this term is to reduce the uncertainty of the strategy through regularization. a is a regularization coefficient used to balance preference learning and uncertainty control.

[0093] More specifically, after the training, the meta-learning needs to have a corresponding reward function to see the degree of effectiveness of this learning; the first loss value (discrimination loss): this loss value is calculated by comparing the difference between the predicted quality scores of the positive and negative replies. This difference measures the model's ability to distinguish between positive and negative samples. For example, for the user request "I want to listen to some music", the quality score of the positive sample reply "Received, play a piece of music for you immediately" may be 0.9, while the quality score of the negative sample reply "It is sunny today" may be only 0.05. If the positive sample reply score is 0.9 and the negative sample reply score is 0.3, the first loss value is 0.6 (the smaller the loss, the better). The second loss value (prediction accuracy loss): this loss value is calculated based on the difference between the quality score predicted by the model and the pre-set target score. For example, if the model presets the target scores of positive and negative samples as 0.9 and 0.3 respectively, but the model actually predicts 0.7 for positive samples and 0.1 for negative samples, then the calculated loss value is (0.9-0.7) 2 +(0.3-0.1) 2 =0.08, the total loss value is the sum of the first loss value and the second loss value.

[0094] In one embodiment, the method further comprises:

[0095] Obtain target data for multiple professional scenarios in the target field,

[0096] The target data includes a second user question, a second real answer corresponding to the second user question, and a second real reward in each professional scenario for answering the corresponding second user question using the second real answer.

[0097] The second real reward is obtained by collecting the scores or selection frequencies of users in each professional scenario who use the second real answer to answer the corresponding second user question.

[0098] In this embodiment, the data itself contains diversity and imbalance. For example, the user-side data may include user data of different ages and interests. Preference data often has the problem of uneven categories. For example, some specific types of products or services may have fewer samples. The model needs to design a mechanism to deal with this imbalance to prevent the model from being biased towards categories with large sample sizes.

[0099] For example, the operator's question and answer data includes daily work exchanges, customer consultation questions and answers, etc. For a basic big model, it can understand simple greetings or daily content, but for the content of the operator system, such as ice cream packages, the big model does not know what this content is and needs to be trained to recognize that the ice cream package belongs to the number card business.

[0100] In response to the above problems, this embodiment collects text data from multiple channels of operators, including customer service question and answer data, operator company internal operation manuals, databases, daily work communication information, etc., and processes and converts them into a unified data format to adapt to the training needs of large models. The data includes positive evaluation data (high-quality samples) and non-positive evaluation data (non-high-quality samples). Through meta-learning, a small number of training samples are used to quickly improve the training effect of the model.

[0101] In one embodiment, S2, using high-quality samples of multiple professional scenarios in the target field to form generalized data, generalizing the basic large model to obtain a generalized large model in the target field, specifically includes:

[0102] From the target data whose second real reward value is higher than the first threshold, obtain a second user question and a corresponding second real answer as generalized data;

[0103] Inputting a second user question in the generalized data into the basic large model so that the basic large model outputs a second output answer;

[0104] The basic large model is optimized according to the loss value between the second output answer and the second true answer to obtain a generalized large model for the target domain.

[0105] In this embodiment, the advantage of meta-learning training is that the parameter distribution can be quickly adjusted through a small number of training samples. In addition, since the parameters are shared under the reward model corresponding to each training, different important parameters can be directly extracted in each training to carry out the next batch of training.

[0106] Specifically, the large model meta-learning algorithm is implemented by using adversarial classifiers across domains to train feature embeddings (feature mapping processing) and task classifiers. Use pre-trained embedding models and classification models for initialization, iteratively update the hyperparameters of the adversarial classifier and feature embedding (internal training process), and iteratively update model parameters for small batches of data in specific fields to quickly implement specific features in each field; (external training) update feature embedding and task classifier models in multiple tasks, and optimize the generalization ability of the model in an integrated manner; based on the above process, the meta-training parameter process is completed, and the feature embedding model and task classifier model can be placed in multiple new tasks, focusing on the model's processing and performance capabilities on unseen data. An example is as follows:

[0107] Meta-learning training core:

[0108]

[0109]

[0110] Among them, θ represents the global parameters of the model, which needs to be optimized in multiple tasks, b represents the loss degree of the inner loop model training, represents the gradient of the loss function and parameter θ of the i-th task, which updates the task-related parameters in the inner loop; θ i represents the parameter value of the i-th task after the inner loop update, which reflects the learning outcome of this task; c represents the learning rate of the outer loop, which is used to update the global parameter θ to improve the model's task generalization ability in multiple aspects; represents the loss of all tasks, where each task is updated by the inner loop θ i Calculation, this synthesis is applied to the global parameter update in the outer loop, that is, the generalization ability for unknown data.

[0111] In one embodiment, S3, using high-quality samples and non-high-quality samples of each professional scene in the target field to form professional data, vertically training the basic reward model and the generalized large model to obtain a special large model for each professional scene in the target field, specifically including:

[0112] Obtain the target data of each professional scene in the target field as each professional data, and use each professional data to perform the following vertical training:

[0113] Inputting a second user question in the professional data into the generalized large model so that the generalized large model outputs a third output answer;

[0114] Inputting the third output answer and the corresponding second user question into the basic reward model so that the basic reward model outputs a second predicted reward for the third output answer;

[0115] Obtain a third real reward for the third output answer, and obtain a second difference between the second predicted reward and the third real reward;

[0116] Optimize the basic reward model based on the second difference to obtain a specific reward model for each professional scenario in the target field;

[0117] Inputting the third output answer and the corresponding second user question into the special reward model, so that the special reward model outputs a third predicted reward for the third output answer;

[0118] The generalized big model is optimized according to the third prediction reward to obtain a specialized big model for each professional scenario in the target field.

[0119] In this embodiment, a new feedback algorithm mechanism is adopted based on a large model, and the new feedback algorithm mechanism involves two-layer optimization. The first outer layer optimizes and updates the policy parameters to improve the alignment between the output generated by the model and human preferences. For example, if a question is biased towards operator consumption information, the outer optimization strategy is to help the model's answer be more in line with normal people's replies, such as biasing automatic replies towards human expectations; the second layer is internal optimization, updating the reward model parameters to minimize the gap between the predicted reward and the real reward. The real reward is an evaluation of the model output based on human feedback or other criteria. For example, in a text generation task, the real reward may be a user's rating or choice of a generated text; the predicted reward is a reward value calculated by the output generated by the model based on the input. The reward model learns these values ​​based on training data, aiming to simulate human judgment. By designing a reward model to handle discrete feedback, and through internal and external double-layer optimization, the model can better cope with performance issues in out-of-distribution situations. In other words, the purpose of designing a reward model is to first optimize the reward model through real rewards. The optimized reward model can simulate humans giving rewards to the model's output answers, so that non-high-quality samples can be added to the training data and the number of training samples can be increased. Finally, the large model is optimized based on the given rewards to improve the applicability and accuracy of the large model in simulating human answers to questions in professional scenarios.

[0120] Specifically, the calculation example of the model optimization strategy is as follows:

[0121]

[0122] in, is the advantage function; is the pruning hyperparameter; E t [] is the expected value at time step t, which represents the average calculation of multiple time step samples, sampled from multiple state-action pairs in one training iteration; r t (θ) is the ratio of the current policy to the old policy, defined as:

[0123]

[0124] The strategy described here is the strategy for corresponding algorithm optimization, where π θ ()express is the previous strategy, and this ratio reflects the ratio of the probability of the current strategy selecting an action in a specific state to the old strategy.

[0125] The clip() function uses the clipping mechanism to t (θ) is limited to prevent the problem of excessive policy updates; minimize this L CLIP (θ) function, which means that this strategy will be better.

[0126] More specifically, this embodiment hopes to generalize better on data from multiple different directions through a large model. The main problem to be solved is how to use limited resources to better train the model. Based on a feature point of meta-learning, the characteristics of different data sets can be generalized. For example, in a specific and detailed task set (there are consultation data on phone bills, consultation data on complaints, material data for writing PPTs for internal discussions, etc.), the overall task can be placed on a data set [task1, task2, task3...]. Meta-learning has its own user orientation for the learning process of task1 (there are corresponding user questions and answers in task1). On this basis, the large model slowly evolves into a human question and answer through training iterations; the same is true for the learning process of task2 and task3; in this way, the model has a better performance effect on different types of task sets, such as:

[0127]

[0128] The above formula is an optimization of the inner loop in each meta-learning; where T is the total number of time steps, Means the corresponding state and loss at time t;

[0129]

[0130] The above formula is the optimization of the outer loop, N is the corresponding task amount, and T is the number of time steps;

[0131] With the joint optimization of the inner and outer layers, the model can achieve better results for multi-task datasets.

[0132] In one embodiment, obtaining a third real reward for the third output answer and obtaining a second difference between the second predicted reward and the third real reward specifically include:

[0133] Obtaining a third real reward for a user's answer to the third output in a professional scenario corresponding to the target field; and / or,

[0134] The similarity between the third output answer and the corresponding second real answer is obtained, and the third real reward for the third output answer is obtained according to the similarity and the corresponding second real reward.

[0135] In this embodiment, a simple example of the difference between the prediction of the quantified reward model and the real reward is that, for example, the standard answer of humans is 10 points, the first answer of the model may be 2 points, and the second answer may be 5 points. The model is constantly compared with the standard answer of humans to adjust the accuracy of the model. In the early training, user participation may be required to obtain better sample data. After that, the algorithm can control the similarity between the comparison model output and the real answer, and refer to the high-quality samples to obtain the simulated human's real evaluation of the model output answer. For example, for non-high-quality samples, the answer output by the model is different from the real answer in the sample. The similarity between the two is calculated first. The closer the two are, the closer the third real reward for the output answer is to the second real reward. A similar process can also be used to obtain the first real reward for the first output answer, and the present disclosure does not exclude that when training the basic reward model, the mode of users giving real rewards corresponding to professional scenes in the target field can be adopted to directly obtain the special reward model of each professional scene as the basic reward model.

[0136] In one embodiment, wherein:

[0137] The basic reward model is optimized according to the following formula:

[0138]

[0139] In the formula, φ represents the parameters of the basic reward model, j represents the number of iterations of the basic reward model optimization, and β represents the learning rate of the basic reward model. represents the gradient of optimizing φ, and L(φ) represents the loss function used to measure the second difference;

[0140] The generalized large model is optimized according to the following formula:

[0141]

[0142] In the formula, θ represents the parameters of the generalized large model, i represents the number of iterations of the generalized large model optimization, and α represents the learning rate of the generalized large model. represents the gradient of optimizing θ, and J(θ,r) represents the objective function for rewarding r based on the third prediction.

[0143] In this embodiment, the specific control method of the double-layer optimization is as follows:

[0144] Outer layer optimization: Among them, J(θ,r) is the objective function based on the reward signal r, which is related to the degree of alignment with human preferences. This function is used to adjust the expected value of human responses and model responses, while preventing some hallucinations or ethical issues in the model responses. α is the outer learning rate;

[0145] Inner layer optimization: Among them, L(φ) is the loss function, which measures the error between the predicted reward and the actual reward, and β is the inner learning rate; in each iteration, the inner optimization is performed multiple times to ensure that the reward model parameters can quickly converge to a better solution, and then the outer optimization is performed to update the strategy parameters.

[0146] In one implementation, S4, deploying each of the special large models to various professional scenarios in the target field for application, specifically includes:

[0147] Deploy the special student model and the special student reward model to various professional scenarios in the target field using the first container docker and the second container docker respectively;

[0148] Deploy data collection functions using the third container Docker in each professional scenario in the target field;

[0149] The data collection function collects third-party user questions newly raised by users, third-party real answers output by the special large-scale model according to the third-party user questions, and user ratings or selection frequencies of the third-party real answers;

[0150] A fourth real reward is obtained according to the user's rating or number of selections of the third real answer;

[0151] In response to the fourth real reward value being lower than the second threshold, updating the professional data of the professional scenario according to the third user question, the third real answer, and the third real reward value;

[0152] Using updated professional data, the special student reward model and the special student big model of this professional scenario are vertically trained for the second time to improve the special student big model of this professional scenario.

[0153] In this embodiment, docker is a mirror service. All services at different levels can be placed in docker. Finally, the internal server only needs to build the docker service to use it. The model is based on docker to achieve personalized encapsulation. There will be different algorithm models for different business systems. For example, business A has A series models, and business B has B series models. For the functions of different businesses in actual business, they will be placed in different mirrors to provide private service capabilities and maintain stability. At the same time, actual user feedback is obtained in the actual application of the model, and the actual user feedback is added to the sample as reward data. Based on the principle of the reward model, the large model can be continuously improved.

[0154] In addition, this embodiment also includes the following technical features:

[0155] 1) Perform randomized training on multiple data sets (such as customer service data and internal problem data); in the initialization stage, select different parameters according to the characteristics of the data set to initialize multiple feedback algorithms; then perform basic supervised fine-tuning training, using labeled data sets to train and optimize model performance; based on the results of supervised fine-tuning training, select the feedback algorithm with the best initialization parameters for performance evaluation and score calculation;

[0156] 2) Based on the supervised fine-tuning training and feedback algorithm optimization, the optimal feedback algorithm model is selected. The model is tested on different data sets based on the results of supervised fine-tuning. This method can use probabilistic statistical models, such as Gaussian distribution, to evaluate the data generalization of clustering results, thereby verifying the effectiveness and adaptability of the selected model.

[0157] 3) Based on the untrained initial data, the model of the feedback algorithm is trained through the target data and the source data, and the data set is batch processed. The parameters of the reward model are updated successively through the model optimized by batch processing; at the same time, the test is performed on the target data, and the reward model parameters are updated according to the calculated loss to improve the accuracy and efficiency of the model;

[0158] 4) Based on the large model, a new feedback algorithm mechanism is used for parameter training, including full parameter training and fine-tuning operations. In order to improve the applicability and efficiency of training, multiple open source small models are selected for data training and testing to verify the versatility and advantages of the new feedback algorithm on different models;

[0159] 5) Use the operator's daily office data to train the large model agent, including data collection, processing and conversion, to adapt it to the training needs of the large model, including but not limited to the user's system operation manual, database operation manual, user's daily work communication information, etc.;

[0160] 6) Operators’ daily office data includes two knowledge feeding methods. The first is to collect data during model training, train the large model itself, and obtain the best open source small model version based on multiple open source small model training. All data sets can be obtained through supervised fine-tuning training and feedback algorithm’s overall score method to obtain corresponding categories. The classification method is targeted at the operator’s internal scenarios. For example, whether this data is positively evaluated, whether this topic belongs to the number card business or the broadband business, etc.;

[0161] 7) The second feeding method is the feedback mode. After the model is deployed, the user feedback on the model application is collected. The m3e open source embedding can be used for deployment. The m3e model will vectorize the data to become the vector unit of the large model, so as to better retrieve and reply data in the pg vector library, and open the pg service as a docker service for individual function encapsulation;

[0162] 8) After professional data training, the parameters of the large model will increase significantly. The large model can be optimized and quantized from float to int using llama cpp. The original 16g model file can be compressed to less than 5g, and the understanding effect of the large model is also guaranteed accordingly; quantization operations are performed on the trained model. Generally, there are dynamic quantization and static quantization. Dynamic quantization dynamically converts floating-point weights and activation values ​​to lower precision according to input data during inference. Static quantization maps weights and activation values ​​to low-precision formats according to the distribution information of the training set after model training;

[0163] 9) It can have autonomous training methods and optimization methods for large models, embedded model block management, and data optimization management; the understanding ability based on the training of the large model needs to be embedded in the company's actual applications, understand the intention of each user's data, and determine whether it is necessary to suspend internal interface services, including but not limited to address book query interface, meeting reservation interface, company general data query interface, etc.;

[0164] 10) Data feature engineering can be done in advance for data in different fields. At the same time, data feature alignment can be used to improve subsequent generalization capabilities. The application side can use the langchain method for pipeline development.

[0165] This embodiment mainly uses meta-learning and a reward function to perform efficient data generalization tuning on the model; after the tuning is completed, the model is quantified, and after being deployed on the company's intranet, resources can be greatly optimized; at the same time, all services can be deployed through docker to reduce maintenance costs. Specifically, the advantages include the following:

[0166] 1) Saving computing resources and generalization in multiple fields: The method of this embodiment utilizes the characteristics of meta-learning and does not need to rely on other specific machine learning or deep learning models, thereby saving computing resources. Due to the generalization ability of meta-learning, this method can be easily adapted and applied to multiple different fields;

[0167] 2) Improving generalization ability and few-sample learning: The method of this embodiment can significantly improve the generalization ability of a large model in an unknown field through its algorithm module. In particular, when there is a small amount of high-quality data, the few-sample learning ability of this method shows obvious advantages;

[0168] 3) Modular deployment structure: The method of this embodiment adopts a modular deployment structure, and different services use different images for point-to-point application deployment, allowing each module to be maintained separately during maintenance without taking the entire application offline;

[0169] 4) Optimize computing resources and service updates: In version updates and service deployment, the method of this embodiment adopts the Ollama framework system to optimize the use of computing resources. Unlike traditional methods, this solution does not need to restart the entire k8s environment when switching models, but can continue to run. In addition, the token authentication function is integrated with other image services to achieve independent operation and management, further improving the efficiency and stability of the service. The functions are integrated with other image services and independently operated and managed;

[0170] 5) Resource requirements: Multi-container management means that only large models need to use GPU resources. The optimization part can perform SFT fine-tuning training on large models, and unified quantization processing can be performed on the trained model part and the original model part. The quantized model still has a good performance form, and all data will be isolated from the public network, and there is no risk of data leakage.

[0171] Embodiment 2:

[0172] like Figure 2 As shown, the present disclosure provides a large model optimization device, the device comprising:

[0173] Base module 1, used to obtain a basic large model base, the basic large model base includes a basic large model and a basic reward model for rewarding the output of the basic large model;

[0174] Generalization module 2, connected to base module 1, is used to use high-quality samples of multiple professional scenarios in the target field to form generalization data, perform generalization training on the basic large model, and obtain a generalized large model in the target field;

[0175] Vertical module 3, connected to generalization module 2, is used to use high-quality samples and non-high-quality samples of each professional scene in the target field to form professional data, and vertically train the basic reward model and the generalization model to obtain a specific large model for each professional scene in the target field;

[0176] The deployment module 4 is connected to the vertical module 3 and is used to deploy the special large models to various professional scenarios in the target field for application.

[0177] In one embodiment, the base module 1 specifically includes:

[0178] A general data unit, used to obtain general data in a general field, the general data at least including a first user question;

[0179] A first output unit, connected to the general data unit, is used to use the general large model as a basic large model, input the first user question into the basic large model, so that the basic large model outputs a first output answer;

[0180] A first prediction unit, connected to the first output unit, is used to input the first output answer and the corresponding first user question into the original reward model, so that the original reward model outputs a first predicted reward for the first output answer;

[0181] A first difference unit, connected to the first prediction unit, is used to obtain a first real reward for the first output answer, and obtain a first difference between the first predicted reward and the first real reward;

[0182] The first optimization unit is connected to the first difference unit and is used to optimize the parameters of the original reward model according to the first difference to obtain a basic reward model.

[0183] In one embodiment, the device further comprises a target data module, which is used to:

[0184] Obtain target data for multiple professional scenarios in the target field,

[0185] The target data includes a second user question, a second real answer corresponding to the second user question, and a second real reward in each professional scenario for answering the corresponding second user question using the second real answer.

[0186] The second real reward is obtained by collecting the scores or selection frequencies of users in each professional scenario who use the second real answer to answer the corresponding second user question.

[0187] In one embodiment, the generalization module 2 specifically includes:

[0188] A generalized data unit, connected to the target data module, for obtaining a second user question and a corresponding second real answer as generalized data from the target data whose second real reward value is higher than the first threshold;

[0189] A second output unit, connected to the generalized data unit, is used to input the second user question in the generalized data into the basic large model so that the basic large model outputs a second output answer;

[0190] The second optimization unit is connected to the second output unit, and is used to optimize the basic large model according to the loss value between the second output answer and the second true answer to obtain a generalized large model of the target domain.

[0191] In one embodiment, the vertical module 3 specifically includes:

[0192] The professional data unit is connected to the target data module and is used to obtain the target data of each professional scene in the target field as each professional data, and each professional data is used to perform the following vertical training:

[0193] A third output unit, connected to the professional data unit, is used to input the second user question in the professional data into the generalized large model, so that the generalized large model outputs a third output answer;

[0194] A second prediction unit, connected to the third output unit, is used to input the third output answer and the corresponding second user question into the basic reward model, so that the basic reward model outputs a second predicted reward for the third output answer;

[0195] A second difference unit, connected to the second prediction unit, is used to obtain a third real reward for the third output answer, and obtain a second difference between the second predicted reward and the third real reward;

[0196] A third optimization unit, connected to the second difference unit, is used to optimize the basic reward model according to the second difference to obtain a special reward model for each professional scenario in the target field;

[0197] A third prediction unit, connected to the third optimization unit, is used to input the third output answer and the corresponding second user question into the special reward model, so that the special reward model outputs a third predicted reward for the third output answer;

[0198] The fourth optimization unit is connected to the third prediction unit, and is used to optimize the generalized big model according to the third prediction reward to obtain a special big model for each professional scenario in the target field.

[0199] In one implementation, the second difference unit is specifically configured to:

[0200] Obtaining a third real reward for a user's answer to the third output in a professional scenario corresponding to the target field; and / or,

[0201] The similarity between the third output answer and the corresponding second real answer is obtained, and the third real reward for the third output answer is obtained according to the similarity and the corresponding second real reward.

[0202] In one embodiment, wherein:

[0203] The third optimization unit optimizes the basic reward model according to the following formula:

[0204]

[0205] In the formula, φ represents the parameters of the basic reward model, j represents the number of iterations of the basic reward model optimization, and β represents the learning rate of the basic reward model. represents the gradient of optimizing φ, and L(φ) represents the loss function used to measure the second difference;

[0206] The fourth optimization unit optimizes the generalized large model according to the following formula:

[0207]

[0208] In the formula, θ represents the parameters of the generalized large model, i represents the number of iterations of the generalized large model optimization, and α represents the learning rate of the generalized large model. represents the gradient of optimizing θ, and J(θ,r) represents the objective function for rewarding r based on the third prediction.

[0209] In one implementation, the deployment module 4 specifically includes:

[0210] The first deployment unit is used to deploy the special student model and the special student reward model to various professional scenarios in the target field using the first container docker and the second container docker respectively;

[0211] The second deployment unit is used to deploy the data collection function using the third container docker in each professional scenario in the target field;

[0212] The model application unit is connected to the first deployment unit and the second deployment unit, and is used to collect, through a data collection function, third user questions newly raised by users, third real answers output by the special large model according to the third user questions, and user ratings or selection frequencies of the third real answers;

[0213] An application feedback unit, connected to the model application unit, for obtaining a fourth real reward according to a score or a number of selections of the third real answer by the user;

[0214] a first updating unit, connected to the application feedback unit, for updating the professional data of the professional scenario according to the third user question, the third real answer and the third real reward value in response to the fourth real reward value being lower than the second threshold;

[0215] The second updating unit is connected to the first updating unit, and is used to use the updated professional data to perform secondary vertical training on the special reward model and the special large model of the professional scenario to improve the special large model of the professional scenario.

[0216] Embodiment 3:

[0217] Embodiment 3 of the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the large model optimization method as described in Embodiment 1 is implemented, or the large model optimization device as described in Embodiment 2 is implemented.

[0218] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0219] In addition, the present disclosure may also provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the large model optimization method as described in Example 1. The computer device may be the large model optimization device as described in Example 2.

[0220] The memory is connected to the processor, the memory may be a flash memory or a read-only memory or other memory, and the processor may be a central processing unit or a single-chip microcomputer.

[0221] Embodiments 1-3 of the present disclosure provide a large model optimization method, a large model optimization device, and a computer-readable storage medium, which adopt the idea of ​​meta-learning to improve the generalization ability of the large model, and increase the number of available samples for vertical training in professional fields by adding a reward model to the large model, thereby avoiding the training imbalance problem caused by the small number of high-quality samples in certain professional scenarios.

[0222] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A large model optimization method, characterized in that: The method comprises: Obtain a basic large model base, the basic large model base including a basic large model and a basic reward model for rewarding the output of the basic large model; Use high-quality samples from multiple professional scenarios in the target field to form generalized data, perform generalization training on the basic large model, and obtain a generalized large model in the target field; Use high-quality samples and non-high-quality samples of each professional scene in the target field to form professional data, and vertically train the basic reward model and the generalized large model to obtain a specific large model for each professional scene in the target field; Deploy the special large models to various professional scenarios in the target field for application.

2. The method according to claim 1, characterized in that Get the basic large model base. The basic large model base includes the basic large model and the basic reward model that rewards the output of the basic large model, including: Acquire general data in a general field, the general data at least including a first user question; Using a general large model as a basic large model, inputting a first user question into the basic large model, so that the basic large model outputs a first output answer; Inputting the first output answer and the corresponding first user question into the original reward model so that the original reward model outputs a first predicted reward for the first output answer; Obtain a first real reward for the first output answer, and obtain a first difference between the first predicted reward and the first real reward; Optimize the parameters of the original reward model according to the first difference to obtain a basic reward model.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Obtain target data for multiple professional scenarios in the target field, The target data includes a second user question, a second real answer corresponding to the second user question, and a second real reward in each professional scenario for answering the corresponding second user question using the second real answer. The second real reward is obtained by collecting the scores or selection frequencies of users in each professional scenario who use the second real answer to answer the corresponding second user question.

4. The method according to claim 3, characterized in that Use high-quality samples from multiple professional scenarios in the target field to form generalized data, and perform generalization training on the basic large model to obtain a generalized large model in the target field, including: From the target data whose second real reward value is higher than the first threshold, obtain a second user question and a corresponding second real answer as generalized data; Inputting a second user question in the generalized data into the basic large model so that the basic large model outputs a second output answer; The basic large model is optimized according to the loss value between the second output answer and the second true answer to obtain a generalized large model for the target domain.

5. The method according to claim 3, characterized in that: Use high-quality samples and non-high-quality samples of each professional scene in the target field to form professional data, and vertically train the basic reward model and the generalized large model to obtain a special large model for each professional scene in the target field, including: Obtain the target data of each professional scene in the target field as each professional data, and use each professional data to perform the following vertical training: Inputting a second user question in the professional data into the generalized large model so that the generalized large model outputs a third output answer; Inputting the third output answer and the corresponding second user question into the basic reward model so that the basic reward model outputs a second predicted reward for the third output answer; Obtain a third real reward for the third output answer, and obtain a second difference between the second predicted reward and the third real reward; Optimize the basic reward model based on the second difference to obtain a specific reward model for each professional scenario in the target field; Inputting the third output answer and the corresponding second user question into the special reward model, so that the special reward model outputs a third predicted reward for the third output answer; The generalized big model is optimized according to the third prediction reward to obtain a specialized big model for each professional scenario in the target field.

6. The method according to claim 5, characterized in that Obtaining a third real reward for the third output answer, and obtaining a second difference between the second predicted reward and the third real reward, specifically including: Obtaining a third real reward for a user's answer to the third output in a professional scenario corresponding to the target field; and / or, The similarity between the third output answer and the corresponding second real answer is obtained, and the third real reward for the third output answer is obtained according to the similarity and the corresponding second real reward.

7. The method according to claim 5, characterized in that in: The basic reward model is optimized according to the following formula: In the formula, φ represents the parameters of the basic reward model, j represents the number of iterations of the basic reward model optimization, and β represents the learning rate of the basic reward model. represents the gradient of optimizing φ, and L(φ) represents the loss function used to measure the second difference; The generalized large model is optimized according to the following formula: In the formula, θ represents the parameters of the generalized large model, i represents the number of iterations of the generalized large model optimization, and α represents the learning rate of the generalized large model. represents the gradient of optimizing θ, and J(θ,r) represents the objective function for rewarding r based on the third prediction.

8. The method according to any one of claims 5 to 7, characterized in that: Deploy the special large models to various professional scenarios in the target field for application, including: Deploy the special student model and the special student reward model to various professional scenarios in the target field using the first container docker and the second container docker respectively; Deploy data collection functions using the third container Docker in each professional scenario in the target field; The data collection function collects third-party user questions newly raised by users, third-party real answers output by the special large-scale model according to the third-party user questions, and user ratings or selection frequencies of the third-party real answers; A fourth real reward is obtained according to the user's rating or number of selections of the third real answer; In response to the fourth real reward value being lower than the second threshold, updating the professional data of the professional scenario according to the third user question, the third real answer, and the third real reward value; Using updated professional data, the special student reward model and the special student big model of this professional scenario are vertically trained for the second time to improve the special student big model of this professional scenario.

9. A large model optimization device, characterized in that: The device comprises: A base module, used to obtain a basic large model base, where the basic large model base includes a basic large model and a basic reward model for rewarding the output of the basic large model; The generalization module is connected to the base module and is used to use high-quality samples of multiple professional scenarios in the target field to form generalization data and perform generalization training on the basic large model to obtain a generalized large model in the target field; The vertical module is connected to the generalization module and is used to use the high-quality samples and non-high-quality samples of each professional scene in the target field to form professional data, and vertically train the basic reward model and the generalization model to obtain the specific large model of each professional scene in the target field; The deployment module is connected to the vertical module and is used to deploy the special large models to various professional scenarios in the target field for application.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the large model optimization method according to any one of claims 1 to 8 is implemented.

Images