Large model fine tuning method and device, electronic equipment and computer storage medium

Abstract

The invention provides a large model fine tuning method and device, electronic equipment and a computer storage medium, and the method comprises the steps: generating a candidate answer sequence according to question information through employing a strategy model after the question information is received; afterwards, generating an evaluation value sequence by using the reference model, and generating a reward value sequence under different reward functions by using a reward model comprising the multi-dimensional reward functions; performing group advantage evaluation according to the reward value sequence under all reward functions to obtain an advantage value sequence; performing difference analysis on the dominant value sequence by using the evaluation value sequence to obtain a difference analysis result; and finally, carrying out fine adjustment on the strategy model based on a difference analysis result to obtain an optimized strategy model. By fusing a multi-dimensional reward mechanism, the accuracy, logicality and robustness of a large model in complex logic and multi-scene tasks are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a large model fine-tuning method and device, electronic equipment and computer storage medium. BACKGROUND

[0002] The existing reinforcement learning method generally depends on a specific reward calculation method and a unified format of policy update, resulting in that the reinforcement learning technology can only be applied to certain specific conditions and scenarios. For example, the existing PPO method completely depends on the reward model for scoring in the policy model update process, and the advantages and disadvantages of the model output completely depend on the reward model itself, which has the problem of singleness; DPO uses a fixed formula to calculate the reward, and excessively depends on the good and bad of the correct answer, lacking the ability to flexibly adjust to different tasks; and GRPO excessively focuses on mathematical problems and fixed format answer evaluation in reward calculation, making it difficult to adapt to more diverse application requirements.

[0003] In addition, the existing method also has deficiencies in reward design dimensions. Taking the above three methods as examples, the reward mechanism usually takes the difference between the "correct answer" and the "wrong answer" as the only evaluation basis, ignoring other key rules and multiple standards in the task. This fixed and single reward design lacks the ability to flexibly adapt to different tasks and domains, and the fixed form of calculating the reward score is not accurate for the advantage calculation of the generated answer, which may eventually cause the policy update deviation.

[0004] The defects in the reward design of the existing reinforcement learning method and the solidification of the reward calculation make it difficult for the model to consider the characteristics and actual needs of each task when facing complex logic and multi-scenario tasks, thereby affecting the generalization ability and overall performance of the model in different scenarios. SUMMARY

[0005] Therefore, the present application provides a large model fine-tuning method, device, electronic equipment and computer storage medium, which effectively improves the accuracy, logic and robustness of the large model in complex logic and multi-scenario tasks.

[0006] The first aspect of the present application provides a large model fine-tuning method, comprising:

[0007] receiving question information, and generating a candidate answer sequence according to the question information by using a policy model; wherein the candidate answer sequence comprises at least one candidate answer;

[0008] inputting the candidate answer sequence into a reference model to output an evaluation value sequence; wherein the evaluation value sequence comprises an evaluation value corresponding to each candidate answer;

[0009] inputting the candidate answer sequence into a reward model to output a reward value sequence under different reward functions; wherein the reward model comprises a reward function based on automatic adaptation accuracy of scene perception, a reward function based on dynamic scaling of content length, a reward function of adaptive format matching, and a reward function of adaptive logical word matching;

[0010] performing group advantage evaluation according to the reward value sequences under all reward functions to obtain an advantage value sequence;

[0011] performing difference analysis on the advantage value sequence by using the evaluation value sequence to obtain a difference analysis result;

[0012] fine-tuning the strategy model based on the difference analysis result to obtain an optimized strategy model.

[0013] Optionally, the reward function based on automatic adaptation accuracy of scene perception performs intent recognition on the scene of the candidate answer in the candidate answer sequence by using an intent recognition model to obtain an intent recognition result.

[0014] determining an accuracy evaluation method according to the intent recognition result, and processing the candidate answer sequence based on the accuracy evaluation method to obtain a first reward value sequence.

[0015] Optionally, the reward function based on dynamic scaling of content length uses a pre-established content length and correctness rate model to determine the correctness rate of the candidate answer according to the content length of the candidate answer in the candidate answer sequence.

[0016] determining a second reward value of the candidate answer according to the correctness rate of the candidate answer, and generating a second reward value sequence according to the second reward values of all candidate answers.

[0017] Optionally, the reward function of adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence.

[0018] Optionally, the reward function of adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence.

[0019] Optionally, the group advantage evaluation according to the reward value sequences under all reward functions to obtain an advantage value sequence comprises:

[0020] dynamically assigning a weight to the reward value sequence under each reward function according to historical reward distribution, task priority, and meta-optimizer to obtain a dynamic weight corresponding to the reward value sequence;

[0021] determining the advantage value sequence according to all reward value sequences and the dynamic weight corresponding to the reward value sequence.

[0022] Optionally, the difference analysis on the advantage value sequence based on the evaluation value sequence comprises:

[0023] The difference between the evaluation value sequence and the advantage value sequence is determined by calculating the relative entropy of the evaluation value sequence and the advantage value sequence, and a difference analysis result is obtained.

[0024] The second aspect of the present application provides a large model fine-tuning device, comprising:

[0025] A candidate answer sequence generation unit is configured to receive question information and generate a candidate answer sequence based on the question information by using a strategy model; wherein the candidate answer sequence comprises at least one candidate answer.

[0026] An evaluation value sequence generation unit is configured to input the candidate answer sequence into a reference model to output an evaluation value sequence; wherein the evaluation value sequence comprises an evaluation value corresponding to each candidate answer.

[0027] A reward value sequence generation unit is configured to input the candidate answer sequence into a reward model to output a reward value sequence under different reward functions; wherein the reward model comprises a reward function based on scene-aware automatic adaptation accuracy, a reward function based on content length dynamic scaling, a reward function based on adaptive format matching, and a reward function based on adaptive logical word matching.

[0028] A group advantage evaluation unit is configured to perform group advantage evaluation based on the reward value sequences under all reward functions to obtain an advantage value sequence.

[0029] A difference analysis unit is configured to perform difference analysis on the advantage value sequence based on the evaluation value sequence to obtain a difference analysis result.

[0030] A fine-tuning unit is configured to fine-tune the strategy model based on the difference analysis result to obtain an optimized strategy model.

[0031] Optionally, the reward function based on scene-aware automatic adaptation accuracy uses an intent recognition model to perform intent recognition on the scene of a candidate answer in the candidate answer sequence to obtain an intent recognition result.

[0032] An accuracy evaluation method is determined based on the intent recognition result, and the candidate answer sequence is processed based on the accuracy evaluation method to obtain a first reward value sequence.

[0033] Optionally, the reward function based on dynamic scaling of content length utilizes a pre-established content length and correctness rate model to determine the correctness rate of the candidate answer according to the content length of the candidate answer in the candidate answer sequence.

[0034] A second reward value of the candidate answer is determined according to the correctness rate of the candidate answer, and a second reward value sequence is generated according to the second reward values of all candidate answers.

[0035] Optionally, the reward function of adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence.

[0036] Optionally, the reward function of adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence.

[0037] Optionally, the group advantage evaluation unit comprises:

[0038] The dynamic weight allocation unit is configured to dynamically allocate weights to the reward value sequences under each reward function according to the historical reward distribution, the task priority, and the meta-optimizer, to obtain dynamic weights corresponding to the reward value sequences.

[0039] The advantage value sequence determination unit is configured to determine an advantage value sequence according to all the reward value sequences and the dynamic weights corresponding to the reward value sequences.

[0040] Optionally, the difference analysis unit comprises:

[0041] The relative entropy calculation unit is configured to determine the difference between the evaluation value sequence and the advantage value sequence by calculating the relative entropy of the evaluation value sequence and the advantage value sequence, to obtain a difference analysis result.

[0042] The third aspect of the present application provides an electronic device comprising:

[0043] One or more processors;

[0044] A storage device having one or more programs stored thereon;

[0045] When the one or more programs are executed by the one or more processors, the one or more processors implement the large model fine-tuning method according to any one of the first aspect.

[0046] The fourth aspect of the present application provides a computer storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the large model fine-tuning method according to any one of the first aspect.

[0047] From the above scheme, the application provides a large model fine-tuning method and device, electronic equipment and computer storage medium, after receiving question information, the strategy model is used to generate a candidate answer sequence according to the question information; then, the reference model is used to generate an evaluation value sequence, and the reward model containing a multi-dimensional reward function is used to generate a reward value sequence under different reward functions; then, the reward value sequence under all reward functions is used for group advantage evaluation to obtain an advantage value sequence; the evaluation value sequence is used for difference analysis of the advantage value sequence to obtain a difference analysis result; finally, the strategy model is fine-tuned based on the difference analysis result to obtain an optimized strategy model. Through the fusion of the multi-dimensional reward mechanism, the accuracy, logic and robustness of the large model in complex logic and multi-scene tasks are effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only a part of the embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0049] Figure 1 A specific flowchart of a large model fine-tuning method provided by an embodiment of the present application;

[0050] Figure 2 A flowchart of a group advantage evaluation method provided by another embodiment of the present application;

[0051] Figure 3 A whole architecture flowchart provided by another embodiment of the present application;

[0052] Figure 4 A schematic diagram of a large model fine-tuning device provided by another embodiment of the present application;

[0053] Figure 5 A schematic diagram of an electronic device for implementing a large model fine-tuning method provided by another embodiment of the present application; DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0055] The term "include" and variations thereof, as used in this document, means "to include, without limitation." The term "based on" means "based at least in part on." The term "one embodiment" means "at least one embodiment." The term "another embodiment" means "at least one additional embodiment." The term "some embodiments" means "at least some embodiments." Related terms shall be construed accordingly. Other definitions will be apparent from the description below.

[0056] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0057] It should be noted that the "first", "second", etc. concepts mentioned in the present application are only used to distinguish different devices, modules or units, and are not used to limit the order or mutual dependence of the functions performed by these devices, modules or units.

[0058] It should be noted that the modification of "one" or "multiple" in the present application is illustrative and not limiting, and those skilled in the art should understand that unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0059] The embodiment of the present application provides a large model fine-tuning method, as shown in Figure 1 The specific steps include the following steps:

[0060] S101, receiving question information, and generating a candidate answer sequence according to the question information by using a policy model.

[0061] The candidate answer sequence includes at least one candidate answer.

[0062] It can be understood that the policy model in reinforcement learning is combined with the large model as an application component of the large model.

[0063] It should be noted that the question information can be a preset training sample question information to fine-tune the policy model, or real-time question information to fine-tune the policy model in real time, which is not limited here.

[0064] S102, inputting the candidate answer sequence into a reference model to output an evaluation value sequence.

[0065] The evaluation value sequence includes an evaluation value corresponding to each candidate answer. The reference model is a standard model established in advance to constrain the generation distribution of the policy model.

[0066] S103, input the candidate answer sequence into the reward model, and output a reward value sequence under different reward functions.

[0067] The reward model comprises a reward function based on automatic adaptation of the accuracy rate of scene perception, a reward function based on dynamic scaling of content length, a reward function of adaptive format matching, and a reward function of adaptive logical word matching.

[0068] The present application integrates multi-dimensional and self-adaptive reward evaluation mechanisms in reward design, realizes flexible evaluation of different task requirements, structural characteristics and logical complexity, and greatly improves the adaptability of the reward mechanism to various scenes and the comprehensive quality of model reasoning generation.

[0069] Optionally, in another embodiment of the present application, the reward function based on automatic adaptation of the accuracy rate of scene perception uses an intent recognition model to perform intent recognition on the scene of the candidate answer in the candidate answer sequence to obtain an intent recognition result; determines an accuracy evaluation method according to the intent recognition result, and processes the candidate answer sequence based on the accuracy evaluation method to obtain a first reward value sequence.

[0070] The training method of the intent recognition model can be, but is not limited to, using a paddlenlp package, fine-tuning an ernie-3.0 model, the ernie model is a classification model developed by Baidu, the data set processing format is defined, the category and the related description, the category is the intent scene, which is not limited here.

[0071] The scene of the candidate answer includes, but is not limited to, a code scene, a general question and answer scene, a mathematical reasoning scene, etc., which is not limited here.

[0072] In the actual application process of the present application, if it is a code scene, the reward function can use, but is not limited to, code_bleu score, which mainly uses the AST (Abstract Syntax Tree) dimension, and the code_bleu AST algorithm mainly measures the similarity between the generated code content and the reference answer in the structure, syntax and data flow.

[0073] CodeBLEU is a code similarity evaluation index. It absorbs the advantages of BLEU in n-gram matching, and further injects code syntax through abstract syntax tree (AST), and injects code semantics through data flow. The similarity can be divided into, but is not limited to, 0.15 or less for dissimilarity, 0.15-0.22 for weak association, 0.22-35 for strong association, and 0.35 or more for complete association. According to these scores, the reward value of 0-2 is calculated, that is, the first reward value.

[0074] In the practical application of the present application, if it is a general question and answer scenario, the reward function can adopt, but is not limited to, a score based on cosine similarity. The cosine similarity calculated by the word embedding model can compare the similarity in semantic expression between the generated text and the standard answer, i.e., the first reward value.

[0075] In the practical application of the present application, if it is a mathematical reasoning scenario, the existing math_verify method is used, which mainly automatically checks and rewards the structured correctness of the mathematical formula and the consistency of the answer to obtain the first reward value. The math_verify can not only accurately extract mathematical expressions and formulas from the text, but also verify the correctness of the extracted mathematical content.

[0076] Optionally, in another embodiment of the present application, the reward function based on dynamic scaling of content length uses a pre-established content length and correctness model to determine the correctness of the candidate answer according to the content length of the candidate answer in the candidate answer sequence; determine the second reward value of the candidate answer according to the correctness of the candidate answer, and generate a second reward value sequence according to the second reward values of all candidate answers.

[0077] The content length and correctness model is to quantify the correctness of the model output by using the accuracy evaluation function (Accuracy), and record the completion length corresponding to each generated result (candidate answer) when the accuracy is high (i.e., the output content is correct or close to the standard answer). By continuously accumulating effective data samples, the statistical relationship curve between the generated length and the correctness is automatically fitted and dynamically updated, thereby establishing a quantifiable mapping between the content length and the output quality, i.e., the content length and correctness model.

[0078] Thereafter, for a new model generation result, its length is first predicted, and then the content length-correctness relationship model formed is used to determine the correlation strength between the current output length and the historical high-quality content. If the model generation content length falls in the historical high-accuracy area or matches the optimal length interval, the second reward value is increased; otherwise, when the content length of the generated result deviates significantly from the high-quality interval, the second reward value is appropriately decayed.

[0079] Optionally, in another embodiment of the present application, the reward function of adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answers in the candidate answer sequence.

[0080] The reward function of adaptive format matching can evaluate the structural integrity of the generated content in multiple dimensions. It can automatically identify and analyze various structural labels (such as <think> 、 <answer> 、 <reasoning> 、 <explanation> 、 <solution>The adaptive format matching reward function can be based on, but not limited to, a regular expression precision matching algorithm, and in combination with the maintenance of the label library, all possible labels are summarized and collected, and unknown labels are also automatically supplemented into the label library for subsequent matching. Subsequently, according to the structural modules required by the expected task, the system compares the coverage and nesting relationship of the labels in the generated content. When the text (candidate answer) can completely cover all predetermined structural components, and the number of matched labels is greater than or equal to 4, a high reward value is given; if there is a structural defect or the number of labels is less than 4, the reward value is correspondingly reduced. The reward value can be set in the range of 0 to 1.5.

[0081] Optionally, in another embodiment of the present application, the reward function of the adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answers in the candidate answer sequence.

[0082] Specifically, by automatically detecting the completeness of the inference chain, the step number, the logical conjunction, the cause-and-effect relationship and the hierarchical relationship, and other dimensions, the reasoning depth and orderliness of the generated content are analyzed.

[0083] In the actual application of the present application, the logical points can be counted by, but not limited to, a small model specially trained: if there are logical words in the paragraph, or there is a logical relationship between sentences, it is counted as a logical point. If the number of logical points of the generated content is greater than or equal to 3, a high reward score is obtained; if the number of logical points is less than 3, the reward value is relatively low, and the reward score is also set between 0 and 1.5.

[0084] Among them, the logical relationship between sentences can be judged by, but not limited to, logical words, which are not limited here.

[0085] S104, group advantage evaluation is performed according to the reward value sequences under all reward functions, and an advantage value sequence is obtained.

[0086] Optionally, in another embodiment of the present application, before the group advantage evaluation of the reward value sequence, the reward score can also be standardized to obtain a standardized reward value sequence.

[0087] Among them, the standardization processing method can be, but not limited to, subtracting the group mean value of the reward value of the candidate answer from the reward value and dividing by the group standard deviation, which is not limited here. The group mean value is the average value of the reward values in the reward value sequence under the reward function, and the group standard deviation is the standard deviation of the reward values in the reward value sequence under the reward function. It can be understood that the group mean value and the standard deviation of different reward functions need to be calculated separately.

[0088] Optionally, in another embodiment of the present application, one embodiment of step S104 is as follows: Figure 2 As shown, comprising:

[0089] S201, according to the historical reward distribution, the task priority and the meta-optimizer, the reward value sequence under each reward function is dynamically allocated weight, and the dynamic weight corresponding to the reward value sequence is obtained.

[0090] As can be seen, the weight allocation mechanism of the application not only depends on the historical reward distribution and the task priority, but also depends on the external meta-learning method for adaptive adjustment.

[0091] Among them, the specific implementation of the external meta-learning method is to introduce a "meta-optimizer" in the training process, which continuously monitors the scoring performance fluctuation of each rule (reward function) in different periods and different tasks, and the correlation between the reward and the final model performance. By constructing a meta-loss function, the meta-optimizer can automatically learn the mapping relationship between the rule weight and the overall performance of the model, and realize the dynamic fine-tuning of the weight parameter.

[0092] Specifically, the method of meta-training can use but is not limited to the gradient-based meta-learning Reptile algorithm, periodically sample the task batch and meta-train the weight adjustment strategy, and make real-time fine update to the weight parameter. Make the weight adaptively and accurately allocated according to the task scene and reward feedback, so as to realize the optimal balance of the reward contribution degree in complex scenes. If a rule continues to score higher (such as a score close to 1), the system will automatically reduce the weight of the rule, and increase the weight proportion of other rules, so as to prevent the reward from being too concentrated and reduce the diversity of the score.

[0093] S202, according to all reward value sequences and the dynamic weight corresponding to the reward value sequence, determine the advantage value sequence.

[0094] In the practical application process of the application, the following formula can be used to calculate the determination of the advantage value:

[0095]

[0096] Specifically, the above formula is used to weight average the scores and weights of each rule to obtain the comprehensive score r of the current candidate answer. Then the r value of each candidate answer is standardized, that is, the weighted mean mean(r) and standard deviation std(r) of all candidate answers in the current batch are used to normalize r to obtain the advantage value, so as to ensure that the advantage value is distributed in different batches and different tasks. It has comparability and robustness.

[0097] In the practical application process of the application, the answers with advantage values greater than the threshold value, the answers with advantage values ranked in the top N, can be selected as the main update target, but not limited to this.

[0098] ​The dynamic weight distribution mechanism can intelligently adjust the weight distribution of each reward index according to the historical reward distribution and the current task performance, effectively avoid the problems of reward concentration and single evaluation, and realize accurate control of multi-task and multi-dimensional evaluation standards.

[0099] S105, difference analysis is performed on the advantage value sequence by using the evaluation value sequence to obtain a difference analysis result.

[0100] In the practical application process of the present application, the difference between the evaluation value sequence and the advantage value sequence can be determined by means of KL divergence, i.e., calculating the relative entropy of the evaluation value sequence and the advantage value sequence, to obtain the difference analysis result, which is not limited here.

[0101] S106, fine-tuning the strategy model based on the difference analysis result to obtain an optimized strategy model.

[0102] As shown in the overall architecture flowchart of the present application, the present scheme is implemented by the method of Present, and the innovation method of the present application is Innovation. Figure 3 Firstly, the strategy model (Policy Model) generates a candidate answer sequence ({o1, o2, …, o G} when receiving a question q, and then inputs the candidate answer sequence into the reference model (Reference Model) and the reward model (Reward Model), respectively. The reference model outputs an evaluation value sequence, which is waiting for difference analysis (such as KL divergence calculation) with the output of the reward model. The reward model includes four reward functions, which respectively process the candidate answer sequence to obtain four reward value sequences. Then, group advantage evaluation (Group Computation) is performed on the four reward value sequences. The group advantage evaluation of the present application adopts a dynamic adjustment of optimal weight method, which uses dynamic weights to weighted average the reward values of the candidate answers of different reward functions to obtain a comprehensive score r of different reward functions, to obtain a sequence {r1, r2, …, r G}, and then performs standardization processing to obtain an advantage value sequence {A1, A2, …, A G}. Finally, difference analysis is performed on the advantage value sequence and the evaluation value sequence, and the strategy model is fine-tuned according to the difference analysis result.

[0103] From the above scheme, the application provides a large model fine-tuning method, after receiving question information, a candidate answer sequence is generated according to the question information by using a strategy model; then, an evaluation value sequence is generated by using a reference model, and a reward value sequence under different reward functions is generated by using a reward model containing a multi-dimensional reward function; then, group advantage evaluation is performed according to the reward value sequences under all reward functions to obtain an advantage value sequence; the advantage value sequence is subjected to difference analysis by using the evaluation value sequence to obtain a difference analysis result; finally, the strategy model is fine-tuned based on the difference analysis result to obtain an optimized strategy model. Through the fusion of the multi-dimensional reward mechanism, the accuracy, logic and robustness of the large model in complex logic and multi-scene tasks are effectively improved.

[0104] Another embodiment of the application provides a large model fine-tuning device, as shown in the figure, specifically comprising: Figure 4

[0105] The candidate answer sequence generation unit 401 is configured to receive question information and generate a candidate answer sequence according to the question information by using a strategy model.

[0106] The candidate answer sequence comprises at least one candidate answer.

[0107] The evaluation value sequence generation unit 402 is configured to input the candidate answer sequence into a reference model to output an evaluation value sequence.

[0108] The evaluation value sequence comprises an evaluation value corresponding to each candidate answer.

[0109] The reward value sequence generation unit 403 is configured to input the candidate answer sequence into a reward model to output a reward value sequence under different reward functions.

[0110] The reward model comprises a reward function based on automatic adaptation accuracy of scene perception, a reward function based on dynamic scaling of content length, a reward function of adaptive format matching, and a reward function of adaptive logic word matching.

[0111] Optionally, in another embodiment of the application, the reward function based on automatic adaptation accuracy of scene perception uses an intent recognition model to perform intent recognition on the scene of the candidate answer in the candidate answer sequence to obtain an intent recognition result; an accuracy evaluation method is determined according to the intent recognition result, and the candidate answer sequence is processed based on the accuracy evaluation method to obtain a first reward value sequence.

[0112] The specific working process of the units disclosed in the above embodiments of the application can be referred to the corresponding method embodiments, which will not be described here.

[0113] ​Optionally, in another embodiment of the present application, the reward function based on dynamic scaling of content length utilizes a pre-established content length and correctness rate model to determine the correctness rate of the candidate answer in the candidate answer sequence according to the content length of the candidate answer; determine the second reward value of the candidate answer according to the correctness rate of the candidate answer, and generate the second reward value sequence according to the second reward values of all candidate answers.

[0114] The specific working processes of the units disclosed in the above embodiments of the present application can be referred to the corresponding method embodiment contents, which will not be repeated here.

[0115] Optionally, in another embodiment of the present application, the reward function based on adaptive format matching generates the third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence.

[0116] The specific working processes of the units disclosed in the above embodiments of the present application can be referred to the corresponding method embodiment contents, which will not be repeated here.

[0117] Optionally, in another embodiment of the present application, the reward function based on adaptive logical word matching generates the fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence.

[0118] The specific working processes of the units disclosed in the above embodiments of the present application can be referred to the corresponding method embodiment contents, which will not be repeated here.

[0119] The group advantage evaluation unit 404 is configured to perform group advantage evaluation according to the reward value sequences under all reward functions to obtain an advantage value sequence.

[0120] Optionally, in another embodiment of the present application, one embodiment of the group advantage evaluation unit 404 comprises:

[0121] The dynamic weight allocation unit is configured to dynamically allocate weights to the reward value sequences under each reward function according to the historical reward distribution, the task priority, and the meta-optimizer to obtain dynamic weights corresponding to the reward value sequences.

[0122] The advantage value sequence determination unit is configured to determine the advantage value sequence according to all the reward value sequences and the dynamic weights corresponding to the reward value sequences.

[0123] The specific working processes of the units disclosed in the above embodiments of the present application can be referred to the corresponding method embodiment contents, as shown in Figure 2 , which will not be repeated here.

[0124] The difference analysis unit 405 is configured to perform difference analysis on the advantage value sequence by using the evaluation value sequence to obtain a difference analysis result.

[0125] Optionally, in another embodiment of the present application, an implementation of the difference analysis unit 405 comprises:

[0126] The relative entropy calculation unit is configured to determine the difference between the evaluation value sequence and the advantage value sequence by calculating the relative entropy of the evaluation value sequence and the advantage value sequence, to obtain a difference analysis result.

[0127] The specific working process of the units disclosed in the above embodiments of the present application can be referred to the corresponding method embodiment contents, which will not be described here.

[0128] The fine-tuning unit 406 is configured to fine-tune the strategy model based on the difference analysis result to obtain an optimized strategy model.

[0129] The specific working process of the units disclosed in the above embodiments of the present application can be referred to the corresponding method embodiment contents, as shown in Figure 1 , which will not be described here.

[0130] As can be seen from the above scheme, the present application provides a large model fine-tuning device, which generates a candidate answer sequence according to the problem information after receiving the problem information; then, an evaluation value sequence is generated by using a reference model, and a reward value sequence under different reward functions is generated by using a reward model containing multi-dimensional reward functions; then, group advantage evaluation is performed according to the reward value sequences under all reward functions to obtain an advantage value sequence; the evaluation value sequence is used to perform difference analysis on the advantage value sequence to obtain a difference analysis result; finally, the strategy model is fine-tuned based on the difference analysis result to obtain an optimized strategy model. By fusing the multi-dimensional reward mechanism, the accuracy, logic and robustness of the large model in complex logic and multi-scene tasks are effectively improved.

[0131] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that can be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0132] Another embodiment of the present application provides an electronic device, as shown in Figure 5 , comprising:

[0133] One or more processors 501.

[0134] A storage device 502 having one or more programs stored thereon.

[0135] When the one or more programs are executed by the one or more processors 501, the one or more processors 501 implement the large model fine-tuning method as described in the above embodiments.

[0136] Another embodiment of the present application provides a computer storage medium having stored thereon a computer program, wherein the computer program is executed by a processor to implement the large model fine-tuning method as described in the above embodiments.

[0137] In the context of the present application, a machine-readable medium can be a tangible medium which can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium can include, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more of the following: an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0138] It should be noted that the computer readable medium described above in the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0139] The computer readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device.

[0140] Another embodiment of the present application provides a computer program product, when the computer program product is executed, for executing the large model fine-tuning method described above.

[0141] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program comprising program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the methods of the embodiments of the present application are performed.

[0142] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0143] Although specific implementation details have been included in the above discussion, these should not be construed as limiting the scope of the application. Some of the features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any appropriate subcombination. For the sake of clarity, the description has not attempted to include all combinations of these various features with each other.

[0144] The above description is merely exemplary of the application and the application principles that have been employed. It is to be understood that the application is not limited to the specific devices, methods, or the like, described above, and that numerous modifications can be made by one skilled in the art without departing from the application. For example, one of ordinary skill in the art will recognize that the application is not limited to the specific implementations described above, but will also extend to other implementations that are consistent with the principles of the application.< / solution> < / explanation> < / reasoning> < / answer> < / think>

Claims

1. A large model fine-tuning method, characterized in that, Comprising: receiving question information, generating a candidate answer sequence according to the question information using a strategy model; wherein the candidate answer sequence includes at least one candidate answer; inputting the candidate answer sequence into a reference model to output an evaluation value sequence; wherein the evaluation value sequence includes an evaluation value corresponding to each candidate answer; inputting the candidate answer sequence into a reward model to output a reward value sequence under different reward functions; wherein the reward model includes a reward function based on scene-aware automatic adaptation accuracy, a reward function based on content length dynamic scaling, a reward function for adaptive format matching, and a reward function for adaptive logical word matching; performing group advantage evaluation according to the reward value sequence under all reward functions to obtain an advantage value sequence; performing difference analysis on the advantage value sequence using the evaluation value sequence to obtain a difference analysis result; fine-tuning the strategy model based on the difference analysis result to obtain an optimized strategy model.

2. The large model fine-tuning method of claim 1, wherein, The reward function based on scene-aware automatic adaptation accuracy uses an intent recognition model to perform intent recognition on the scene of the candidate answer in the candidate answer sequence to obtain an intent recognition result; determine an accuracy evaluation method according to the intent recognition result, and process the candidate answer sequence based on the accuracy evaluation method to obtain a first reward value sequence.

3. The large model fine-tuning method of claim 1, wherein, The reward function based on content length dynamic scaling uses a pre-established content length and correctness model to determine the correctness of the candidate answer according to the content length of the candidate answer in the candidate answer sequence; determine a second reward value of the candidate answer according to the correctness of the candidate answer, and generate a second reward value sequence according to the second reward values of all candidate answers.

4. The large model fine-tuning method of claim 1, wherein, The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence.

5. The large model fine-tuning method of claim 1, wherein, The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence.

6. The large model fine-tuning method of claim 1, wherein, The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence.

7. The large model fine-tuning method of claim 1, wherein, The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence.

8. A large model fine-tuning apparatus, characterized by, The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a fourth reward value sequence according to the logical normativity and orderliness of the candidate answer in the candidate answer sequence. The reward function for adaptive format matching generates a third reward value sequence according to the label coverage and nesting relationship of the candidate answer in the candidate answer sequence. The reward function for adaptive logical word matching generates a An evaluation value sequence generation unit is configured to input the candidate answer sequence into a reference model, and output an evaluation value sequence, wherein the evaluation value sequence includes an evaluation value corresponding to each candidate answer; A reward value sequence generation unit is configured to input the candidate answer sequence into a reward model, and output a reward value sequence under different reward functions, wherein the reward model includes a reward function based on automatic adaptation of accuracy rate with scene perception, a reward function based on dynamic scaling of content length, a reward function based on adaptive format matching, and a reward function based on adaptive logical word matching; A group advantage evaluation unit is configured to perform group advantage evaluation according to the reward value sequences under all reward functions, and obtain an advantage value sequence; A difference analysis unit is configured to perform difference analysis on the advantage value sequence by using the evaluation value sequence, and obtain a difference analysis result; A fine-tuning unit is configured to fine-tune the strategy model based on the difference analysis result, and obtain an optimized strategy model.

9. An electronic device, comprising: One or more processors; A storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the large model fine-tuning method according to any one of claims 1 to 7. A computer program is stored thereon, and when the computer program is executed by a processor, the large model fine-tuning method according to any one of claims 1 to 7 is implemented.

10. A computer storage medium, characterized in that ​

Images