Enterprise large model training system and method based on self-iteration

Abstract

The invention discloses a self-iteration-based enterprise large model training system and method, belongs to the technical field of artificial intelligence, and aims to solve the technical problem that a large model is poor in performance when facing non-professional users in the field of enterprise knowledge questions and answers. Comprises: a self-iteration data set construction module for constructing a question and answer corpus based on a target domain corpus and a teacher model, performing question answer analysis based on the target model and the teacher model, and constructing a self-iteration data set through multiple iterations of the answer analysis; the training data construction module constructs the self-iteration data set into a self-iteration training data set based on a preset format; and the model training module is used for configuring a reward strategy and training a model through a GRPO training method based on the reward strategy and the self-iteration training data set.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a self-iterative enterprise large model training system and method. Background Art

[0002] With the continuous development of artificial intelligence (AI) technology, big models have shown tremendous potential for application in enterprise knowledge management. Leveraging their powerful semantic understanding and data processing capabilities, big models have revolutionized enterprise knowledge management. Through various co-engineering techniques, big models can automatically integrate massive amounts of information from within and outside the enterprise, construct comprehensive and accurate knowledge graphs, and output precise answers in natural language. However, despite the enormous potential of big models in enterprise knowledge management, practical challenges remain. In particular, big models often perform poorly with non-expert users within the enterprise. For example, they are prone to erroneous or inaccurate interpretations of presentation order, complex tables, and multimodal data.

[0003] Large models perform poorly when facing non-professional users in the field of enterprise knowledge question answering, which is a technical problem that needs to be solved. Summary of the Invention

[0004] The technical task of the present invention is to address the above shortcomings and provide a self-iterative enterprise big model training system and method to solve the technical problem that big models perform poorly when facing non-professional users in the field of enterprise knowledge question answering.

[0005] In a first aspect, the present invention provides a self-iterative enterprise large model training system, comprising a self-iterative dataset construction module, a training data construction module, and a model training module;

[0006] The self-iterative dataset construction module is used to construct a question-answer corpus based on the target domain corpus and the teacher model, perform answer analysis on the questions based on the target model and the teacher model, and construct a self-iterative dataset through multiple iterations of answer analysis, wherein the self-iterative dataset includes all questions corresponding to correct answers and analysis results of incorrect answers;

[0007] The training data construction module is used to construct the self-iterative dataset into a self-iterative training dataset based on a predetermined format;

[0008] The model training module is used to configure the reward strategy and train the model through the GRPO training method based on the reward strategy and the self-iterative training data set.

[0009] Preferably, the self-iterative dataset construction module is configured to perform the following operations:

[0010] Constructing a question-answer corpus: Based on the target domain corpus and the teacher model, a question-answer corpus is generated. The corpus includes a series of representative question-answer pairs that cover common questions and scenarios in enterprise knowledge Q&A.

[0011] Target model execution and evaluation: The questions in the question-answering corpus are used as input questions, and the target model is called to answer the input questions, obtain preliminary results, and evaluate the preliminary results;

[0012] Error example analysis and feedback: For questions answered incorrectly by the target model, the incorrect question and the corresponding incorrect answer are used as input. The teacher model is called to analyze the target model's incorrect answer and generate analysis results. The analysis results indicate the reasons for the incorrect answer and provide improvement suggestions.

[0013] Iteratively optimize the question-answering corpus: Merge the questions answered incorrectly by the target model and the analysis results output by the teacher model to form new questions. Use the new questions as input questions, execute and judge the target model, and analyze and feedback the error examples to obtain the second round of iterative results. Repeat multiple iterations until the target model answers correctly or reaches the predetermined iteration limit.

[0014] Self-iterative dataset construction: During the iteration process, all correctly answered questions and the analysis results of the teacher model are summarized to form a self-iterative dataset, which includes question-answer pairs and improvement suggestions output by the teacher model.

[0015] Preferably, when judging preliminary results, the answers are evaluated for accuracy, completeness, and logic.

[0016] Preferably, the model training module is used to perform the following operations:

[0017] Configure multiple reward strategies, including correctness reward strategy, integer reward strategy, strict reward strategy, loose reward strategy and XML counting reward strategy;

[0018] Configure the overall reward calculation strategy: assign weights to the reward functions corresponding to various reward strategies, and construct the overall reward function by weighted summation of the reward functions corresponding to various reward strategies. The overall reward function value is expressed as:

[0019] R=ω c R c +ω i R i +ω s R s +ω f R f +ω x R x,

[0020] Among them, R c represents the reward function corresponding to the correctness reward strategy, ω c Represents R c The weight, R i represents the reward function corresponding to the integer reward strategy, ω i Represents R i The weight, R s represents the reward function corresponding to the strict format reward strategy, ω s Represents R s The weight, R f represents the reward function corresponding to the relaxed format reward strategy, ω f Represents R f The weight, R x represents the reward function corresponding to the XML counting reward strategy, ω x Represents R x The weight of .

[0021] In a second aspect, the present invention provides a self-iterative enterprise large model training method, which is used to optimize and train a large model using a self-iterative enterprise large model training system as described in any one of the first aspects, the method comprising the following steps:

[0022] Self-iterative dataset construction: Build a question-answer corpus based on the target domain corpus and the teacher model. Perform answer analysis on the target model and the teacher model. Build a self-iterative dataset through multiple iterations of answer analysis. The self-iterative dataset includes all questions corresponding to correct answers and the analysis results of incorrect answers.

[0023] Training data construction: constructing a self-iterative dataset into a self-iterative training dataset based on a predetermined format;

[0024] Model training: Configure the reward strategy, train the model based on the reward strategy and self-iterative training dataset using the GRPO training method.

[0025] Preferably, the self-iterative dataset construction includes the following operations:

[0026] Constructing a question-answer corpus: Based on the target domain corpus and the teacher model, a question-answer corpus is generated. The corpus includes a series of representative question-answer pairs that cover common questions and scenarios in enterprise knowledge Q&A.

[0027] Target model execution and evaluation: The questions in the question-answering corpus are used as input questions, and the target model is called to answer the input questions, obtain preliminary results, and evaluate the preliminary results;

[0028] Error example analysis and feedback: For questions answered incorrectly by the target model, the incorrect question and the corresponding incorrect answer are used as input. The teacher model is called to analyze the target model's incorrect answer and generate analysis results. The analysis results indicate the reasons for the incorrect answer and provide improvement suggestions.

[0029] Iteratively optimize the question-answering corpus: Merge the questions answered incorrectly by the target model and the analysis results output by the teacher model to form new questions. Use the new questions as input questions, execute and judge the target model, and analyze and feedback the error examples to obtain the second round of iterative results. Repeat multiple iterations until the target model answers correctly or reaches the predetermined iteration limit.

[0030] Self-iterative dataset construction: During the iteration process, all correctly answered questions and the analysis results of the teacher model are summarized to form a self-iterative dataset, which includes question-answer pairs and improvement suggestions output by the teacher model.

[0031] Preferably, when judging the preliminary results, the self-iterative dataset construction module is used to evaluate the accuracy, completeness and logic of the answers.

[0032] Preferably, model training includes the following operations:

[0033] Configure multiple reward strategies, including correctness reward strategy, integer reward strategy, strict reward strategy, loose reward strategy and XML counting reward strategy;

[0034] Configure the overall reward calculation strategy: assign weights to the reward functions corresponding to various reward strategies, and construct the overall reward function by weighted summation of the reward functions corresponding to various reward strategies. The overall reward function value is expressed as:

[0035] R=ω c R c +ω i R i +ω s R s +ω f R f +ω x R x ,

[0036] Among them, R c represents the reward function corresponding to the correctness reward strategy, ω c Represents R c The weight, R i represents the reward function corresponding to the integer reward strategy, ω i Represents R i The weight, R srepresents the reward function corresponding to the strict format reward strategy, ω s Represents R s The weight, R f represents the reward function corresponding to the relaxed format reward strategy, ω f Represents R f The weight, R x represents the reward function corresponding to the XML counting reward strategy, ω x Represents R x The weight of .

[0037] The self-iterative enterprise large model training system and method of the present invention have the following advantages:

[0038] 1. Significantly improve the model's performance in the enterprise knowledge question-answering domain: Through the construction and training of self-iterative datasets, the target model can optimize question-answer pairs for target domains (such as enterprise knowledge). This effectively addresses the issues of existing large models providing poor answer quality, confusing logic, and even incorrect answers to non-expert users, making the model more accurate when handling instances with simple inputs and complex outputs.

[0039] 2. Effectively reduce the resource cost of enterprise knowledge reinforcement training with extended contexts: Leveraging GRPO (Group Relative Policy Optimization) technology, this solves the high resource and cost issues associated with enterprise knowledge reinforcement training with extended contexts, improving training efficiency and reducing training costs. This enables enterprises to more cost-effectively utilize large models for knowledge management.

[0040] 3. Enhance the model's self-optimization capabilities: During the self-iterative dataset construction process, the target model optimizes based on feedback from the teacher model, gradually improving the quality of its answers. This self-optimization mechanism enables the model to continuously learn and improve, better adapting to the question-answering needs of the target domain.

[0041] 4. Improve the quality and relevance of training data: The self-iterative dataset not only contains high-quality question-and-answer corpus, but also includes analysis and improvements of the teacher model. This provides rich and targeted training materials for subsequent model training, further improving the model's training effect.

[0042] 5. Implement strict control over the model output format: By setting a series of reward strategies, such as correctness reward strategy, integer reward strategy, strict format reward strategy, loose format reward strategy, and XML counting reward strategy, the target model is guided to generate answers that conform to the predefined format, ensuring the standardization and consistency of the model output, which is convenient for subsequent processing and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] The present invention will be further described below with reference to the accompanying drawings.

[0045] Figure 1 This is a flowchart of a self-iterative enterprise large model training method according to Example 2. DETAILED DESCRIPTION

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments given are not intended to limit the present invention. Unless there is a conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0047] The embodiments of the present invention provide a self-iteration-based enterprise big model training system and method, which are used to solve the technical problem that big models perform poorly when facing non-professional users in the field of enterprise knowledge question answering.

[0048] Example 1:

[0049] The present invention provides a large enterprise model training system based on self-iteration, comprising a self-iteration data set construction module, a training data construction module and a model training module.

[0050] The self-iterative dataset construction module is used to build a question-answering corpus based on the target domain corpus and the teacher model, perform answer analysis on questions based on the target model and the teacher model, and construct a self-iterative dataset through multiple iterations of answer analysis. The self-iterative dataset includes all questions corresponding to correct answers and the analysis results of incorrect answers.

[0051] As a concrete implementation of the self-iterative dataset building module, this module is used to perform the following operations:

[0052] (1) Constructing a question-answer corpus: Based on the target domain corpus and the teacher model, a question-answer corpus is generated. The question-answer corpus includes a series of representative question-answer pairs that cover common problems and scenarios in enterprise knowledge question answering.

[0053] (2) Target model execution and evaluation: Take the questions in the question-answering corpus as input questions, call the target model to answer the input questions, obtain preliminary results, and evaluate the preliminary results;

[0054] (3) Error example analysis and feedback: For questions that the target model answers incorrectly, the incorrectly answered question and the corresponding incorrect answer are used as input, and the teacher model is called to analyze the target model's incorrect answer, generate analysis results, and point out the reasons for the incorrect answer and improvement suggestions through the analysis results;

[0055] (4) Iterative optimization of the question-answering corpus: The questions answered incorrectly by the target model and the analysis results output by the teacher model are merged to form new questions. The new questions are used as input questions, and the target model is executed and judged, as well as the error example analysis feedback is performed to obtain the second round of iterative results. The iterations are repeated multiple times until the target model answers correctly or reaches the predetermined iteration limit.

[0056] (5) Self-iterative dataset construction: During the iteration process, all correctly answered questions and the analysis results of the teacher model are summarized to form a self-iterative dataset. The self-iterative dataset includes question-answer pairs and improvement suggestions output by the teacher model. This dataset not only contains high-quality question-answer pairs, but also includes improvement suggestions for the teacher model, providing rich training materials for subsequent model training.

[0057] When judging the preliminary results, the responses were evaluated for accuracy, completeness, and logic.

[0058] The training data construction module is used to construct the self-iterative dataset into a self-iterative training dataset based on a predetermined format.

[0059] This module constructs the self-iterative dataset into a self-iterative training dataset according to a certain format.

[0060] The training data format is as follows:

[0061] [{

[0062] “input”:question_1,

[0063] “output”:“

[0064] <thinking>

[0065] reason_1_1;

[0066] reason_1_2;

[0067] reason_1_3;

[0068] < / thinking>

[0069] answer_1

[0070] ”]

[0071] Among them, the "input" value is the problem in the self-iterative dataset, and the "output" is the analysis label ( <thinking> …< / thinking> ), analysis content (reason_i) and answer composition.

[0072] The above format settings ensure that the target model can output according to the workflow of self-iterative analysis, thereby improving the quality of the answer.

[0073] The model training module is used to configure the reward strategy and train the model based on the reward strategy and self-iterative training dataset using the GRPO training method.

[0074] As a specific implementation of the model training module, this module is used to perform the following operations:

[0075] (1) configuring multiple reward strategies, including correctness reward strategy, integer reward strategy, strict reward strategy, loose reward strategy, and XML counting reward strategy;

[0076] (2) Configure the overall reward calculation strategy: assign weights to the reward functions corresponding to various reward strategies, and construct the overall reward function by weighted summation of the reward functions corresponding to various reward strategies. The overall reward function value is expressed as:

[0077] R=ω c R c +ω i R i +ω s R s +ω f R f +ω x R x ,

[0078] Among them, R c represents the reward function corresponding to the correctness reward strategy, ω c Represents R c The weight, R i represents the reward function corresponding to the integer reward strategy, ω i Represents R i The weight, R s represents the reward function corresponding to the strict format reward strategy, ω s Represents R s The weight, R f represents the reward function corresponding to the relaxed format reward strategy, ω f Represents R f The weight, R x represents the reward function corresponding to the XML counting reward strategy, ω x Represents R x The weight of .

[0079] The model training module is based on the unsloth training framework and self-iterative training dataset. By setting a series of reward strategies, it uses the GRPO training method proposed by DeepSeek to train the model. Compared with traditional PPO and DPO reinforcement learning methods, this method can greatly reduce the resources required for training while ensuring learning stability.

[0080] Correctness reward strategy: By extracting the key content in the answer and comparing it with the standard answer, check whether the answer generated by the model is consistent with the standard answer. If it is consistent, a high reward (2.0) is given; otherwise, a low reward (0.0) is given.

[0081] Integer reward strategy: Use regular expressions to extract the answer content and determine whether it is a number. If it is a number, give a reward (0.5), otherwise give a low reward (0.0).

[0082] Strict format reward strategy: Match the answer content through regular expressions to check whether the answer strictly conforms to the predefined format ( <thinking>reason_i...< / thinking> answer), and gives a reward (0.5) if it fully matches the format, and a low reward (0.0) otherwise.

[0083] Loose format reward strategy: Use regular expressions to match the answer content and check whether the answer meets the loose format requirements ( <thinking>reason_i...< / thinking> answer, allowing for some deviations in format) gives a reward (0.5) if it conforms to the format, and a low reward (0.0) otherwise.

[0084] XML Count Reward Strategy: rewards are given based on the number and position of specific tags in the answer, encouraging answers to conform to a specific structure. For example, <thinking>The number and position of the label will affect the reward value.

[0085] The training parameter table is shown in Table 1.

[0086] Table 1. Training parameter table

[0087]

[0088] The system of this embodiment constructs a self-iterative dataset for the target domain based on the teacher model and uses GRPO (Group Relative Policy Optimization) technology for fine-tuning. The system includes a self-iterative dataset construction module, a training data construction module, and a GRPO training and verification module.

[0089] Example 2:

[0090] A self-iterative large-scale enterprise model training method includes three steps: self-iterative dataset construction, training data construction, and model training.

[0091] Step S100: Self-iterative dataset construction: A question-answer corpus is constructed based on the target domain corpus and the teacher model, and answer analysis of the questions is performed based on the target model and the teacher model. A self-iterative dataset is constructed through multiple iterations of answer analysis, wherein the self-iterative dataset includes all questions corresponding to correct answers and analysis results of incorrect answers.

[0092] As a specific implementation of self-iterative dataset construction, this step is used to perform the following operations:

[0093] (1) Constructing a question-answer corpus: Based on the target domain corpus and the teacher model, a question-answer corpus is generated. The question-answer corpus includes a series of representative question-answer pairs that cover common problems and scenarios in enterprise knowledge question answering.

[0094] (2) Target model execution and evaluation: Take the questions in the question-answering corpus as input questions, call the target model to answer the input questions, obtain preliminary results, and evaluate the preliminary results;

[0095] (3) Error example analysis and feedback: For questions that the target model answers incorrectly, the incorrectly answered question and the corresponding incorrect answer are used as input, and the teacher model is called to analyze the target model's incorrect answer, generate analysis results, and point out the reasons for the incorrect answer and improvement suggestions through the analysis results;

[0096] (4) Iterative optimization of the question-answering corpus: The questions answered incorrectly by the target model and the analysis results output by the teacher model are merged to form new questions. The new questions are used as input questions, and the target model is executed and judged, as well as the error example analysis feedback is performed to obtain the second round of iterative results. The iterations are repeated multiple times until the target model answers correctly or reaches the predetermined iteration limit.

[0097] (5) Self-iterative dataset construction: During the iteration process, all correctly answered questions and the analysis results of the teacher model are summarized to form a self-iterative dataset. The self-iterative dataset includes question-answer pairs and improvement suggestions output by the teacher model. This dataset not only contains high-quality question-answer pairs, but also includes improvement suggestions for the teacher model, providing rich training materials for subsequent model training.

[0098] When judging the preliminary results, the responses were evaluated for accuracy, completeness, and logic.

[0099] Step S200: Training data construction: constructing the self-iterative dataset into a self-iterative training dataset based on a predetermined format.

[0100] This step constructs the self-iterative dataset into a self-iterative training dataset according to a certain format.

[0101] The training data format is as follows:

[0102] [{

[0103] "input":question_1,

[0104] "output”:"

[0105] <thinking>

[0106] reason_1_1;

[0107] reason_1_2;

[0108] reason_1_3;

[0109] < / thinking>

[0110] answer_1

[0111] ”]

[0112] Among them, the "input" value is the problem in the self-iterative dataset, and the "output" is the analysis label ( <thinking> …< / thinking> ), analysis content (reason_i) and answer composition.

[0113] The above format settings ensure that the target model can output according to the workflow of self-iterative analysis, thereby improving the quality of the answer.

[0114] Step S300: Model training: configure the reward strategy, and train the model based on the reward strategy and the self-iterative training data set through the GRPO training method.

[0115] As a specific implementation of model training, this step includes the following operations:

[0116] (1) configuring multiple reward strategies, including correctness reward strategy, integer reward strategy, strict reward strategy, loose reward strategy, and XML counting reward strategy;

[0117] (2) Configure the overall reward calculation strategy: assign weights to the reward functions corresponding to various reward strategies, and construct the overall reward function by weighted summation of the reward functions corresponding to various reward strategies. The overall reward function value is expressed as:

[0118] R=ω c R c +ω i R i +ω s R s +ω f R f +ω x R x ,

[0119] Among them, R c represents the reward function corresponding to the correctness reward strategy, ω c Represents R c The weight, R i represents the reward function corresponding to the integer reward strategy, ω i Represents R i The weight, R s represents the reward function corresponding to the strict format reward strategy, ω s Represents R s The weight, R f represents the reward function corresponding to the relaxed format reward strategy, ω f Represents R f The weight, R x represents the reward function corresponding to the XML counting reward strategy, ω x Represents R x The weight of .

[0120] The model training process is based on the unsloth training framework and self-iterative training dataset. By setting a series of reward strategies, the model is trained using the GRPO training method proposed by DeepSeek. Compared with traditional PPO and DPO reinforcement learning methods, this method can greatly reduce the resources required for training while ensuring learning stability.

[0121] Correctness reward strategy: By extracting the key content in the answer and comparing it with the standard answer, check whether the answer generated by the model is consistent with the standard answer. If it is consistent, a high reward (2.0) is given; otherwise, a low reward (0.0) is given.

[0122] Integer reward strategy: Use regular expressions to extract the answer content and determine whether it is a number. If it is a number, give a reward (0.5), otherwise give a low reward (0.0).

[0123] Strict format reward strategy: Match the answer content through regular expressions to check whether the answer strictly conforms to the predefined format ( <thinking>reason_i...< / thinking> answer), and gives a reward (0.5) if it fully matches the format, and a low reward (0.0) otherwise.

[0124] Loose format reward strategy: Use regular expressions to match the answer content and check whether the answer meets the loose format requirements ( <thinking>reason_i...< / thinking> answer, allowing for some deviations in format) gives a reward (0.5) if it conforms to the format, and a low reward (0.0) otherwise.

[0125] XML Count Reward Strategy: rewards are given based on the number and position of specific tags in the answer, encouraging answers to conform to a specific structure. For example, <thinking>The number and position of the label will affect the reward value.

[0126] The above is a detailed introduction to the self-iterative enterprise large model training system and method provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.< / thinking> < / thinking>

Claims

1. A self-iterative enterprise large model training system, characterized by: Includes self-iterative dataset construction module, training data construction module and model training module; The self-iterative dataset construction module is used to construct a question-answer corpus based on the target domain corpus and the teacher model, perform answer analysis on the questions based on the target model and the teacher model, and construct a self-iterative dataset through multiple iterations of answer analysis, wherein the self-iterative dataset includes all questions corresponding to correct answers and analysis results of incorrect answers; The training data construction module is used to construct the self-iterative dataset into a self-iterative training dataset based on a predetermined format; The model training module is used to configure the reward strategy and train the model through the GRPO training method based on the reward strategy and the self-iterative training data set.

2. The self-iterative enterprise large model training system according to claim 1 is characterized in that: The self-iterative dataset construction module is used to perform the following operations: Constructing a question-answer corpus: Based on the target domain corpus and the teacher model, a question-answer corpus is generated. The corpus includes a series of representative question-answer pairs that cover common questions and scenarios in enterprise knowledge Q&A. Target model execution and evaluation: The questions in the question-answering corpus are used as input questions, and the target model is called to answer the input questions, obtain preliminary results, and evaluate the preliminary results; Error example analysis and feedback: For questions answered incorrectly by the target model, the incorrect question and the corresponding incorrect answer are used as input. The teacher model is called to analyze the target model's incorrect answer and generate analysis results. The analysis results indicate the reasons for the incorrect answer and provide improvement suggestions. Iteratively optimize the question-answering corpus: Merge the questions answered incorrectly by the target model and the analysis results output by the teacher model to form new questions. Use the new questions as input questions, execute and judge the target model, and analyze and feedback the error examples to obtain the second round of iterative results. Repeat multiple iterations until the target model answers correctly or reaches the predetermined iteration limit. Self-iterative dataset construction: During the iteration process, all correctly answered questions and the analysis results of the teacher model are summarized to form a self-iterative dataset, which includes question-answer pairs and improvement suggestions output by the teacher model.

3. The self-iterative enterprise large model training system according to claim 2 is characterized in that: When judging the preliminary results, the responses were evaluated for accuracy, completeness, and logic.

4. The self-iterative enterprise large model training system according to claim 1 or 2, characterized in that: The model training module is used to perform the following operations: Configure multiple reward strategies, including correctness reward strategy, integer reward strategy, strict reward strategy, loose reward strategy and XML counting reward strategy; Configure the overall reward calculation strategy: assign weights to the reward functions corresponding to various reward strategies, and construct the overall reward function by weighted summation of the reward functions corresponding to various reward strategies. The overall reward function value is expressed as: R=ω c R c +oh i R i +oh s R s +oh f R f +oh x R x , Among them, R c represents the reward function corresponding to the correctness reward strategy, ω c Represents R c The weight, R i represents the reward function corresponding to the integer reward strategy, ω i Represents R i The weight, R s represents the reward function corresponding to the strict format reward strategy, ω s Represents R s The weight, R f represents the reward function corresponding to the relaxed format reward strategy, ω f Represents R f The weight, R x represents the reward function corresponding to the XML counting reward strategy, ω x Represents R x The weight of .

5. A self-iterative enterprise large model training method, characterized in that: The method for optimizing and training a large model by using a self-iterative enterprise large model training system according to any one of claims 1 to 4 comprises the following steps: Self-iterative dataset construction: Build a question-answer corpus based on the target domain corpus and the teacher model. Perform answer analysis on the target model and the teacher model. Build a self-iterative dataset through multiple iterations of answer analysis. The self-iterative dataset includes all questions corresponding to correct answers and the analysis results of incorrect answers. Training data construction: constructing a self-iterative dataset into a self-iterative training dataset based on a predetermined format; Model training: Configure the reward strategy, train the model based on the reward strategy and self-iterative training dataset using the GRPO training method.

6. The enterprise large model training method based on self-iteration according to claim 5 is characterized in that: The self-iterative dataset construction includes the following operations: Constructing a question-answer corpus: Based on the target domain corpus and the teacher model, a question-answer corpus is generated. The corpus includes a series of representative question-answer pairs that cover common questions and scenarios in enterprise knowledge Q&A. Target model execution and evaluation: The questions in the question-answering corpus are used as input questions, and the target model is called to answer the input questions, obtain preliminary results, and evaluate the preliminary results; Error example analysis and feedback: For questions answered incorrectly by the target model, the incorrect question and the corresponding incorrect answer are used as input. The teacher model is called to analyze the target model's incorrect answer and generate analysis results. The analysis results indicate the reasons for the incorrect answer and provide improvement suggestions. Iteratively optimize the question-answering corpus: Merge the questions answered incorrectly by the target model and the analysis results output by the teacher model to form new questions. Use the new questions as input questions, execute and judge the target model, and analyze and feedback the error examples to obtain the second round of iterative results. Repeat multiple iterations until the target model answers correctly or reaches the predetermined iteration limit. Self-iterative dataset construction: During the iteration process, all correctly answered questions and the analysis results of the teacher model are summarized to form a self-iterative dataset, which includes question-answer pairs and improvement suggestions output by the teacher model.

7. The enterprise large model training method based on self-iteration according to claim 6 is characterized in that: When judging the preliminary results, the self-iterative dataset construction module is used to evaluate the accuracy, completeness and logic of the answers.

8. The enterprise large model training method based on self-iteration according to claim 5 is characterized in that: Model training includes the following operations: Configure multiple reward strategies, including correctness reward strategy, integer reward strategy, strict reward strategy, loose reward strategy and XML counting reward strategy; Configure the overall reward calculation strategy: assign weights to the reward functions corresponding to various reward strategies, and construct the overall reward function by weighted summation of the reward functions corresponding to various reward strategies. The overall reward function value is expressed as: R=ω c R c +oh i R i +oh s R s +oh f R f +oh x R x , Among them, R c represents the reward function corresponding to the correctness reward strategy, ω c Represents R c The weight, R i represents the reward function corresponding to the integer reward strategy, ω i Represents R i The weight, R s represents the reward function corresponding to the strict format reward strategy, ω s Represents R s The weight, R f represents the reward function corresponding to the relaxed format reward strategy, ω f Represents R f The weight, R x represents the reward function corresponding to the XML counting reward strategy, ω x Represents R x The weight of .

Images