Foreign trade field large model construction method and system

Abstract

The invention discloses a foreign trade field large model construction method. The method comprises the steps of 1) preparing training and test data for fine tuning of a foreign trade field large model with a supervision instruction; 2) data preprocessing: cleaning all the integrated data, and preprocessing the data, including operations of noise removal, format standardization, dialogue segmentation, duplicate removal and the like, so as to ensure the quality and consistency of the data; (3) loading an LLaMA2-7B model, and training a reward model by utilizing part of artificially marked domain data and open-source harmlessness data to obtain a reward model which better conforms to the domain intention and harmlessness; 4) performing reverse cleaning on all instruction following data integrated in the processes of 1) and 2) by using the trained reward model; and 5) a loading instruction follows the fine-tuned large model and the reward model, and a near-end optimization strategy in reinforcement learning is adopted to train the SFT model so as to obtain a large model conforming to the intention of a field user.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and natural language processing, and in particular to a method for constructing a large model in the field of foreign trade. Background Art

[0002] In the field of deep learning and natural language processing, the construction of large models has always been one of the research hotspots. Large models can achieve more accurate and intelligent task processing by learning patterns and knowledge from large-scale data. However, there are some challenges and problems when building large models in specific fields.

[0003] First, data in a specific domain is often limited and unbalanced. For example, in the field of dialogue systems, the domain dialogue data obtained may be limited, and the distribution of dialogue data in different domains may be uneven. This makes it difficult to fully utilize limited data to train an accurate and comprehensive model when building a large domain model.

[0004] Secondly, due to training data issues, the data distribution of general-purpose large models does not conform to the knowledge distribution in certain fields, which makes it difficult for general-purpose large models to complete tasks in certain specific fields.

[0005] CN202410297532.X is a method and system for training a large model in the financial field based on multi-dimensional data evaluation, S1, the large model in the financial field processes historical financial data through supervised learning, and obtains the core concepts and data patterns in the financial field; S2, the large model in the financial field after obtaining the core concepts and data patterns in the financial field is preliminarily predicted; S3, multi-dimensional analysis is performed according to the preliminary prediction results, and multiple evaluation model groups consisting of evaluation models are set according to the multi-dimensional analysis results; S4, the multi-dimensional analysis results are substituted into the evaluation model group for model training, and output as a reward analysis model. The present invention can make more reasonable predictions and decisions in various tasks of financial services through continuous learning and adjustment, and improve its performance in practical applications.

[0006] CN202311088641.2 A method for constructing a large model in a vertical field, the method comprising: determining the structure and form of an instruction data set according to the needs of various industry scenarios in the vertical field; generating an initial instruction data set according to literature with professional descriptions in various industries and performing logical review and data enhancement to form a high-quality instruction data set; training a fine-tuning large model based on the high-quality instruction data set; constructing a hierarchical industry knowledge base and an industry knowledge graph to integrate into the fine-tuning large model; training multiple industry small models based on various industry themes or categories, and constructing an application framework of a fine-tuning large model combined with an industry small model as a large model in the vertical field for semantic understanding and professional content generation. By constructing a high-quality industry data set and an application framework of a fine-tuning large model + multiple professional small models, the accurate introduction of professional knowledge is achieved, and the comprehensive capabilities of the large model in the vertical field are improved.

[0007] CN202410931827.8 discloses a method, device and readable storage medium for fine-tuning a large model in the field of security, which introduces the control concept of control domain into the fine-tuning stage training of the large model, attempts to solve the data difference problem between the general large model pre-training data set and the domain large model fine-tuning data set, and improves the generalization of large model fine-tuning. At the same time, inspired by federated learning, a model parameter fusion mechanism is proposed to expand the fine-tuning data set in the form of parameters, which solves the problem of data set imbalance and improves the robustness of the model output.

[0008] In addition, traditional supervised learning methods are prone to overfitting or underfitting when faced with limited data and domain-specific tasks. This limits the generalization ability and performance of large models. How to improve the performance of the model through optimization methods is a problem that needs to be solved. Summary of the invention

[0009] Therefore, in response to the above problems, the present invention proposes a method and system for constructing a large model in the foreign trade field based on user intention. This method combines supervised instruction fine-tuning with a construction method based on reinforcement learning. By training the reward model with manually labeled partial domain data and human preference data, a preference reward model that is more in line with the domain intention is obtained. After that, the model is trained using a reward model that conforms to the user intention in the foreign trade scenario and conforms to human preferences, and the proximal optimization strategy in reinforcement learning is used to train the model, so that the large model can further fit the intention of users in the foreign trade field, so that the large model's response can meet the user's real needs. This method can solve the problem that the general large model is insufficient in the foreign trade field and the response cannot meet user needs, and improve the performance of the domain large model in vertical fields.

[0010] In summary, the method and system for building a large domain model based on user intent provided by the present invention provide an effective solution to the challenges and problems in building large models in the foreign trade field, and have broad application prospects and promotion value.

[0011] The technical solution of the present invention is a method and system for constructing a domain large model based on user intent, which aims to overcome the challenges and problems in constructing a large model in a specific domain and improve the performance and generalization ability of the model. The specific content of the invention will be described in detail below.

[0012] First, the method for constructing a domain model of the invention comprises the following steps:

[0013] Step 1: Prepare training and test data for supervised instruction fine-tuning of large domain models. The data includes open-source general instruction following data, domain knowledge-intensive documents, domain dialogue data, and human preference data.

[0014] Step 2: Data preprocessing. In this step, all the integrated data is cleaned and preprocessed, including noise removal, standardized format, segmented conversations, deduplication, and sensitive data removal to ensure data quality and consistency. Next, a larger model is used to extract the instruction following data set from knowledge-intensive documents in the foreign trade field, and then merged with the open source instruction following data collected in step 1. Then, some conversation data is selected from the domain conversation data for manual quality scoring, and opposite replies are written for high-quality and low-quality conversations to generate partial order pairs that meet user intentions.

[0015] Step 3: Train the preference reward model. In this step, use the manually labeled partial domain data and human preference data to train the reward preference model. Load the LLaMA2-7B model, use the manually labeled partial domain data and open source harmless data to train the reward model, so as to obtain a reward model that is more in line with the domain intention and harmlessness;

[0016] Step 4: Reverse clean the instruction following data. That is, pass the instruction following data of steps one and two through the reward model, obtain the score of the reward model, and sort the data based on the score. By selecting instruction following data with a score greater than 0.6, the high quality of the instruction following data is ensured. Use the trained reward model to reverse clean all the instruction following data integrated in the process of step one, and select the instruction following data with the top 60% score to perform supervised instruction fine-tuning on the LLaMA2-13B / 70B model;

[0017] Step 5: Load the fine-tuned large model and reward model, and use the proximal optimization strategy in reinforcement learning to train the SFT model to obtain a large model that meets the user intent in the domain. Reinforcement learning fine-tuning based on user intent feedback. In this step, load the fine-tuned model and reward model with supervised instructions, and use the proximal optimization strategy (PPO algorithm) in reinforcement learning to train the fine-tuned model with supervised instructions.

[0018] In the step one, collect open source general instruction following data, which contains instructions and corresponding execution results; collect knowledge documents in the foreign trade field, which contain dense vertical knowledge in the foreign trade field; these data come from public code bases, open source databases, and dense knowledge documents in the foreign trade industry, and are used to train the large model to understand and execute in general fields. Collect domain dialogue data, which contains dialogue records in the foreign trade field; these dialogues include both interactive dialogues between users and the system and discussion dialogues between experts about foreign trade, which are used to train the large model to better understand context and domain intent in the foreign trade field. Collect human preference data, which comes from open source databases and foreign trade field dialogue data, and is only used to train reward models, which are used to reduce the toxicity and harmfulness of the model during reinforcement learning fine-tuning training based on user intent feedback. In the step one, preparing training and test data is a key step in building a large domain model.

[0019] Instruction following data refers to instruction following data extracted from public code bases, open source databases, and dense knowledge documents in the foreign trade industry in general fields. Domain dialogue data refers to interactive dialogues between users and systems and discussions between experts in a specific field. Human preference data refers to data containing two categories, harmfulness and usefulness, which constitute a partial order pair of usefulness and harmfulness for a prompt.

[0020] In the step 2, all the integrated data are cleaned and preprocessed, including noise removal, format standardization, conversation segmentation, and deduplication, to ensure data quality and consistency. Next, a larger model is used to extract the instruction following data set from knowledge-intensive documents in the foreign trade field, and then merged with the open source instruction following data collected in the first step. 40,000 data are extracted from the open source human preference data set, and then 20,000 conversation data are randomly selected from the domain data. For the high-quality and low-quality conversations in the extracted domain data, the opposite responses are written for the user's questions, thereby generating a partial order pair that meets the user's intention.

[0021] In the step three, manually labeled partial domain data and human preference data are used to train the reward model.

[0022] In step 4, all the instruction following data integrated in steps 1 and 2 are reverse cleaned using the trained reward model. That is, the data in step 1 is passed through the reward model to obtain the score of the reward model, and the data is sorted based on the score. The high quality of the instruction following data is ensured by selecting instruction following data with a score greater than 0.6. This is then used for supervised instruction fine-tuning of the large model to further stimulate the capabilities of the large model.

[0023] In step 5, the fine-tuned large model and reward model are loaded, and the proximal optimization strategy (PPO algorithm) in reinforcement learning is used to further fine-tune them. In this way, the performance of the model can be further improved to make it more in line with the intentions of domain users, while reducing the toxicity and illusion of the model.

[0024] Beneficial effects: The present invention discloses a construction method of a domain large model with supervised instruction fine-tuning (SFT) based on user intention plus reinforcement learning based on user feedback, which is used to supplement the knowledge or ability of the general large model in the field of foreign trade. The present invention can obtain a preference reward model that is more in line with the domain intention, non-toxic and hallucinatory, and is convenient for subsequent data cleaning and reinforcement learning fine-tuning based on user feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The present invention is a flowchart of the method for constructing a large domain model. DETAILED DESCRIPTION

[0026] The present invention will be further described below in conjunction with the accompanying drawings and exemplary embodiments:

[0027] like Figure 1 As shown, the present invention discloses a method for constructing a domain big model, 1) preparing training and test data for supervised instruction fine-tuning of the domain big model, the data including general domain instruction following data, domain knowledge-intensive documents, domain dialogue data and human preference data; 2) data preprocessing, cleaning all integrated data, formatting and deduplication, extracting instruction following data sets from knowledge-intensive documents in the foreign trade field, and then selecting part of the domain dialogue data for manual labeling; 3) loading the LLaMA2-7B model, using part of the foreign trade field data that has been manually labeled and open source harmless data to train the reward model, so as to obtain a reward model that is more in line with the domain intention and harmlessness; 4) using the trained reward model to reversely clean all the instruction following data integrated in the process of 1), and selecting the instruction following data with a reward model score greater than 0.6 to perform supervised instruction fine-tuning on the LLaMA2-13B / 70B model; 5) loading the supervised instruction fine-tuning model and the reward model, and using the proximal optimization strategy in reinforcement learning to train the big model, so as to obtain a big model that meets the domain user intention.

[0028] Specifically include:

[0029] Step 11: Prepare data sets, collect open source instruction following data, domain conversation data, domain knowledge-intensive documents, and human preference data sets. Instruction following data can be obtained from public conversation data sets or data sources related to forums, social media, etc. Domain conversation data can be collected through communication with domain experts, user surveys, or data collection tools. Human preference data can be text data related to a specific domain collected from the Internet, such as domain expertise, normative documents, and some open source human preference data.

[0030] Step 12: Clean all the integrated data and pre-process the data, including removing noise, standardizing the format, segmenting the conversation, deduplication, etc., to ensure the quality and consistency of the data. Next, extract the instruction following data set from the knowledge-intensive documents in the foreign trade field with the help of a larger model, and merge it with the open source instruction following data collected in the first step. Extract 40,000 data from the open source human preference data set, and then randomly select 20,000 conversation data from the domain data, and write opposite replies to the user's questions for the high-quality and poor-quality conversations in the extracted domain data, thereby generating a partial order pair that meets the user's intention.

[0031] Step 13: Use manually annotated partial domain data and human preference data to train LLaMA2-7B to make it a preference reward model. The reward model uses LLaMA2-7B. Since a large reward model will lead to high training costs, and a small training data will make it difficult to converge the loss value, the RM model uses a 7B parameter model. Modify the model architecture of LLaMA2-7B, set the output layer dimension of the last LM-Head layer to 1, and use Pair-Wise Loss for training. The formula is as follows, where r θ (x, y) is the reward score of the reward model for the prompt word x and the model response y under the parameter θ, D is our preferred dataset, and σ is the activation function.

[0032]

[0033] During the training process, the last layer is zero-normalized to prevent gradient explosion during the initial training process, and the first four layers of LLaMA2-7B are frozen to ensure that the shallow semantics of the text extracted by the model remain unchanged, and only the deep semantics behind are trained. By training the reward model, a preference reward model that is more in line with the domain intent, non-toxic and hallucinatory can be obtained, which is convenient for subsequent data cleaning and reinforcement learning fine-tuning based on user feedback.

[0034] Step 14: Back-clean the instruction following data: Use the trained reward model to back-clean all the instruction following data integrated in the step. According to the score of the reward model, the instruction following data with a reward model score greater than 0.6 is used for the supervised instruction fine-tuning of the subsequent model.

[0035] Step 15: Load the fine-tuned model and reward model, and use the PPO algorithm in reinforcement learning to optimize and adjust the fine-tuned model to improve the performance and generalization ability of the model.

[0036] The present invention is mainly used to provide a method for constructing a domain big model based on a general big model and domain data. The model can extract key domain knowledge and patterns from limited domain data through supervised instruction fine-tuning and model optimization based on reinforcement learning, thereby achieving more accurate and intelligent task processing. Through reasonable data preparation, model training and optimization, an intelligent dialogue system that can meet the needs of specific fields can be constructed to provide users with a more accurate and personalized dialogue interaction experience. The above embodiments do not limit the present invention in any way, and any other improvements and applications made to the above embodiments in an equivalent transformation manner are within the scope of protection of the present invention.

Claims

1. A method for constructing a large model in the field of foreign trade, characterized in that: The following steps are involved: Step 1: Prepare training and testing data for supervised instruction fine-tuning of large models in the foreign trade field. The data includes open-source general instruction following data, domain knowledge-intensive documents, domain dialogue data, and human preference data. Step 2: Data preprocessing: clean all the integrated data and preprocess the data, including removing noise, standardizing the format, segmenting conversations, removing duplicates, etc., to ensure the quality and consistency of the data; Next, we extract the instruction-following dataset from knowledge-intensive documents in the foreign trade field with the help of a larger model, and merge it with the open-source instruction-following data collected in the first step. Then, we select some human preference data and domain conversation data, manually mark and write the domain conversation data to form partial order pairs of user intentions, and then merge them with the human preference data. Step 3: Load the LLaMA2-7B model and use the manually labeled partial domain data and open-source harmless data to train the reward model to obtain a reward model that is more in line with domain intent and harmlessness; Step 4: Use the trained reward model to reverse clean all the command following data integrated in steps 1 and 2, and select the command following data with a reward score greater than 0.6 to perform supervised command fine-tuning on the LLaMA2-13B / 70B model; Step 5: Load the instruction-following fine-tuned large model and reward model, and use the proximal optimization strategy in reinforcement learning to train the SFT model to obtain a large model that meets the domain user's intentions.

2. The method for constructing a large domain model according to claim 1, characterized in that: In the step 1, open source general instruction following data and private instruction following data sets are collected, which contain instructions and corresponding execution results; these data come from public code bases, open source databases, and dense knowledge documents from the foreign trade industry, and are used to train the large model's understanding and execution capabilities in general fields; Collect domain dialogue data, which includes dialogue records in the foreign trade field. These dialogues include both interactive dialogues between users and the system and discussions between experts. They are used to train large models to better understand context and domain intent in the foreign trade field. Collect human preference data from open source databases and use them only to train reward models to reduce the toxicity and harmfulness of the model during reinforcement learning fine-tuning training based on user intent feedback.

3. The method for constructing a large domain model according to claim 2, characterized in that: In step one, instruction following data refers to instruction following data extracted from public code bases in general fields, open source databases, and dense knowledge documents in the foreign trade industry; domain dialogue data refers to interactive dialogues between users and systems and discussions between experts in a specific field; human preference data refers to data including two categories, harmfulness and usefulness, which constitute a partial order pair of usefulness and harmfulness for a prompt.

4. The method for constructing a large domain model according to claim 1, characterized in that: In the step 2, all the integrated data are first cleaned and preprocessed, including noise removal, format standardization, conversation segmentation, and duplication removal, to ensure data quality and consistency; Next, we extract the instruction following data set from knowledge-intensive documents in the foreign trade field with the help of a larger model, and merge it with the open source instruction following data collected in step one. Then, we select some of the collected domain dialogue data for manual annotation to form user partial order pair data, which reflects the user's satisfaction with the reply and is used for the training of the subsequent reward model.

5. The method for constructing a large domain model according to claim 4, characterized in that: In the step 2, 40,000 data are extracted from the open source human preference dataset, and then 20,000 conversation data are randomly selected from the domain data. For the high-quality and low-quality conversations in the extracted domain data, opposite replies are written for the user's questions, thereby generating a partial order pair that meets the user's intention.

6. The method for constructing a large domain model according to claim 1, characterized in that: In the step 3, the preference reward model is trained using manually labeled partial domain data and human preference data; the reward model uses LLaMA2-7B. Since a large reward model will lead to high training costs, and small training data will make it difficult to converge the loss value, the RM model uses a 7B parameter model; the model architecture of LLaMA2-7B is modified, the output layer dimension of the last LM-Head layer is set to 1, and Pair-Wise Loss is used for training. The formula is as follows, where r θ (x, y) is the reward score of the reward model for the prompt word x and the model response y under the parameter θ, D is our preferred dataset, and σ is the activation function; During the training process, the last layer is zero-normalized to prevent gradient explosion during the initial training process, and the first four layers of LLaMA2-7B are frozen to ensure that the shallow semantics of the text extracted by the model remains unchanged, and only the subsequent deep semantics are trained. By training the reward model, a preference reward model that is more in line with the domain intent, non-toxic and hallucinatory can be obtained, which is convenient for subsequent data cleaning and reinforcement learning fine-tuning based on user feedback.

7. The method for constructing a large domain model according to claim 1, characterized in that: Use the trained reward model to reverse clean all the instruction following data integrated in steps one and two; that is, pass the data in step one through the reward model to obtain the score of the reward model, and sort the data based on the score; select the instruction following data with a score greater than 0.6, and select the data with a reward score greater than 0.6 to perform supervised instruction fine-tuning on the LLaMA2-13B / 70B model.

8. The method for constructing a large domain model according to claim 1, characterized in that: In step five, the algorithm used in the reinforcement learning is the proximal optimization strategy (PPO), which is loaded with a model fine-tuned by supervised instructions and a reward model, and the PPO algorithm is used to perform reinforcement learning fine-tuning on the large model based on user intention feedback to obtain a large model that meets the domain user intention and is harmless.

Images