Construction and optimization method of question-answering system adopting river crab knowledge question-answering large model Crab GPT

Abstract

The invention relates to a method for constructing and optimizing a question-answering system by adopting a river crab knowledge question-answering large model Crab GPT. The method comprises the following steps: constructing the question-answering system of the river crab knowledge question-answering large model Crab GPT, and optimizing the river crab knowledge question-answering large model Crab GPT. The beneficial effects of the invention are that deep integration of the local knowledge base of river crab culture is realized; a more efficient and reliable solution is provided for knowledge management and service of the river crab industry, and the user experience of using the river crab knowledge question and answer large model Crab GPT is continuously improved; through combination of SFT supervision fine tuning and a human feedback reinforcement learning method RLHF, data from different types of river crab practitioners and feedback information of real users are fully utilized, and deep optimization is successfully performed on a river crab knowledge question and answer large model Crab GPT. According to the method, the performance of the river crab knowledge question and answer large model Crab GPT on specific tasks is improved, and the satisfaction degree and experience feeling of final users are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the cross-technical field of information technology and agricultural crab breeding, and in particular relates to a method for constructing and optimizing a question-answering system using a crab knowledge question-answering large model CrabGPT. Background Art

[0002] Existing river crab farming knowledge question-and-answer systems usually use natural language processing technology to parse user queries and provide corresponding answers or suggestions; however, these existing river crab farming knowledge question-and-answer systems have the following defects:

[0003] 1. Existing technical solutions often make it difficult to effectively integrate the local knowledge base of river crab farming, making it difficult to put the local knowledge base of river crab farming into practical use. In order to integrate the knowledge base of river crab farming professional knowledge, the existing technology uses langchain+Faiss for deep integration. The knowledge base integrated in this way can query the knowledge documents in the relevant river crab field, but this method requires temporary query of files and real-time model summary generation, so it is necessary to temporarily integrate the knowledge base information, which is prone to inaccurate industry knowledge.

[0004] 2. The existing river crab farming knowledge question and answer system often lacks a deep understanding of the professional knowledge of river crab farming and cannot accurately grasp the specific needs of farmers.

[0005] 3. With the development of the industry, new information and knowledge continue to emerge; however, the update speed of the local knowledge base of river crab farming often cannot keep up with the speed of emergence of information and knowledge, and the updating and maintenance of the knowledge base is relatively difficult, resulting in the local knowledge base of river crab farming gradually becoming outdated; the knowledge base of the existing river crab farming knowledge question and answer system often cannot reflect the latest technological advances and market trends in a timely manner, and cannot bring a good experience to users (newcomers, experts, or merchants) when conducting knowledge questions and answers.

[0006] In summary, although the integration and updating of the existing river crab farming knowledge question and answer system and the river crab farming local knowledge base have met some of the needs of farmers to a certain extent, there are still obvious defects in the deep integration and maintenance of the knowledge base. Summary of the invention

[0007] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a method for constructing and optimizing a question-answering system using a large crab knowledge question-answering model CrabGPT.

[0008] A method for constructing a question-answering system using a CrabGPT model, comprising the following steps:

[0009] Step 1: Fine-tune the Qwen model to obtain the CrabGPT model. The CrabGPT model includes a word embedding layer, three root mean square normalization layers, a self-attention module, a feed-forward layer with a SwiGLU activation function, a linear layer, and a Softmax layer.

[0010] Step 2: Use the crab knowledge question-answering model CrabGPT to build a crab farming knowledge question-answering system.

[0011] As a preferred method, in the crab knowledge question answering model CrabGPT obtained in step 1:

[0012] The output of the word embedding layer enters the first RMS normalization layer for processing;

[0013] The processing results of the first RMS normalization layer (query, key and value) enter the self-attention module and use the grouped query attention method GQA to perform attention query;

[0014] Perform a logical AND operation on the attention queried from the self-attention module and the output of the word embedding layer;

[0015] The logical AND operation results of the attention and word embedding layers queried from the self-attention module are input into the second RMS normalization layer for processing;

[0016] The processing result of the second RMS normalization layer is input into the feed-forward layer with SwiGLU activation function for processing;

[0017] The processing result of the feedforward layer of the SwiGLU activation function is logically ANDed with the attention queried by the self-attention module to obtain the RoPE position encoding;

[0018] The RoPE position code is input into the third RMS normalization layer for processing;

[0019] The result of the third RMS normalization layer is input into the linear layer for processing;

[0020] The output of the linear layer enters the Softmax layer for processing;

[0021] The output of the Softmax layer is used as the output of the CrabGPT knowledge question answering model.

[0022] A method for optimizing a CrabGPT model of knowledge question answering constructed by a method for constructing a question answering system using a CrabGPT model of knowledge question answering, comprising the following steps:

[0023] Step 1: Scenario case mining: First, select data in fields closely related to the river crab industry as training materials to obtain question-answering benchmark data;

[0024] Step 2: Based on the question-answering benchmark data obtained from scenario case mining, perform SFT supervised fine-tuning on the CrabGPT knowledge question-answering model;

[0025] Step 3: Collect feedback from actual users: Invite different types of Hexie practitioners to participate in the test of the question-answering system using the Hexie knowledge question-answering model CrabGPT. Hexie practitioners use the question-answering system to interact with each other and give positive or negative comments on the question-answering system.

[0026] Step 4: Use the human feedback reinforcement learning method RLHF to analyze and process the collected user feedback information. The CrabGPT knowledge question answering model uses the results of real-time analysis and processing as real-time feedback, continuously adjusts its output strategy, improves the accuracy of the CrabGPT knowledge question answering model, and combines the DPO algorithm to perform style alignment on the large model.

[0027] Preferably, step 2 specifically comprises the following steps:

[0028] Step 2.1: Data cleaning of professional data documents in the question-answer benchmark data obtained by scenario case mining;

[0029] Step 2.2, generate questions based on professional data documents;

[0030] Step 2.3: Generate answers based on professional data documents and questions, organize them into question-answer pairs, and use the Kimi large model to form a set of supervised fine-tuning instructions;

[0031] Step 2.4: Use these fine-tuning instruction sets to perform SFT supervised fine-tuning on the CrabGPT knowledge question answering model.

[0032] Preferably, the SFT supervised fine-tuning in step 2.4 specifically includes the following steps:

[0033] Step 2.4.1, initialization stage: using Gaussian distribution A = N (0, σ 2 ) Initialize the weight parameters of matrix A and set matrix B to zero matrix;

[0034] Step 2.4.2, add additional matrices A and B to the CrabGPT model, and calculate the value of the output feature h of the hidden layer:

[0035] h=Wx+BAx

[0036] In the above formula, h is the output feature of the hidden layer, W is the weight matrix in the large model CrabGPT, and x represents the input feature.

[0037] Step 2.4.3, add the trained matrix product BA to the original weight matrix W:

[0038] h=Wx+BAx=(W+BA)x

[0039] Step 2.4.4, form a new weight parameter W+BA, and use the new weight parameter W+BA to replace the original weight matrix W in the large model CrabGPT.

[0040] Preferably, the value of σ is set to 0.01.

[0041] Preferably, step 4 specifically comprises the following steps:

[0042] Step 4.1: Improve the accuracy of the CrabGPT knowledge question answering model:

[0043] The user asks a question and the large model CrabGPT obtains the corresponding answer. If the user is satisfied, it is marked as the model's chosen answer (preferred answer, y w ), and if the user is not satisfied, it is marked as rejected by the model (non-preferred answer, yl); according to the model, chosen (preferred answer, y w ) and the model rejects (non-preferred answers, y l ) is used to optimize the CrabGPT model for river crab knowledge question answering. The optimization goal is to maximize the value of the function in the following formula:

[0044]

[0045] In the above formula, π θ is the current strategy, π ref is the reference strategy; L DPO (π θ ; π ref ) is the loss function in the DPO algorithm, which is used to measure the current strategy π θ With reference strategy π ref The difference between E(x, y w ,y l )~D is the expectation operator, which means that according to all the samples (x, y) in the data distribution D w ,y l ) to calculate the expected value of the internal expression, where x represents the state and y w and l Represent positive samples and negative samples respectively, which are used to compare the performance of the current strategy and the reference strategy; Refers to the cumulative probability difference between the generated preferred answer and the initial generated preferred answer, represents the cumulative probability difference between the generated non-preferred answer and the initial generated non-preferred answer, and σ is the σ value in the Gaussian distribution;

[0046] After optimizing the CrabGPT model, the model has a better understanding of chosen (preferred answer, y w ) and the model rejects (non-preferred answers, y l ) is the largest, the reward of the CrabGPT model for input data is maximized, and then the human preference data is learned;

[0047] Step 4.2: Use the DPO algorithm combined with the provided human preference data to perform style alignment on the large model.

[0048] The beneficial effects of the present invention are:

[0049] The method of the present invention realizes the deep integration of the local knowledge base of river crab farming by methods such as targeted combing of industry question and answer benchmarks and targeted alignment of user styles; it provides a more efficient and reliable solution for knowledge management and services in the river crab industry, and continuously improves the user experience of using the river crab knowledge question and answer large model CrabGPT.

[0050] By combining SFT supervised fine-tuning and human feedback reinforcement learning method RLHF, the present invention makes full use of data from different types of river crab practitioners (river crab farming novices, river crab farming experts and river crab head merchants) and feedback information from real users, and successfully optimizes the river crab knowledge question and answer model CrabGPT in depth. This not only improves the performance of the river crab knowledge question and answer model CrabGPT on specific tasks, but also significantly improves the satisfaction and experience of end users. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a logical structure diagram of the crab knowledge question-answering model CrabGPT of the present invention;

[0052] Figure 2 Flowchart for supervised fine-tuning of Lora-based SFT;

[0053] Figure 3 It is a schematic diagram of an optimization method of the crab knowledge question-answering model CrabGPT of the present invention;

[0054] Figure 4 This is a comparison chart of the response effects of the crab knowledge question-and-answer model CrabGPT of the present invention and the domestic open source model "Baichuan" on questions related to crab farming and legal knowledge;

[0055] Figure 5This is a comparison chart of the response effects of the crab knowledge question-and-answer model CrabGPT of the present invention and the domestic open source model "Baichuan" on issues related to crab environmental management and feed management. DETAILED DESCRIPTION

[0056] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0057] Example 1

[0058] like Figure 1 As shown, this embodiment relates to a method for constructing a crab knowledge question-answering system. This embodiment uses a large model base such as a domestic large model such as Qwen, and fine-tunes the Qwen large model to obtain the crab knowledge question-answering large model CrabGPT of the present invention;

[0059] The CrabGPT model for Crab knowledge question answering includes: word embedding layer, three RMS normalization layers, self-attention module, feedforward layer with SwiGLU activation function, linear layer and Softmax layer; the output of the word embedding layer enters the first RMS normalization layer for processing; the processing result of the first RMS normalization layer (query, key and value) enters the self-attention module to perform attention query using the grouped query attention method GQA; the attention queried by the self-attention module is logically ANDed with the output of the word embedding layer; the attention queried by the self-attention module is logically ANDed with the output of the word embedding layer The results of the logic and operation are input into the second RMS normalization layer for processing; the processing results of the second RMS normalization layer are input into the feedforward layer with SwiGLU activation function for processing; the processing results of the feedforward layer with SwiGLU activation function are logically ANDed with the attention queried by the self-attention module to obtain the RoPE position code; the RoPE position code is input into the third RMS normalization layer for processing; the result of the third RMS normalization layer is input into the linear layer for processing; the output result of the linear layer enters the Softmax layer for processing; the output of the Softmax layer is used as the output of the CrabGPT model of Crab knowledge question and answering.

[0060] A river crab farming knowledge question-answering system was built using the river crab knowledge question-answering model CrabGPT.

[0061] Example 2

[0062] A method for optimizing the crab knowledge question-answering model CrabGP in Example 1, such as Figure 3 As shown, the following steps are included:

[0063] Step 1: Scenario Case Mining:

[0064] The CrabQA dataset was expanded to construct a benchmark question-and-answer dataset CrabQA-SFT in the field of river crabs. The CrabQA-SFT dataset includes not only 1,000 objective questions from the river crab farming professional qualification exam, the aquaculture physician professional certificate exam, the agricultural aquaculture department management certification exam, and the practicing veterinary qualification certificate (aquatic biology) exam, but also 500 subjective questions collected from the river crab industry question-and-answer benchmark, the river crab professional qualification exam benchmark, and the river crab commercial application field. In addition, it also includes 10,000 question-and-answer system response content, user type, and corresponding feedback tags;

[0065] The CrabQA-SFT dataset was divided into six categories of questions, including farm management, seed selection, feed management, disease prevention and control, breeding management, and aquaculture laws and regulations. These questions were used as training materials to obtain benchmark data for question answering. These data covered more than 1,000 objective questions and 500 subjective questions, providing a rich knowledge background and practical scenarios for the model.

[0066] Step 2: Figure 2 As shown in the figure, based on the question-answering benchmark data obtained from scenario case mining, the CrabGPT knowledge question-answering model is fine-tuned by SFT supervision;

[0067] Step 2.1: Data cleaning of professional data documents in the question-answer benchmark data obtained by scenario case mining;

[0068] Step 2.2, generate questions based on professional data documents;

[0069] Step 2.3: Generate answers based on professional data documents and questions, and organize them into question-answer pairs. Use the Kimi large model and the long text processing capability of the Kimi large model to form a set of supervised fine-tuning instructions (a set of question-answer pairs with more than 200 pairs of supervised fine-tuning instructions).

[0070] Step 2.4: Through these fine-tuning instruction sets, the CrabGPT model is fine-tuned by SFT supervision (LoRA fine-tuning); the purpose is to adjust the parameters of the CrabGPT model through a small number of labeled samples to make it better suited to specific tasks or fields; due to the use of SFT supervised fine-tuning, the accuracy and generalization ability of the CrabGPT model can be effectively improved under limited resources;

[0071] SFT supervised fine-tuning includes the following process:

[0072] 1) Initialization stage: Using Gaussian distribution A=N(0,σ 2 ) Initialize the weight parameters of matrix A and set matrix B to zero matrix to ensure that the output of the newly added pathway BA is zero at the beginning of training, so as to ensure that it does not affect the initial prediction results of the model; the value of σ is generally set to 0.01;

[0073] 2) Add additional matrices A and B to the CrabGPT model to calculate the value of the output feature h of the hidden layer:

[0074] h=Wx+BAx

[0075] In the above formula, h is the output feature of the hidden layer, W is the weight matrix in the large model CrabGPT, x represents the input feature, and x refers to the input text data, which is usually organized into a tensor of batchsize*seq_len*dim, where batchsize can be adjusted by the trainer, and seq_len and dim are usually fixed values; in this embodiment, seq_len and dim are both 4096, and batchsize is 1, that is, the tensor is 1*4096*4096;

[0076] 3) Add the trained matrix product BA to the original weight matrix W:

[0077] h=Wx+BAx=(W+BA)x

[0078] 4) Form a new weight parameter W+BA, and use the new weight parameter W+BA to replace the original weight matrix W in the large model CrabGPT; the above calculation process will not bring additional calculation burden;

[0079] Step 3, collecting feedback information from actual users: The present invention invites more than 100 different types of river crab practitioners (river crab farming novices, river crab farming experts and river crab head merchants) to participate in the question-answering system test using the river crab knowledge question-answering model CrabGPT. River crab practitioners not only use the question-answering system for daily question-answering interactions, but also give positive or negative comments on the question-answering system based on their own experience; through these real user feedback, more than 10,000 groups of question-answering data and the classification and evaluation of the corresponding users are collected; the river crab industry employment needs (for river crab farming novices, river crab farming experts and river crab head merchants) are solved;

[0080] Step 4: Use the human feedback reinforcement learning method RLHF to analyze and process the collected user feedback information. The CrabGPT large model uses the results of real-time analysis and processing as real-time feedback, continuously adjusts its output strategy, improves the accuracy of the CrabGPT large model, and combines the DPO algorithm to align the style of the large model;

[0081] Step 4.1: Improve the accuracy of CrabGPT, a large model for answering questions: Users give questions, and the large model CrabGPT obtains the corresponding answers. If the user is satisfied, it is marked as chosen by the model (preferred answer, y w ), and if the user is not satisfied, it is marked as rejected by the model (non-preferred answer, yl); according to the model, chosen (preferred answer, y w ) and the model rejects (non-preferred answers, y l ) to optimize the CrabGPT model for Crab knowledge question answering. l The reward for the input data is greater than the preferred answer y w For the reward of input data, the larger the gradient of the loss function, the greater the gradient of the loss function will be, and the more preferred answer y will be generated. w and reduce the probability of generating non-preferred answers y l The optimization goal is to maximize the value of the function in the following formula:

[0082]

[0083] In the above formula, π θ is the current strategy, π ref is the reference strategy; L DPO (π θ ; π ref ) is the loss function in the DPO algorithm, which is used to measure the current strategy π θ With reference strategy π ref The difference between E(x,y w ,y l )~D is the expectation operator, which means that according to all the samples (x,y) in the data distribution D w ,y l ) to calculate the expected value of the internal expression, where x represents the state and y w and l Represent positive samples and negative samples respectively, which are used to compare the performance of the current strategy and the reference strategy; Refers to the cumulative probability difference between the generated preferred answer and the initial generated preferred answer, represents the cumulative probability difference between the generated non-preferred answer and the initial generated non-preferred answer, and σ is the σ value in the Gaussian distribution;

[0084] After optimizing the CrabGPT model, the model has a better understanding of chosen (preferred answer, y w ) and the model rejects (non-preferred answers, y l ) is the largest, the reward of the CrabGPT model for input data is maximized, and then the human preference data is learned;

[0085] Step 4.2: Use the DPO algorithm to merge the reward model and reinforcement learning steps of the CrabGPT knowledge question-answering model, and combine the provided human preference data to perform style alignment on the large model.

[0086] Improve the performance of CrabGPT, a large knowledge question-answering model; enable CrabGPT, a large knowledge question-answering model, to more accurately capture user needs and preferences, achieve higher user satisfaction, and optimize user experience.

[0087] This embodiment conducts comparative experiments on multiple models on the test data set CrabQA-SFT for subjective and objective questions, and quantitatively compares the performance of the general artificial intelligence large model Qwen1.5, the open source large model LLaMA-3-7B, and Alpaca-7B, and the crab knowledge question and answer large model CrabGPT (free knowledge base method) of this embodiment. GPT4 is mainly used as an evaluator to score the answers of different large models in subjective and objective questions (scoring from 0 to 10 in terms of accuracy, clarity and completeness, the higher the better). Table 1 below includes the scoring of the results of replying to the following multiple different large models using GPT4.

[0088] Table 1 Performance comparison of CrabGPT and general large models in answering subjective and objective questions in the Crab domain

[0089]

[0090]

[0091] The experimental results show that within the Crab domain, the CrabGPT question-answering system, which uses supervised fine-tuning SFT and human feedback reinforcement learning method RLHF to perform user style alignment on the Crab knowledge question-answering large model, has improved the accuracy, clarity, and completeness of objective question-answering tasks in various fields by about 10% to 20%. The performance improvement of the CrabGPT large model of Crab knowledge question-answering on specific tasks mostly exceeds that of mainstream chatGPT and meta open source LLaMA-2 and other open source general-purpose large models, verifying the effectiveness of the model optimization method of this embodiment.

[0092] In addition, if Figure 4 and Figure 5 As shown, the crab knowledge question-and-answer model CrabGPT constructed by the present invention is compared with the domestic open source large model "Baichuan" in terms of response effects on issues related to crab environmental management and feed management, and on issues related to crab farming and legal knowledge. From the comparison results, it can be seen that CrabGPT shows richer response dimensions and more information sources, such as the specific sources of relevant national laws.

Claims

1. A method for constructing a question-answering system using the CrabGPT large-scale question-answering model, characterized in that: The following steps are involved: Step 1: Fine-tune the Qwen model to obtain the CrabGPT model; CrabGPT, the Crab Knowledge Question Answering model, includes: word embedding layer, three RMS normalization layers, self-attention module, feed-forward layer with SwiGLU activation function, linear layer and Softmax layer; Step 2: Use the crab knowledge question-answering model CrabGPT to build a crab farming knowledge question-answering system.

2. The method for constructing a question-answering system using the CrabGPT large-scale question-answering model according to claim 1 is characterized in that: In the CrabGPT model of the Crab knowledge question answering obtained in step 1: The output of the word embedding layer enters the first RMS normalization layer for processing; The processing result of the first RMS normalization layer enters the self-attention module and uses the group query attention method GQA to perform attention query; Perform a logical AND operation on the attention queried from the self-attention module and the output of the word embedding layer; The logical AND operation results of the attention and word embedding layers queried from the self-attention module are input into the second RMS normalization layer for processing; The processing result of the second RMS normalization layer is input into the feed-forward layer with SwiGLU activation function for processing; The processing result of the feedforward layer of the SwiGLU activation function is logically ANDed with the attention queried by the self-attention module to obtain the RoPE position encoding; The RoPE position code is input into the third RMS normalization layer for processing; The result of the third RMS normalization layer is input into the linear layer for processing; The output of the linear layer enters the Softmax layer for processing; The output of the Softmax layer is used as the output of the CrabGPT knowledge question answering model.

3. A method for optimizing the crab knowledge question-answering model CrabGPT constructed by the method according to any one of claims 1 or 2, characterized in that: The steps include: Step 1: Scenario case mining: First, select data in fields closely related to the river crab industry as training materials to obtain question-answering benchmark data; Step 2: Based on the question-answering benchmark data obtained from scenario case mining, perform SFT supervised fine-tuning on the CrabGPT knowledge question-answering model; Step 3: Collect feedback from actual users: Invite different types of Hexie practitioners to participate in the test of the question-answering system using the Hexie knowledge question-answering model CrabGPT. Hexie practitioners use the question-answering system to interact with each other and give positive or negative comments on the question-answering system. Step 4: Use the human feedback reinforcement learning method RLHF to analyze and process the collected user feedback information. The CrabGPT knowledge question answering model uses the results of real-time analysis and processing as real-time feedback, continuously adjusts its output strategy, improves the accuracy of the CrabGPT knowledge question answering model, and combines the DPO algorithm to perform style alignment on the large model.

4. The optimization method according to claim 3, characterized in that: Step 2 specifically includes the following steps: Step 2.1: Data cleaning of professional data documents in the question-answer benchmark data obtained by scenario case mining; Step 2.2, generate questions based on professional data documents; Step 2.3: Generate answers based on professional data documents and questions, organize them into question-answer pairs, and use the Kimi large model to form a set of supervised fine-tuning instructions; Step 2.4: Use these fine-tuning instruction sets to perform SFT supervised fine-tuning on the CrabGPT knowledge question answering model.

5. The optimization method according to claim 4, characterized in that: The SFT supervised fine-tuning in step 2.4 specifically includes the following steps: Step 2.4.1, initialization stage: using Gaussian distribution A = N (0, σ 2 ) Initialize the weight parameters of matrix A and set matrix B to zero matrix; Step 2.4.2, add additional matrices A and B to the CrabGPT model, and calculate the value of the output feature h of the hidden layer: h=Wx+BAx In the above formula, h is the output feature of the hidden layer, W is the weight matrix in the large model CrabGPT, and x represents the input feature. Step 2.4.3, add the trained matrix product BA to the original weight matrix W: h=Wx+BAx=(W+BA)x Step 2.4.4, form a new weight parameter W+BA, and use the new weight parameter W+BA to replace the original weight matrix W in the large model CrabGPT.

6. The optimization method according to any one of claim 5, characterized in that: The value of σ is set to 0.

01.

7. The optimization method according to any one of claims 3 to 5, characterized in that: Step 4 specifically includes the following steps: Step 4.1: Improve the accuracy of the CrabGPT knowledge question answering model: The user asks a question and the large model CrabGPT obtains the corresponding answer. If the user is satisfied, it is marked as the model's chosen answer (preferred answer, y w ), and if the user is not satisfied, it is marked as rejected by the model (non-preferred answer, y l ); According to the model, choose (preference answer, y w ) and the model rejects (non-preferred answers, y l ) is used to optimize the CrabGPT model for river crab knowledge question answering. The optimization goal is to maximize the value of the function in the following formula: In the above formula, π θ is the current strategy, π ref is the reference strategy; L DPO (π θ ; π ref ) is the loss function in the DPO algorithm, which is used to measure the current strategy π θ With reference strategy π ref The difference between E(x,y w ,y l )~D is the expectation operator, which means that according to all the samples (x,y) in the data distribution D w ,y l ) to calculate the expected value of the internal expression, where x represents the state and y w and l Represent positive samples and negative samples respectively, which are used to compare the performance of the current strategy and the reference strategy; Refers to the cumulative probability difference between the generated preferred answer and the initial generated preferred answer, represents the cumulative probability difference between the generated non-preferred answer and the initial generated non-preferred answer, and σ is the σ value in the Gaussian distribution; After optimizing the CrabGPT model, the model has a better understanding of chosen (preferred answer, y w ) and the model rejects (non-preferred answers, y l ) is the largest, the reward of the CrabGPT model for input data is maximized, and then the human preference data is learned; Step 4.2: Use the DPO algorithm combined with the provided human preference data to perform style alignment on the large model.

Images