End-to-end large model fine tuning method and system based on retrieval enhancement generation

Abstract

The invention provides an end-to-end large model fine tuning method and system based on retrieval enhancement generation. The method comprises the steps that text information of historical questions of a user is read; performing deep semantic extraction on the text information by utilizing a semantic extraction module of the word segmentation device model to obtain key semantic output; the encoded semantic vector is obtained from the database, the deep semantic information is encoded by using the BGE retrieval model and the M3E vector model, similar information is recalled, and the two paths of recalled vector semantic information are combined; respectively calculating KL divergences of the deep semantic information vector and the text information vector with a target vector in a semantic space, and selecting and inputting the deep semantic information vector and the text information vector with relatively low cost into an examination module; and training the model according to the preference data by using a DPO algorithm to obtain a model conforming to human preferences. The fine-tuned model obtained through the method does not need to be preprocessed in the reasoning process, and the problems of model illusion, toxicity and the like caused by inaccurate retrieval are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to an end-to-end large model fine-tuning method and system based on retrieval-enhanced generation. Background Art

[0002] With the rapid development of natural language processing technology, the number of parameters in pre-trained language models has greatly expanded, showing "emergent capabilities" such as contextual learning and zero-shot learning. These emergent capabilities have played a significant role in conversational robots, but the knowledge learned by the pre-trained language model during the training phase is stored in the model structure in the form of fixed parameters. These parameters will not change over time, that is, the model can only obtain the knowledge learned during the pre-training phase and cannot update the knowledge without re-training.

[0003] Retrieval-Augmented Generation (RAG) technology encodes new knowledge through a vector model and stores the encoded text information in a vector database. When a user asks a question about new knowledge, the vector model encodes the question and calculates the similarity with the vector information in the vector database, selects the content with the best similarity, and splices the content with the user's query question. When a user asks a question about new knowledge, the vector model encodes the question and calculates the similarity with the vector information in the vector database, selects the content with the best similarity, and splices the content with the user's query question.

[0004] However, although RAG can solve the problem of knowledge updating, it still faces many challenges. For example, the large amount of information recalled by the retrieval model lacks key semantic information and also contains a large amount of unimportant redundant information. The introduction of this information will mislead the output of the model; the retrieval model is not aligned with the large language model. The information that the retrieval model believes is relevant to the user's question is not relevant in the large language model, while the information that the large language model believes is relevant is not recalled by the retrieval model; there is no guarantee that the model will not produce toxic, unstable, hallucinatory and other outputs that do not conform to human preferences after receiving relevant information.

[0005] In view of this, it is very meaningful to propose an end-to-end large model fine-tuning method and system based on retrieval-enhanced generation. Summary of the invention

[0006] In order to solve the above-mentioned problems and reduce the unreliability of model-generated content caused by retrieval, the present invention provides an end-to-end large model fine-tuning method and system based on retrieval-enhanced generation, which accurately recalls relevant information through a two-way recall module and uses a review module to align the retrieval model and the large language model's understanding of user preference information, ultimately obtaining a model that meets human preferences. The fine-tuned model can be directly inferred without preprocessing, thereby solving the above-mentioned technical defects.

[0007] In a first aspect, the present invention proposes an end-to-end large model fine-tuning method based on retrieval-enhanced generation, the method comprising the following steps:

[0008] S1. Read the text information of the user's historical questions;

[0009] S2, using the semantic extraction module of the word segmentation model to perform deep semantic extraction on the text information to obtain key semantic output;

[0010] S3, obtain the encoded semantic vector from the database, use the BGE retrieval model to encode the user question and recall similar semantic information, and use the M3E vector model to encode the deep semantic information and recall similar information, merge the two recalled vector semantic information, and use 1D convolution for feature extraction to obtain the target vector;

[0011] S4, respectively calculating the KL divergence of the deep semantic information vector and the text information vector with the target vector in the semantic space, selecting the one with the smaller cost and inputting it into the review module, wherein the review module includes the Deberta, Roberta and Albert backbone networks, and the review module determines whether the sample data meets the preference according to the information and background information input by the user, and obtains the preference data;

[0012] S5. Use the DPO algorithm to train the model with the preference data to obtain a model that meets human preferences.

[0013] Preferably, the semantic extraction module in S2 includes a word segmenter model and a Bi-LSTM backbone network. The word segmenter model is used to segment text information into words, which needs to be consistent with the word segmenter of the retrieval model. The Bi-LSTM backbone network is used to extract features and generate semantic vector information.

[0014] Further preferably, the structure of the semantic extraction module in S2 specifically includes:

[0015] S21, the set of user historical questions is D his , D his =(Q1, Q2...Q n), where Q1 is the text information in the set, and when obtaining the user's historical questions, D his Read Q n (n∈[1,n]);

[0016] S22, output function Where T is the word segmenter, which is used to convert Q n The information in is divided into {T1, T2, ..., T n}, Map is the mapping from Token to word vector ID, Map = {(T1, ID1), (T2, ID2), ... (T n , ID n )}, p is the hyperparameter information in DropOut;

[0017] S23, Linear Time Series Model LSTM: LSTM (T n )=o t (f t *c t-1 +i t *g t ), where: i t =Sigmoid(W ii T nt +b ii +W hi T n(t-1) +b hi ), f t =Sigmoid(W if T nt +b if +W hf T n(t-1) +b hf ), g t =tanh(W ig T nt +b ig +W hg T n(t-1) +b hg ), o t =Sigmoid(W io T nt +b io +W ho T n(t-1) +b ho ), after passing through the LSTM network, we get T′ n =LSTM(T n ), T′ n The dimension is three-dimensional T′ n ∈R 1×len×Dim , where len is the maximum length supported by the model and Dim is the dimension of the word vector;

[0018] S24. Get T' n Then input the feedforward neural network to get T″ n :

[0019] T" n =FNN(T′ n ), FNN(T′ n )=(T′ n A T +b1)B T +b2

[0020] Among them, A and B are coefficient matrices, b1 and b2 are bias terms, and FNN is a two-layer fully connected network, which is used to map the dimension of the semantic feature vector extracted by the semantic extraction module to make it consistent with the number of words in the retrieval model vocabulary;

[0021] S25, finally through CRF (T ″ n )={(s1, Φ1), (s2, Φ2)...(s n , Φ n )} decoding obtains the text representation of deep semantic information, where Φ = (n, nz, a, m, c...w) is the part-of-speech information.

[0022] Preferably, obtaining the encoded semantic vector from the database in S3 specifically includes:

[0023] S31, obtain the encoded semantic vector E from the database in two parallel batches SN and E QN Then, the BGE retrieval model and the M3E encoding model are used to encode the user's question. The encoders are the encoding layer of the BGE model and the encoding layer of the M3E model. The obtained encoding vectors are S N and Q N ;

[0024] S32. Semantics of vector libraries SN and E QN Splice and obtain the spliced ​​encoding vector information E N , E N ∈R 2 ×len×Dim , then E N Perform a one-dimensional convolution operation to obtain E′ N , E′ N =Conv1D(E N , K 2×1 ), where K is a 2×1 convolution kernel;

[0025] S33, E' N The semantic vector of is regarded as the target distribution of the semantic space, and S is calculated respectively.N and Q N Fitting E′ N Distributed cost loss, get cost loss

[0026]

[0027] S34, by selecting background information with a smaller cost to approximate the semantic distribution of the user's question text and obtain its corresponding text information: The text information corresponding to the vector with the smaller cost is obtained as the background information.

[0028] Preferably, selecting the input with lower cost to the review module in S4 includes:

[0029] The data sample input to the review module is: rele ={(Q N , S N , Score N )}, where S coreN For the smaller cost loss in S3, assuming The cost is smaller, and the formula input to the review module is: F D =Deberta(Q N ,S N ,Score N ), F R =Roberta(Q N , S N ,Score N ), F A =Albert(Q N , S N , Score N )

[0030] The user question is Q N , background knowledge is S N , the cost loss is Score N , F D , F R , F A They are the output feature vectors corresponding to the Deberta, Roberta and Albert backbone networks respectively.

[0031] Further preferably, it also includes:

[0032] The three features F D , F R , F A The fusion strategy is as follows:

[0033]

[0034] Among them, F RA It is the fusion of the feature vector extracted by Roberta and the feature vector extracted by Albert. α and β are artificially set hyper parameters.

[0035] Get F RA After that, it is fused with the vector features extracted by Deberta. The fusion strategy is as follows:

[0036]

[0037] Among them, λ is a hyperparameter set manually. Then, the feature vector after the fusion of the three backbone models is input into the binary classifier. The specific operation is as follows:

[0038] PD is the preference data and NPD is the non-preference data.

[0039] Further preferably, in S5, the preference data is trained on a model using a DPO algorithm to obtain a model that conforms to human preferences, including:

[0040] Obtain preference data X from S4 PD and X NPD At the same time, X PD and X NPD As a set of samples of training data, DPO training is performed, and S1 operation is performed synchronously; the objective function of the DPO training method is:

[0041]

[0042] Among them, the policy function π θ and π ref are causal language models, and are initialized to the model to be trained π0; Q is the text information of the user in the current step, X PD , X NPD is the recall information of the text information corresponding to the current Q, and β is a hyperparameter set artificially.

[0043] In a second aspect, an embodiment of the present invention provides an end-to-end large model fine-tuning system based on retrieval-enhanced generation, comprising:

[0044] A text reading module is configured to read the text information of the user's historical questions;

[0045] A semantic extraction module, configured to use the semantic extraction module of the word segmentation model to perform deep semantic extraction on the text information to obtain key semantic output;

[0046] An information recall module is configured to obtain the encoded semantic vector from the database, encode the user question using the BGE retrieval model and recall similar semantic information, and encode the deep semantic information using the M3E vector model and recall similar information, merge the two recalled vector semantic information, and use 1D convolution for feature extraction to obtain the target vector;

[0047] A review module is configured to respectively calculate the KL divergence between the deep semantic information vector and the text information vector in the semantic space and the target vector, and select the input with the smaller cost to the review module, wherein the review module includes Deberta, Roberta and Albert backbone networks, and the review module determines whether the sample data meets the preference according to the information and background information input by the user, and obtains the preference data;

[0048] The training module is configured to train the model using the preference data using the DPO algorithm to obtain a model that conforms to human preferences.

[0049] Further preferably, the semantic extraction module includes a word segmenter model and a Bi-LSTM backbone network. The word segmenter model is used to segment text information into words, which needs to be consistent with the word segmenter of the retrieval model. The Bi-LSTM backbone network is used to extract features and generate semantic vector information.

[0050] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0051] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0052] Compared with the prior art, the beneficial results of the present invention are:

[0053] The present invention designs a bidirectional parallel recall module to address the recall redundancy and omission problems existing in RAG. The key text information in the database is accurately located by extracting the shallow semantic information (sentence information) and deep semantic information (key semantic information) of the text information in parallel; at the same time, the KL divergence is used to evaluate and filter out the redundant information in the recall information, which helps to recall accurate information to assist the reliability and accuracy of the subsequent content generation of the large language model. Afterwards, in order to align the human preferences of the retrieval model and the large language model, the present invention designs a review module, which uses three lightweight models to read the spliced ​​content (text information and recall information) for division, obtains the preference data PD and the non-preference data NPD, and finally uses the divided data to fine-tune the causal language model. Thanks to the end-to-end training method, the fine-tuned model obtained by this method does not need to be preprocessed during the reasoning process, and alleviates the problems of model hallucination, toxicity, etc. caused by inaccurate retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present invention. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0055] Figure 1 is a diagram of an exemplary device architecture in which an embodiment of the present invention may be applied;

[0056] Figure 2 A schematic diagram of a flow chart of an end-to-end large model fine-tuning method based on retrieval-enhanced generation according to an embodiment of the present invention;

[0057] Figure 3 This is a flow chart of an end-to-end large model fine-tuning method based on retrieval enhancement generation according to an embodiment of the present invention;

[0058] Figure 4 It is a structural diagram of a semantic extraction module in an end-to-end large model fine-tuning method based on retrieval enhancement generation in an embodiment of the present invention;

[0059] Figure 5 A structural diagram of the data input of the review module in the end-to-end large model fine-tuning method based on retrieval enhancement generation in an embodiment of the present invention;

[0060] Figure 6 A schematic diagram of an end-to-end large model fine-tuning system based on retrieval-enhanced generation according to an embodiment of the present invention;

[0061] Figure 7It is a schematic diagram of the overall architecture of an end-to-end large model fine-tuning system based on retrieval-enhanced generation according to an embodiment of the present invention;

[0062] Figure 8 It is a schematic diagram of the structure of a computer device suitable for implementing the electronic device of the embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0064] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0065] Figure 1 An exemplary system architecture 100 is shown to which an end-to-end large model fine-tuning method based on retrieval-enhanced generation or an end-to-end large model fine-tuning system based on retrieval-enhanced generation according to an embodiment of the present invention may be applied.

[0066] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0067] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0068] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or they can be implemented as a single software or software module. No specific limitation is made here.

[0069] The server 105 may be a server providing various services, such as a background information processing server that processes the verification request information sent by the terminal devices 101, 102, and 103. The background information processing server may analyze the received verification request information and obtain a processing result.

[0070] It should be noted that the end-to-end large model fine-tuning method based on retrieval enhancement generation provided in the embodiment of the present invention is generally executed by the server 105, and accordingly, the end-to-end large model fine-tuning system based on retrieval enhancement generation is generally set in the server 105. In addition, the end-to-end large model fine-tuning method based on retrieval enhancement generation provided in the embodiment of the present invention is generally executed by the terminal devices 101, 102, and 103, and accordingly, the end-to-end large model fine-tuning system based on retrieval enhancement generation is generally set in the terminal devices 101, 102, and 103.

[0071] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or multiple software modules, which is not specifically limited here.

[0072] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided as required. In the case where the processed data does not need to be obtained remotely, the above device architecture may not include a network, but only a server or a terminal device.

[0073] At present, although some excellent retrieval models (BGE, M3E, GTE, etc.) and some vector libraries (FAISS, PROXIAMI, etc.) have been designed, RAG still faces many problems:

[0074] (1) The large amount of information recalled by the retrieval model lacks key semantic information and also contains a large amount of unimportant redundant information. The introduction of this information will mislead the model's output; (2) The retrieval model is not aligned with the large language model. The information that the retrieval model believes is relevant to the user's question is not relevant in the eyes of the large language model, while the information that the large language model believes is relevant is not recalled by the retrieval model; (3) There is no guarantee that after receiving relevant information, the model will not produce toxic, unstable, hallucinatory, or other outputs that do not conform to human preferences.

[0075] In order to solve the above-mentioned problems, the embodiment of the present invention takes the retrieval recall generation of a large language model as the research object, and proposes a semantic enhancement method to address the problem that the recall information of the retrieval model is inaccurate and redundant, and designs a parallel semantic recall structure to extract the shallow semantic information (sentence information) and deep semantic information (key semantic information) of the user's question to assist in recall. At the same time, the review module is designed to divide the data into preference data and non-preference data, providing real-time data for subsequent model training, realizing end-to-end fine-tuning training of the large language model, and greatly reducing the problems of hallucinations, toxicity, etc. of the large model caused by retrieval.

[0076] First, Figure 2 It is shown that an embodiment of the present invention discloses an end-to-end large model fine-tuning method based on retrieval enhancement generation, such as Figure 2 and 3 As shown, the method comprises the following steps:

[0077] S1. Read the text information of the user's historical questions;

[0078] Specifically, in this embodiment, it is assumed that the set of user historical questions is D his , D his =(Q1, Q2, ... Q n ), where Q1 is the text information in the set, and when obtaining the user's historical questions, D his Read Q n (n∈[1,n]).

[0079] S2, using the semantic extraction module of the word segmentation model to perform deep semantic extraction on the text information to obtain key semantic output;

[0080] Specifically, using Figure 4 The semantic extraction module shown in FIG. 1 performs deep semantic extraction on text information to obtain key semantic output; Figure 4 Figure 2 shows the structure of the semantic extraction module, which includes:

[0081] The word segmentation model (which needs to be consistent with the word segmentation model of the retrieval model) is used to segment single text information, and then input into the Bi-LSTM backbone network to extract features to generate semantic vector information. The network uses a bidirectional LSTM network to better capture bidirectional semantic dependencies and facilitate the extraction of sentence information and semantic information.

[0082] S22, output function

[0083] Where T is the word segmenter, which is used to convert Q n The information in is divided into {T1, T2, ..., T n}, Map is the mapping from Token to word vector ID, Map = {(T1, ID1), (T2, ID2), ... (T n ,ID n )}, p is the hyperparameter information in DropOut;

[0084] S23, Linear Time Series Model LSTM: LSTM (T n )=o t (f t *c t-1 +i t *g t ),in:

[0085] i t =Sigmoid(W ii T nt +b ii +W hi T n(t-1) +b hi ),

[0086] f t =Sigmoid(W if T nt +b if +W hf T n(t-1) +b hf ),

[0087] g t =tanh(W ig T nt +b ig +W hg T n(t-1) +b hg ),

[0088] o t =Sigmoid(W io T nt +b io +W ho T n(t-1) +b ho ),

[0089] After passing through the LSTM network, we get T′ n =LSTM(T n ), T′ n The dimension is three-dimensional T′ n ∈R 1×len×Dim , where len is the maximum length supported by the model and Dim is the dimension of the word vector;

[0090] S24. Get T' nThen input the feedforward neural network to get T″ n :

[0091] T″ n =FNN(T′ n ),

[0092] FNN(T′ n )=(T′ n A T +b1)B T +b2,

[0093] Among them, A and B are coefficient matrices, b1 and b2 are bias terms, and FNN is a two-layer fully connected network, which is used to map the dimension of the semantic feature vector extracted by the semantic extraction module to make it consistent with the number of words in the retrieval model vocabulary;

[0094] S25, finally through CRF (T ″ n )={(s1, Φ1), (s2, Φ2)...(s n , Φ n )} decoding to obtain the text representation of deep semantic information, where Φ∈(n, nz, a, m, c...w) is the part-of-speech information.

[0095] S3, obtain the encoded semantic vector from the database, use the BGE retrieval model to encode the user question and recall similar semantic information, and use the M3E vector model to encode the deep semantic information and recall similar information, merge the two recalled vector semantic information, and use 1D convolution for feature extraction to obtain the target vector;

[0096] Specifically, this step includes: S31, obtaining the encoded semantic vector E from the database in two parallel batches SN and E QN , the acquisition method is dual-path parallel batch acquisition. Then the user's question is encoded using the BGE retrieval model and the M3E encoding model. The encoders are the encoding layer of the BGE model and the encoding layer of the M3E model. The obtained encoding vectors are S N and Q N ;

[0097] In this embodiment, the BGE model uses graph embedding technology to convert text data into a graph structure and generate semantic vectors by learning the graph structure. This method can better capture the complex relationships between texts and improve the accuracy and richness of semantic representation. The M3E model introduces a large amount of (word, sentence) data for pre-training in the pre-training stage, avoiding the problem of fuzzy similarity of inclusion relationships and is conducive to extracting deep semantic information.

[0098] S32. Semantics of vector librariesSN and E QN Splice and obtain the spliced ​​encoding vector information E N , E N ∈R 2 ×len×Dim , then E N Perform a one-dimensional convolution operation to obtain E′ N , E′ N =Conv1D(E N , K 2×1 ), where K is a 2×1 convolution kernel;

[0099] S33, E' N The semantic vector of is regarded as the target distribution of the semantic space, and S is calculated respectively. N and Q N Fitting E′ N Distributed cost loss, get cost loss

[0100]

[0101] We approximate the semantic distribution of the user's question text and obtain its corresponding text information by selecting background information with a smaller cost.

[0102] S34, by selecting background information with a smaller cost to approximate the semantic distribution of the user's question text and obtain its corresponding text information:

[0103]

[0104] The text information corresponding to the vector with the smaller cost is obtained as the background information, and step S3 is completed.

[0105] S4, respectively calculating the KL divergence of the deep semantic information vector and the text information vector with the target vector in the semantic space, selecting the one with the smaller cost and inputting it into the review module, wherein the review module includes the Deberta, Roberta and Albert backbone networks, and the review module determines whether the sample data meets the preference according to the information and background information input by the user, and obtains the preference data;

[0106] Specifically, the information with a lower cost obtained in step S3 and the decoded text information are input into the review module.

[0107] The data sample input to the review module is: rele ={(Q N , S N , Score N )}, where Score N is the smaller cost loss in S3. In this embodiment, it is assumed that The cost is lower. The review module determines whether the sample data meets the preference based on the information input by the user and the background information, and obtains the preference data. Figure 5 The data reading structure diagram for the query module. The formula input to the review module is:

[0108] F D =Deberta(Q N , S N ,Score N ), F R =Roberta(Q N , S N ,Score N ),F A =Albert(Q N ,S N ,Score N )

[0109] The user question is Q N , background knowledge is S N , the cost loss is Score N , F D , F R , F A They are the output feature vectors corresponding to the Deberta, Roberta and Albert backbone networks respectively.

[0110] In this embodiment, the significance of selecting three models here is to respectively address the problems of inconsistent text length, character-level errors, and grammatical rule diversity in the database. Deberta's virtual adversarial training algorithm (SiFT) normalizes word embedding vectors into random vectors, and then applies perturbations to the normalized embedding vectors. This operation helps to alleviate character-level error information in the vector database; Roberta's dynamic masking mechanism makes it superior to other models in capturing the diversity of sentence expressions; in order to alleviate the impact of uneven distribution of long and short information in the database, Albert's weight sharing mechanism makes it more stable when processing shorter information.

[0111] In order to enhance the robustness of the review module and obtain the advantages of the above models at the same time, this embodiment fuses the three features. The fusion strategy is as follows:

[0112]

[0113] Among them, F RA It is the fusion of the feature vector extracted by Roberta and the feature vector extracted by Albert. α and β are artificially set hyper parameters.

[0114] Get F RA After that, it is fused with the vector features extracted by Deberta. The fusion strategy is as follows:

[0115]

[0116] Among them, λ is a hyperparameter set artificially.

[0117] Then the feature vector after the fusion of the three backbone models is input into the binary classifier. The specific operation is as follows:

[0118]

[0119] PD is the preference data and NPD is the non-preference data.

[0120] S5. Use the DPO algorithm to train the model with the preference data to obtain a model that meets human preferences.

[0121] Furthermore, the preference data is used as training data, and the DPO algorithm is used to train the model to obtain a model that conforms to human preferences.

[0122] Obtain preference data X from S4 PD and X NPD At the same time, X PD and X NPD DPO training is performed as a set of samples of training data, and S1 operation is performed synchronously. The objective function of the DPO training method is:

[0123]

[0124] Among them, the policy function π θ and π ref All of them are causal language models, and are initialized to the model to be trained π0. Q is the text information of the user in the current step, X PD , X NPD is the recall information of the text information corresponding to the current Q, and β is a hyperparameter set artificially.

[0125] The innovation and effect of this method include: for the first time, a multi-way (two-way in the embodiment of the present invention) recall model is simultaneously introduced to recall the index content in parallel, and sentence information and semantic information are recalled separately, so that more relevant information can be recalled without affecting the recall speed.

[0126] One-dimensional convolution and KL divergence are introduced to calculate similarity information. Compared with the general cosine similarity, KL divergence can simultaneously consider the sentence information and semantic information of the above recall. The retrieved information satisfies the relevance of sentence structure and semantics at the same time, greatly improving the recall accuracy.

[0127] An embodiment of the present invention proposes a review module, which can directly convert questions, recall information, and confidence into preference data and non-preference data, and can directly perform end-to-end training. This end-to-end method can align the semantic differences between the large model and the retrieval model, thereby improving the final output accuracy of the large model.

[0128] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a system, which is similar to Figure 2 Corresponding to the method embodiment shown, the system can be specifically applied to various electronic devices.

[0129] In a second aspect, the embodiment of the present invention also discloses an end-to-end large model fine-tuning system based on retrieval enhancement generation, such as Figure 6 As shown, it includes: a text reading module 61, a semantic extraction module 62, an information recall module 63, a review module 64 and a training module 65. Figure 7 A schematic diagram of the overall architecture of an embodiment of the present invention is shown.

[0130] In a specific embodiment, a text reading module 61 is configured to read the text information of the user's historical questions; a semantic extraction module 62 is configured to use the semantic extraction module of the word segmentation model to perform deep semantic extraction on the text information to obtain key semantic output; an information recall module 63 is configured to obtain the encoded semantic vector from the database, encode the user question using the BGE retrieval model and recall similar semantic information, and use the M3E vector model to encode the deep semantic information and recall similar information, merge the two recalled vector semantic information, and use 1D convolution to perform feature extraction to obtain a target vector;

[0131] The review module 64 is configured to respectively calculate the KL divergence of the deep semantic information vector and the text information vector with the target vector in the semantic space, and select the input with the lower cost to the review module. The review module includes Deberta, Roberta and Albert backbone networks. The review module determines whether the sample data meets the preference based on the information and background information input by the user to obtain the preference data; the training module 65 is configured to train the model with the preference data using the DPO algorithm to obtain a model that meets human preferences.

[0132] In this embodiment, the semantic extraction module 62 includes a word segmentation model and a Bi-LSTM backbone network. The word segmentation model is used to segment text information into words, which needs to be consistent with the word segmentation of the retrieval model. The Bi-LSTM backbone network is used to extract features and generate semantic vector information.

[0133] The functions of the above modules correspond to the methods and will not be repeated here.

[0134] Reference below Figure 8 , which shows an electronic device (eg Figure 1 A schematic diagram of the structure of a computer device 600 (a server or terminal device as shown). Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0135] like Figure 8 As shown, the computer device 600 includes a central processing unit (CPU) 601 and a graphics processing unit (GPU) 602, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 603 or a program loaded from a storage part 609 to a random access memory (RAM) 604. In the RAM 604, various programs and data required for the operation of the device 600 are also stored. The CPU 601, GPU 602, ROM 603 and RAM 604 are connected to each other through a bus 605. An input / output (I / O) interface 606 is also connected to the bus 605.

[0136] The following components are connected to the I / O interface 606: an input section 607 including a keyboard, a mouse, etc.; an output section 608 including a display such as a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 609 including a hard disk, etc.; and a communication section 610 including a network interface card such as a LAN card, a modem, etc. The communication section 610 performs communication processing via a network such as the Internet. A drive 611 may also be connected to the I / O interface 606 as needed. A removable medium 612, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 611 as needed so that a computer program read therefrom is installed into the storage section 609 as needed.

[0137] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 610, and / or installed from the removable medium 612. When the computer program is executed by the central processing unit (CPU) 601 and the graphics processing unit (GPU) 602, the above-mentioned functions defined in the method of the present invention are executed.

[0138] It should be noted that the computer-readable medium described in the present invention may be a computer-readable signal medium or a computer-readable medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus or device, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution device, apparatus or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution device, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0139] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0140] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the device, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0141] The modules involved in the embodiments of the present invention may be implemented in software or hardware, and the modules described may also be arranged in a processor.

[0142] As another aspect, the present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device executes the method steps described in the first aspect.

[0143] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

Claims

1. An end-to-end large model fine-tuning method based on retrieval-enhanced generation, characterized in that: The method comprises the following steps: S1. Read the text information of the user's historical questions; S2, using the semantic extraction module of the word segmentation model to perform deep semantic extraction on the text information to obtain key semantic output; S3, obtain the encoded semantic vector from the database, use the BGE retrieval model to encode the user question and recall similar semantic information, and use the M3E vector model to encode the deep semantic information and recall similar information, merge the two recalled vector semantic information, and use 1D convolution for feature extraction to obtain the target vector; S4, respectively calculating the KL divergence of the deep semantic information vector and the text information vector with the target vector in the semantic space, selecting the one with the smaller cost and inputting it into the review module, wherein the review module includes the Deberta, Roberta and Albert backbone networks, and the review module determines whether the sample data meets the preference according to the information and background information input by the user, and obtains the preference data; S5. Use the DPO algorithm to train the model with the preference data to obtain a model that meets human preferences.

2. The end-to-end large model fine-tuning method based on retrieval-enhanced generation according to claim 1 is characterized in that: The semantic extraction module in S2 includes a word segmenter model and a Bi-LSTM backbone network. The word segmenter model is used to segment text information into words, which needs to be consistent with the word segmenter of the retrieval model. The Bi-LSTM backbone network is used to extract features and generate semantic vector information.

3. The end-to-end large model fine-tuning method based on retrieval-enhanced generation according to claim 2 is characterized in that: The structure of the semantic extraction module in S2 specifically includes: S21, the set of user historical questions is D his , D his =(Q1, Q2...Q n ), where Q1 is the text information in the set, and when obtaining the user's historical questions, D his Read Q n (n∈[1,n]); S22, output function Where T is the word segmenter, which is used to convert Q n The information in is divided into {T1, T2, ..., T n }, Map is the mapping from Token to word vector ID, Map = {(T1, ID1), (T2, ID2), ... (T n , ID n )}, p is the hyperparameter information in DropOut; S23, Linear Time Series Model LSTM: LSTM (T n )=o t (f t *c t-1 +i t *g t ), where: i t =Sigmoid(W ii T nt +b ii +W hi T n(t-1) +b hi ), f t =Sigmoid(W if T nt +b if +W hf T n(t-1) +b hf ), g t =tanh(W ig T nt +b ig +W hg T n(t-1) +b hg ), o t =Sigmoid(w Wo T nt +b io +W ho T n(t-1) +b ho ), after passing through the LSTM network, we get T′ n =LSTM(T n ), T′ n The dimension is three-dimensional T′ n ∈R 1×len×Dim , where len is the maximum length supported by the model and Dim is the dimension of the word vector; S24. Get T' n Then input the feedforward neural network to get T″ n : T″ n =FNN(T′ n ),FNN(T′ n )=(T′ n A T +b1)B T +b2 Among them, A and B are coefficient matrices, b1 and b2 are bias terms, and FNN is a two-layer fully connected network, which is used to map the dimension of the semantic feature vector extracted by the semantic extraction module to make it consistent with the number of words in the retrieval model vocabulary; S25, finally through CRF (T ″ n )={(s1,φ1), (s2,φ2)...(s n ,φ n )} decoding to obtain the text representation of deep semantic information, where φ∈(n,nz,a,m,c…w) is the part-of-speech information.

4. The end-to-end large model fine-tuning method based on retrieval-enhanced generation according to claim 1, characterized in that: Obtain the encoded semantic vector from the database in S3, including: S31, obtain the encoded semantic vector E from the database in two parallel batches SN and E QN Then, the BGE retrieval model and the M3E encoding model are used to encode the user's question. The encoders are the encoding layer of the BGE model and the encoding layer of the M3E model. The obtained encoding vectors are S N and Q N ; S32. Semantics of vector libraries SN and E QN Splice and obtain the spliced ​​encoding vector information E N , E N ∈R 2×len×Dim , then E N Perform a one-dimensional convolution operation to obtain E′ N , E′ N =Conv1D(E N , K 2×1 ), where K is a 2×1 convolution kernel; S33, E' N The semantic vector of is regarded as the target distribution of the semantic space, and S is calculated respectively. N and Q N Fitting E′ N Distributed cost loss, get cost loss and S34, by selecting background information with a smaller cost to approximate the semantic distribution of the user's question text and obtain its corresponding text information: The text information corresponding to the vector with the smaller cost is obtained as the background information.

5. The end-to-end large model fine-tuning method based on retrieval-enhanced generation according to claim 1 is characterized in that: In S4, the input with lower cost is selected to the review module including: The data sample input to the review module is: rele ={(Q N ,S N , Score N )}, where Score N For the smaller cost loss in S3, assuming The cost is smaller, and the formula input to the review module is: F D =Deberta(Q N ,S N ,Score N ),F R =Roberta(Q N ,S N ,Score N ),F A =Albert(Q N ,S N ,Score N ) The user question is Q N , background knowledge is S N , the cost loss is Score N , F D , F R , F A They are the output feature vectors corresponding to the Deberta, Roberta and Albert backbone networks respectively.

6. The end-to-end large model fine-tuning method based on retrieval-enhanced generation according to claim 5 is characterized in that: Also includes: The three features F D , F R , F A The fusion strategy is as follows: Among them, F RA It is the fusion of the feature vector extracted by Roberta and the feature vector extracted by Albert. α and β are artificially set hyper parameters. Get F RA After that, it is fused with the vector features extracted by Deberta. The fusion strategy is as follows: Among them, λ is a hyperparameter set manually, and then the feature vector after the fusion of the three backbone models is input To the binary classifier, the specific operations are: PD is the preference data and NPD is the non-preference data.

7. The end-to-end large model fine-tuning method based on retrieval-enhanced generation according to claim 6 is characterized in that: In S5, the preference data is used to train the model using the DPO algorithm to obtain a model that conforms to human preferences, including: Obtain preference data X from S4 PD and X NPD At the same time, X PD and X NPD As a set of samples of training data, DPO training is performed, and S1 operation is performed synchronously; the objective function of the DPO training method is: Among them, the policy function π θ and π ref are causal language models, and are initialized to the model to be trained π0; Q is the text information of the user in the current step, X PD , X NPD is the recall information of the text information corresponding to the current Q, and β is a hyperparameter set artificially.

8. An end-to-end large model fine-tuning system based on retrieval-enhanced generation, characterized in that: include: A text reading module is configured to read the text information of the user's historical questions; A semantic extraction module, configured to use the semantic extraction module of the word segmentation model to perform deep semantic extraction on the text information to obtain key semantic output; An information recall module is configured to obtain the encoded semantic vector from the database, encode the user question using the BGE retrieval model and recall similar semantic information, and encode the deep semantic information using the M3E vector model and recall similar information, merge the two recalled vector semantic information, and use 1D convolution for feature extraction to obtain the target vector; A review module is configured to respectively calculate the KL divergence between the deep semantic information vector and the text information vector in the semantic space and the target vector, and select the input with the smaller cost to the review module, wherein the review module includes Deberta, Roberta and Albert backbone networks, and the review module determines whether the sample data meets the preference according to the information and background information input by the user, and obtains the preference data; The training module is configured to train the model using the preference data using the DPO algorithm to obtain a model that conforms to human preferences.

9. The end-to-end large model fine-tuning system based on retrieval-enhanced generation according to claim 8, characterized in that: The semantic extraction module includes a word segmenter model and a Bi-LSTM backbone network. The word segmenter model is used to segment text information into words, which needs to be consistent with the word segmenter of the retrieval model. The Bi-LSTM backbone network is used to extract features and generate semantic vector information.

10. An electronic device, comprising: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images