A method for constructing a document vector search engine based on a decoder-only architecture
By using a large model and a large language model with a decoder-only architecture to generate training data in a vector search engine, and building an embedding vector generator, the problem of being unable to utilize the latest models and difficulty in obtaining training data is solved in the existing technology, and it is realized in enterprises Fast semantic search in document libraries improves work efficiency.
Patent Information
- Application Number
- CN202410892615.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-04
- Publication Date
- 2025-01-24
- Estimated Expiration
- 2044-07-04
AI Technical Summary
The existing vector search engines are mainly based on large models with encoder-only architecture, and cannot use the latest large language model with decoder-only architecture, and it is difficult to obtain search training data in special fields, so it is impossible to achieve fast semantics for users in the enterprise document library. search.
Use a large model based on the decoder architecture to build an embedded directional generator, generate corporate document search training data through large language models, and train a search engine specifically for corporate documentation content for the content of the corporate documentation.
The rapid search and positioning documents in the corporate document library have been achieved, which improves the efficiency of corporate employees and has high commercial value.
Smart Images
Figure CN118689994B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of document search engines, and in particular to a method for constructing a document vector search engine based on a decoder-only architecture. Background Art
[0002] Enterprises will generate a large number of documents during operation. These documents have the characteristics of complex structure, large number and diverse content. How to quickly search for the documents required by users from a huge enterprise document library is an important function. Traditional search engine technology mainly searches and matches based on word frequency, and cannot search based on user semantic intelligence. Vector search technology is a new search technology based on artificial intelligence, but the common vector search engines are currently based on large models of encoder architecture only, and cannot use the latest large language models of decoder architecture only (such as internLM, GPT). In addition, it is also quite difficult to obtain search training data in special fields. In response to these problems, the present invention proposes a new vector search engine based on decoder architecture only and a corresponding training method, which can use an advanced decoder architecture only large model as a base, and use a large language model to construct training data, and train a search engine specifically for enterprise document content, allowing users to quickly find and locate the required enterprise document library from the enterprise document library. This method has broad prospects, can improve the work efficiency of enterprise employees, and has high commercial value. Summary of the invention
[0003] The present invention provides a method for constructing a document vector search engine based on a decoder-only architecture, comprising:
[0004] Step 1: Build enterprise document retrieval training data based on the large language model;
[0005] Step 2: Build an embedding vector generator with a large decoder-only model.
[0006] Step 3: Use enterprise document retrieval training data to train and tune the embedding vector generator of the decoding-only architecture;
[0007] Step 4. Use the trained embedding vector generator of the decoding-only architecture to calculate the embedding vector of the enterprise document and save it in the vector database.
[0008] Step 5. Build the search engine backend based on the vector database.
[0009] The method for constructing a document vector search engine based on a decoder-only architecture as described above, wherein enterprise document retrieval training data is constructed based on a large language model, specifically includes the following sub-steps:
[0010] Collect the enterprise's document data from its internal document management system;
[0011] Format the document data and extract the document paragraphs therein;
[0012] Use a large language model to generate corresponding question words for each document paragraph;
[0013] Combine document paragraphs and question word pairs to form enterprise document retrieval training data.
[0014] As described above, a method for constructing a document vector search engine based on a decoder-only architecture, wherein an embedding vector generator is constructed using a decoder-only architecture large model, specifically comprises the following sub-steps:
[0015] Use a decoder-only large model to output a representation vector for a document paragraph.
[0016] A layer is added to the decode-only architecture model to calculate the embedding vector based on the representation vector to form the embedding vector generator of the decode-only architecture.
[0017] A method for constructing a document vector search engine based on a decoder-only architecture as described above, wherein the embedding vector generator of the decoder-only architecture is trained and tuned using enterprise document retrieval training data, specifically comprising the following sub-steps:
[0018] Randomly extract 10% of the data from the training data as the validation set;
[0019] Use the loss function to evaluate the performance of the embedding vector generator on the training data, and tune the parameters in the embedding vector generator based on the evaluation results;
[0020] Determine whether to terminate training based on the performance of the embedding vector generator on the validation set.
[0021] As described above, a method for constructing a document vector search engine based on a decoder-only architecture, in which a search engine backend is built based on a vector database, specifically includes the following sub-steps:
[0022] Create an interactive page to obtain the user's query terms;
[0023] Encode the query words into embedding vectors using an embedding vector generator with a decode-only architecture;
[0024] Search the vector database for relevant document paragraphs based on the embedding vector of the query word;
[0025] The searched document paragraphs and their original documents are returned to the interactive page.
[0026] The present invention also provides a document vector search engine based on a decoder-only architecture, comprising a data preparation module, an embedding vector generation module, an embedding vector storage module and an online service module;
[0027] Data preparation module for building enterprise document retrieval training data using large oracle models
[0028] Embedding vector generation module, used to encode document paragraphs and question words into embedding vectors;
[0029] Embedding vector storage module, used to store document paragraph data and its embedding vector;
[0030] An online service module is used to provide users with fast and semantically relevant search results based on the vector database.
[0031] A document vector search engine based on a decoder-only architecture as described above, wherein the data preparation module includes a document data collection submodule, a document paragraph extraction submodule, and a question word generation submodule;
[0032] The document data collection submodule is used to collect the enterprise's document data from the enterprise's internal document management system;
[0033] The document paragraph extraction submodule is used to extract the document paragraphs after formatting the document data;
[0034] The question word generation submodule is used to generate corresponding question words for each document paragraph using a large language model.
[0035] A document vector search engine based on a decoder-only architecture as described above, wherein the embedding vector generation module includes an embedding vector generator construction submodule, an embedding vector generator training submodule, and an embedding vector generator use submodule;
[0036] An embedding vector generator construction submodule, used to construct a vector generator of only decoder architecture with a large decoder architecture model;
[0037] An embedding vector generator training submodule, which is used to train and tune the embedding vector generator of the decoding-only architecture using enterprise document retrieval training data;
[0038] The embedding generator uses a submodule to compute the embedding vectors of enterprise documents using the trained embedding generator of the decode-only architecture.
[0039] The beneficial effects achieved by the present invention are as follows: using an advanced decoder-only architecture large model as a base, and utilizing a large language model to construct training data, a search engine specifically targeting corporate document content is trained, allowing users to quickly find and locate the required corporate document library from the corporate document library. This method has broad prospects, can improve the work efficiency of corporate employees, and has high commercial value. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0041] Figure 1 This is a flow chart of a method for constructing a document vector search engine based on a decoder-only architecture provided in Embodiment 1 of the present invention. DETAILED DESCRIPTION
[0042] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0043] Embodiment 1
[0044] like Figure 1 As shown, the first embodiment of the present invention provides a method for constructing a document vector search engine based on a decoder-only architecture, including:
[0045] Step S10: constructing enterprise document retrieval training data based on the large language model;
[0046] Collect the document data required by the enterprise, format the data, form standard document paragraphs, and build virtual question words for each paragraph through the large language model to obtain document paragraph and question word pairs. The specific steps include the following:
[0047] Step S11: Collecting the enterprise's document data from the enterprise's internal document management system;
[0048] The internal document management system of an enterprise maintains a large number of documents generated during the operation of the enterprise, and these documents are the data source of the search engine.
[0049] Step S12: formatting the document data and extracting the document paragraphs therein;
[0050] The collected document data has various formats and structures, so when obtaining the document data, it is necessary to convert the document data into a unified format, and then extract the standard document paragraphs according to the document structure and number the document paragraphs. A standard document paragraph means that each paragraph is an independent semantic unit, that is, it should express a complete idea or information.
[0051] Step S13: Generate corresponding question words for each document paragraph using a large language model;
[0052] The model can be trained to generate questions based on the content of a paragraph, or the model can be used to extract key information from a paragraph as question words. A sample can be manually reviewed to ensure that the generated question words can guide users to find the paragraph. The generated question words are numbered according to their corresponding paragraphs so that the question words and the corresponding paragraphs are saved in pairs.
[0053] Step S14: combining the document paragraphs and the question word pairs to form enterprise document retrieval training data;
[0054] The enterprise document retrieval training data set is represented as: A = {(Q1, P1), (Q2, P2), ..., (Q n ,P n )}, where P1~P n Represent different document paragraphs, Q1~Q n are the question words of each document paragraph, and n is the number of document paragraphs.
[0055] Step S20: construct an embedding vector generator using only the decoder architecture large model;
[0056] The embedding vector generator is used to map text to a mathematical vector. Constructing the embedding vector generator includes the following sub-steps:
[0057] Step S21: Use only the decoder architecture large model to output the representation vector of the document paragraph;
[0058] The decoder-only model is a self-attention Transformer that takes a plain text sequence X = {x1, x2, x3, …, x n} as input, and obtain the representation vector through the vocabulary Where X is the input set, F (0) is the output set, x1~x n Represent different texts respectively, n is the number of input texts, They represent the representation vectors of different texts respectively.
[0059] In each layer of the decoder-only model, the representation vectors will interact with each other through the self-attention mechanism. The kth layer of the decoder-only model is represented as:
[0060] Q=FW Q K=FW K V=FW V
[0061]
[0062] The kth layer is a layer randomly extracted from the decoder-only architecture model. F represents the input of the kth layer, F′ represents the output of the kth layer, Q, K, and V represent the query, key, and value in the self-attention mechanism, respectively. W Q , W K , W V , W F Represents the linear transformation matrices of Q, K, V, and F respectively, K T represents the transposed matrix of K, softmax represents the softmax function, and dk represents the dimension of Q, K, and V.
[0063] Take the document paragraph as the input plain text sequence and output the representation vector sequence of the document paragraph, expressed as: N is the total number of layers in the decoder-only model. are the representation vectors of each paragraph, and n is the number of input document paragraphs.
[0064] Step S22: adding a layer to the decoding-only architecture large model, for calculating the embedding vector according to the representation vector to form an embedding vector generator of the decoding-only architecture;
[0065] Based on the decoder-only architecture model, the representation vector of each document paragraph is obtained, and then the formula is used:
[0066] To calculate the embedding vector of each document paragraph, where F (N) is the input parameter, which represents the set of representation vectors obtained from the decoder-only architecture model. δ and ∈ are adjustable parameters. j represents the ordinal number of the layer in the decoder-only architecture model, ranging from 1 to N, where N is the total number of layers in the decoder-only architecture model. Indicates the input parameter F (N) The representation vector of the i-th document paragraph in , i ranges from 1 to n, and n is the number of document paragraphs.
[0067] Step S30: training and tuning the embedding vector generator of the decoding-only architecture using the enterprise document retrieval training data;
[0068] Through contrastive learning technology, the parameters of the embedding vector generator are tuned during the training process to ensure the practicality of the generator. Specifically:
[0069] Step S31: randomly extract 10% of the data from the training data as a validation set;
[0070] The validation set needs to be relatively independent from the training data so that the generalization ability of the vector generator on unfamiliar samples can be evaluated.
[0071] Step S32: using the loss function to evaluate the performance of the embedding vector generator on the training data, and tuning the parameters in the embedding vector generator according to the evaluation results;
[0072] First, the training data is divided into multiple training batches, and then these training batches are used to train and tune the embedding vector generator. Several samples are randomly selected in a training batch, and the loss function is calculated respectively. The parameters in the embedding vector generator are optimized by the stochastic gradient descent method. The loss function of the vector generator is expressed as:
[0073] Among them l t is the tth sample (Q t ,P t ) loss value, s(Q t ,P t )=h(Q t )·h(P t ) represents the document paragraph Q in the tth sample t and query word P t The embedding vector h(Q t ) and h(P t ), (Q t ,P k ) represents the kth negative sample for the tth sample, τ is the adjustable parameter of the loss function, s(Q t ,P k )=h(Q t )·h(P k ) means that for the kth negative sample of the tth sample, the document paragraph Q t and query word P k The embedding vector h(Q t ) and h(P t ), k takes values from 1 to B-1, and B is the size of the batch where the tth sample is located.
[0074] Step S33: judging whether to terminate the training according to the performance of the embedding vector generator on the validation set;
[0075] The performance of the embedding vector generator on the validation set is also reflected in the calculation results of the loss function. Observe the changes in the loss function curve on the validation set. When the loss function value no longer decreases, stop training.
[0076] Step S40: using the trained embedding vector generator of the decoding-only architecture to calculate the embedding vector of the enterprise document and save it in the vector database;
[0077] Using the trained embedding vector generator of the decoder-only architecture, all enterprise documents are encoded and saved in a vector database such as faiss or qdrant or weaviate.
[0078] Step S50: Building a search engine backend based on the vector database;
[0079] The search engine backend is used to provide fast and semantically relevant search results. Building a search engine backend includes the following sub-steps:
[0080] Step S51: creating an interactive page to obtain the user's query words;
[0081] This interactive page includes at least two functions: inputting query terms and displaying search results.
[0082] Step S52: Encode the query word into an embedding vector using an embedding vector generator of a decoding-only architecture;
[0083] Step S53: searching for relevant document paragraphs in the vector database according to the embedding vector of the query word;
[0084] Calculate the cosine similarity between the question word embedding vector and the document paragraph embedding vector in the vector database, and return the document paragraphs with the highest similarity as search results. The number of returned items can be customized according to needs.
[0085] Step S54: Return the searched document paragraph and its original document to the interactive page.
[0086] Embodiment 2
[0087] Embodiment 2 of the present invention provides a document vector search engine based on a decoder-only architecture, comprising a data preparation module, an embedding vector generation module, an embedding vector storage module and an online service module;
[0088] (1) A data preparation module, which is used to construct enterprise document retrieval training data using a large prediction model, including a document data collection submodule, a document paragraph extraction submodule, and a question word generation submodule;
[0089] 1. Document data collection submodule, used to collect the enterprise's document data from the enterprise's internal document management system;
[0090] The internal document management system of an enterprise maintains a large number of documents generated during the operation of the enterprise, and these documents are the data source of the search engine.
[0091] 2. Document paragraph extraction submodule, used to extract document paragraphs after formatting document data;
[0092] The collected document data has various formats and structures, so when obtaining the document data, it is necessary to convert the document data into a unified format, and then extract the standard document paragraphs according to the document structure and number the document paragraphs. A standard document paragraph means that each paragraph is an independent semantic unit, that is, it should express a complete idea or information.
[0093] 3. Question word generation submodule, which is used to generate corresponding question words for each document paragraph using a large language model;
[0094] The model can be trained to generate questions based on the content of a paragraph, or the model can be used to extract key information from a paragraph as question words. A sample can be manually reviewed to ensure that the generated question words can guide users to find the paragraph. The generated question words are numbered according to their corresponding paragraphs so that the question words and the corresponding paragraphs are saved in pairs.
[0095] Combining document paragraphs with question word pairs forms enterprise document retrieval training data;
[0096] The enterprise document retrieval training data set is represented as: A = {(Q1, P1), (Q2, P2), ..., (Q n ,P n )}, where P1~P n Represent different document paragraphs, Q1~Q n are the question words of each document paragraph, and n is the number of document paragraphs.
[0097] (2) an embedding vector generation module, which is used to encode document paragraphs and question words into embedding vectors, including an embedding vector generator construction submodule, an embedding vector generator training submodule, and an embedding vector generator use submodule;
[0098] 1. Embedding vector generator construction submodule, used to construct a decoder-only architecture vector generator with a decoder-only architecture large model, specifically including:
[0099] ① Use a decoder-only model to output a representation vector for a document paragraph;
[0100] The decoder-only model is a self-attention Transformer that takes a plain text sequence X = {x1, x2, x3, …, x n} as input, and obtain the representation vector through the vocabulary Where X is the input set, F (0) is the output set, x1~x nRepresent different texts respectively, n is the number of input texts, They represent the representation vectors of different texts respectively.
[0101] In each layer of the decoder-only model, the representation vectors will interact with each other through the self-attention mechanism. The kth layer of the decoder-only model is represented as:
[0102] Q=FW Q K=FW K V=FW V The kth layer is randomly drawn from the decoder-only architecture of the large model.
[0103]
[0104] layer, F represents the input of the kth layer, F′ represents the output of the kth layer, Q, K, and V represent the query, key, and value in the self-attention mechanism respectively, and W Q , W K , W V , W F Represents the linear transformation matrices of Q, K, V, and F respectively, K T represents the transposed matrix of K, softmax represents the softmax function, and dk represents the dimension of Q, K, and V.
[0105] Take the document paragraph as the input plain text sequence and output the representation vector sequence of the document paragraph, expressed as: In this case, N is the total number of layers of the decoder architecture model only. are the representation vectors of each paragraph, and n is the number of input document paragraphs.
[0106] ② Add a layer of embedding vector generator for calculating embedding vector based on representation vector to form a decoding-only architecture on the large model of decoding-only architecture;
[0107] Based on the decoder-only architecture model, the representation vector of each document paragraph is obtained, and then the formula is used:
[0108] To calculate the embedding vector of each document paragraph, where F (N) is the input parameter, which represents the set of representation vectors obtained from the decoder-only architecture model. δ and ∈ are adjustable parameters. j represents the ordinal number of the layer in the decoder-only architecture model, ranging from 1 to N, where N is the total number of layers in the decoder-only architecture model. Indicates the input parameter F (N) The representation vector of the i-th document paragraph in , i ranges from 1 to n, and n is the number of document paragraphs.
[0109] 2. Embedding vector generator training submodule, which is used to train and tune the embedding vector generator of the decoding-only architecture using enterprise document retrieval training data;
[0110] Through contrastive learning technology, the parameters of the embedding vector generator are tuned during the training process to ensure the practicality of the generator. Specifically:
[0111] ① Randomly extract 10% of the data from the training data as the validation set;
[0112] The validation set needs to be relatively independent from the training data so that the generalization ability of the vector generator on unfamiliar samples can be evaluated.
[0113] ② Use the loss function to evaluate the performance of the embedding vector generator on the training data, and tune the parameters in the embedding vector generator based on the evaluation results;
[0114] First, the training data is divided into multiple training batches, and then these training batches are used to train and tune the embedding vector generator. Several samples are randomly selected in a training batch, and the loss function is calculated respectively. The parameters in the embedding vector generator are optimized by the stochastic gradient descent method. The loss function of the vector generator is expressed as:
[0115] Among them l t is the tth sample (Q t ,P t ) loss value, s(Q t ,P t )=h(Q t )·h(P t ) represents the document paragraph Q in the tth sample t and query word P t The inner product of the embedding vector, (Q t ,P k ) represents the kth negative sample for the tth sample, τ is the adjustable parameter of the loss function, s(Q t ,P k )=h(Q t )·h(P k ) means that for the kth negative sample of the tth sample, the document paragraph Q t and query word P k The inner product of the embedding vector, k takes values from 1 to B-1, and B is the size of the batch where the tth sample is located.
[0116] ③Determine whether to terminate training based on the performance of the embedding vector generator on the validation set;
[0117] The performance of the embedding vector generator on the validation set is also reflected in the calculation results of the loss function. Observe the changes in the loss function curve on the validation set. When the loss function value no longer decreases, stop training.
[0118] 3. The embedding vector generator uses a submodule to calculate the embedding vector of the enterprise document using the embedded vector generator of the trained decoding-only architecture.
[0119] (3) an embedding vector storage module, used to store document paragraph data and its embedding vector;
[0120] The document paragraph data includes the original text of the document paragraph and the index of the document where it is located.
[0121] (4) An online service module, which is used to provide users with fast and semantically relevant search results based on the vector database, including:
[0122] ①Use interactive pages to obtain user query terms;
[0123] This interactive page includes at least two functions: inputting query terms and displaying search results.
[0124] ② Encode the query words into embedding vectors using an embedding vector generator with a decoding-only architecture;
[0125] ③ Search for relevant document paragraphs in the vector database based on the embedding vector of the query word;
[0126] Calculate the cosine similarity between the question word embedding vector and the document paragraph embedding vector in the vector database, and use the document paragraphs with the highest similarity as search results. The number of search results can be customized according to needs.
[0127] ④ Return the searched document paragraph and its original document to the interactive page.
[0128] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for constructing a document vector search engine based on a decoder-only architecture, characterized in that: The method comprises: Step 1: Build enterprise document retrieval training data based on the large language model; Step 2: Build an embedding vector generator with a large decoder-only model. Step 3: Use enterprise document retrieval training data to train and tune the embedding vector generator of the decoding-only architecture; Step 4. Use the trained embedding vector generator of the decoding-only architecture to calculate the embedding vector of the enterprise document and save it in the vector database. Step 5. Build the search engine backend based on the vector database; Building the embedding vector generator with a decoder-only architecture model consists of the following sub-steps: Use a decoder-only large model to output a representation vector for a document paragraph. A layer is added to the decode-only architecture model to calculate the embedding vector based on the representation vector to form the embedding vector generator of the decode-only architecture.
2. A method for constructing a document vector search engine based on a decoder-only architecture according to claim 1, characterized in that: Building enterprise document retrieval training data based on a large language model includes the following sub-steps: Collect the enterprise's document data from its internal document management system; Format the document data and extract the document paragraphs therein; Use a large language model to generate corresponding question words for each document paragraph; Combine document paragraphs and question word pairs to form enterprise document retrieval training data.
3. A method for constructing a document vector search engine based on a decoder-only architecture according to claim 1, characterized in that: The embedding vector generator for the decode-only architecture is trained and fine-tuned using the enterprise document retrieval training data. The specific steps include: Randomly extract 10% of the data from the training data as the validation set; Use the loss function to evaluate the performance of the embedding vector generator on the training data, and tune the parameters in the embedding vector generator based on the evaluation results; Determine whether to terminate training based on the performance of the embedding vector generator on the validation set.
4. A method for constructing a document vector search engine based on a decoder-only architecture according to claim 1, characterized in that: Building a search engine backend based on a vector database includes the following sub-steps: Create an interactive page to obtain the user's query terms; Encode the query words into embedding vectors using an embedding vector generator with a decode-only architecture; Search the vector database for relevant document paragraphs based on the embedding vector of the query word; The searched document paragraphs and their original documents are returned to the interactive page.
5. A document vector search engine based on a decoder-only architecture, characterized in that It includes a data preparation module, an embedding vector generation module, an embedding vector storage module and an online service module; Data preparation module, used to build enterprise document retrieval training data using the big oracle model; Embedding vector generation module, used to encode document paragraphs and question words into embedding vectors; Embedding vector storage module, used to store document paragraph data and its embedding vector; An online service module for providing users with fast and semantically relevant search results based on the vector database; The embedding vector generation module includes an embedding vector generator construction submodule, an embedding vector generator training submodule, and an embedding vector generator use submodule; An embedding vector generator construction submodule, used to construct a vector generator of only decoder architecture with a large decoder architecture model; An embedding vector generator training submodule, which is used to train and tune the embedding vector generator of the decoding-only architecture using enterprise document retrieval training data; The embedding vector generator uses a submodule for computing the embedding vector of the enterprise document using the trained embedding vector generator of the decoding-only architecture; The decoder-only architecture vector generator is constructed based on the decoder-only architecture model, including: Use a decoder-only large model to output a representation vector for a document paragraph. A layer is added to the decode-only architecture model to calculate the embedding vector based on the representation vector to form the embedding vector generator of the decode-only architecture.
6. A document vector search engine based on a decoder-only architecture according to claim 5, characterized in that: The data preparation module includes a document data collection submodule, a document paragraph extraction submodule, and a question word generation submodule; The document data collection submodule is used to collect the enterprise's document data from the enterprise's internal document management system; The document paragraph extraction submodule is used to extract the document paragraphs after formatting the document data; The question word generation submodule is used to generate corresponding question words for each document paragraph using a large language model.
Citation Information
Patent Citations
Model training method and question answering method for question answering system
CN118093841A