Language model training method and device and information processing method and device

By filtering associated documents and building document relationship diagrams, the language model is trained by extracting target context information, which solves the context coherence and data repetition problems of large language models in complex tasks, and improves cross-document inference capabilities and performance.

CN119025655APending Publication Date: 2024-11-26INFLY TECH (SHANGHAI) CO LTD

Patent Information

Application Number
CN202411191440.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

When large language models deal with complex cross-document inference and reading comprehension tasks, there are problems such as inability to accurately follow instructions, limited inference ability of conditional document, and high variance of context learning. The existing pre-training methods cannot guarantee context coherence and data duplication and Overfitting problem.

Method used

By obtaining the document collection of the initial language model, filtering the associated documents to build a document relationship diagram, determining the document splicing path and extracting the target associated documents, obtaining the target context information, and training the language model until the training stop condition is met.

Benefits of technology

Improves the performance of language models in complex contexts and cross-document inference tasks, ensures context coherence and avoids data duplication, and improves the performance of models in complex tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025655A_ABST
    Figure CN119025655A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a language model training method and device and an information processing method and device. The language model training method comprises the steps of obtaining a document set of an initial language model, and screening associated documents in the document set to form an associated document set; constructing a document relation graph corresponding to the associated document set according to the associated documents in the associated document set and the similarity between the associated documents; determining a document splicing path in the document relation graph, and extracting a target associated document from the associated document set according to the document splicing path for splicing processing to obtain target context information; and training the initial language model based on the target context information until a target language model meeting a training stop condition is obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of this specification relate to the field of machine learning technology, and in particular to a language model training method and device, and an information processing method and device. Background Art

[0002] With the development of computer technology and Internet technology, large language models are applied in more and more scenarios, and in order to meet the diverse needs of users, the capabilities of large language models are gradually enhanced. The large language models deployed in current businesses are mainly implemented by pre-training on large-scale document data. These models usually only use the contextual information of the document prefix when performing document prediction tasks. However, with the expansion of model scale and the diversification of application scenarios, large language models still face many challenges when dealing with complex cross-document reasoning, instruction execution, and reading comprehension tasks; for example, the model may not be able to accurately follow instructions, the reasoning ability on conditional documents is limited, and there is a high variance in context learning. In the prior art, in order to avoid this problem, standard pre-training methods or retrieval-based pre-training methods are usually used to improve the model's understanding ability. However, these two methods cannot guarantee the coherence of the context in the model pre-training stage, and there will be data duplication problems, which will lead to poor performance of the model on complex tasks and overfitting problems. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of this, an embodiment of this specification provides a language model training method. One or more embodiments of this specification also relate to an information processing method, a language model training device, an information processing device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a language model training method is provided, including:

[0005] Obtaining a document set of an initial language model, and screening related documents from the document set to form a related document set;

[0006] Constructing a document relationship graph corresponding to the associated document set according to the associated documents in the associated document set and the similarities between the associated documents;

[0007] Determine a document splicing path in the document relationship graph, and extract target associated documents from the associated document set according to the document splicing path for splicing processing to obtain target context information;

[0008] The initial language model is trained based on the target context information until a target language model that meets a training stop condition is obtained.

[0009] Optionally, the step of screening related documents from the document set to form a related document set includes:

[0010] Constructing a document embedding vector corresponding to each initial document in the document collection using a text retrieval model;

[0011] Calculate the cosine similarity between the document embedding vectors corresponding to each initial document;

[0012] According to the cosine similarity calculation result, related documents are selected from the document collection to form a related document collection.

[0013] Optionally, after the step of constructing a document embedding vector corresponding to each initial document in the document collection by using a text retrieval model is performed, the method further includes:

[0014] Quantize the document embedding vectors corresponding to each initial document using a preset quantizer, and construct a document index based on the quantized document embedding vectors;

[0015] The document collection is searched according to a preset search strategy and the document index, and related documents are determined based on the search results to form a related document collection.

[0016] Optionally, constructing a document relationship graph corresponding to the associated document set according to the associated documents in the associated document set and the similarities between the associated documents includes:

[0017] Constructing graph nodes according to the associated documents in the associated document set, and constructing graph edges according to the similarities between the associated documents in the associated document set;

[0018] A connectivity relationship between the graph nodes is established based on the graph edges, and a document relationship graph corresponding to the associated document set is generated according to the connectivity relationship establishment result.

[0019] Optionally, determining a document splicing path in the document relationship graph includes:

[0020] Determine the i-th graph node in the document relationship graph, and construct an initial document path corresponding to the i-th graph node, wherein i is a positive integer, i starts at 1, and the maximum value of i is the number of graph nodes in the document relationship graph;

[0021] Selecting a j-th graph node from the neighboring graph nodes connected to the i-th graph node, and updating the initial document path to an intermediate document path based on the j-th graph node, wherein j is a positive integer, j starts at 1, and the maximum value of j is the number of neighboring graph nodes;

[0022] Determine whether there is an unvisited neighbor graph node among the neighbor graph nodes connected to the i-th graph node;

[0023] If not, j is incremented by 1, and the step of selecting the jth graph node from the neighboring graph nodes connected to the i-th graph node is executed;

[0024] If so, i is incremented by 1, and the step of determining the i-th graph node in the document relationship graph is executed until i is equal to the number of graph nodes, and multiple intermediate document paths are obtained, and the intermediate document paths that meet the document splicing conditions are extracted from the multiple intermediate document paths as the document splicing paths.

[0025] Optionally, extracting target associated documents from the associated document set according to the document splicing path and performing splicing processing to obtain target context information includes:

[0026] Determine a splicing graph node included in the document splicing path, and extract a target associated document from the associated document set based on the splicing graph node;

[0027] The target associated documents are spliced ​​according to the path direction information of the document splicing path to obtain target context information that matches the initial language model input condition.

[0028] Optionally, the training the initial language model based on the target context information until a target language model that satisfies a training stop condition is obtained includes:

[0029] Inputting the target context information into the initial language model for processing to obtain predicted text information;

[0030] Optimizing the initial language model based on the reference text information corresponding to the target context information and the predicted text information;

[0031] Check whether the optimized initial language model meets the training stop condition;

[0032] If not, screening candidate associated documents from the document set to form a candidate associated document set, taking the candidate associated document set as the associated document set, and executing the step of constructing a document relationship graph corresponding to the associated document set according to the associated documents in the associated document set and the similarities between the associated documents;

[0033] If so, the optimized initial language model is used as the target language model.

[0034] According to a second aspect of the embodiments of this specification, there is provided an information processing method, including:

[0035] Obtaining question information submitted by a client, and searching a business knowledge base for multiple answer documents based on the question information;

[0036] Splicing the multiple answer documents, and inputting the splicing result and the question information into a target language model for processing to obtain answer information, wherein the target language model is trained by the above method;

[0037] The answer information is fed back to the client.

[0038] According to a third aspect of the embodiments of this specification, a language model training device is provided, including:

[0039] An acquisition module is configured to acquire a document set of an initial language model, and select related documents from the document set to form a related document set;

[0040] A construction module, configured to construct a document relationship graph corresponding to the associated document set according to the associated documents in the associated document set and the similarities between the associated documents;

[0041] A splicing module is configured to determine a document splicing path in the document relationship graph, and extract target associated documents from the associated document set according to the document splicing path for splicing processing to obtain target context information;

[0042] The training module is configured to train the initial language model based on the target context information until a target language model that meets a training stop condition is obtained.

[0043] According to a fourth aspect of the embodiments of this specification, there is provided an information processing device, including:

[0044] An information acquisition module is configured to acquire question information submitted by a client, and query a plurality of answer documents in a business knowledge base based on the question information;

[0045] A model prediction module is configured to concatenate the multiple answer documents and input the concatenation result and the question information into a target language model for processing to obtain answer information, wherein the target language model is trained by the above method;

[0046] The feedback information module is configured to feed back the answer information to the client.

[0047] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0048] Memory and processor;

[0049] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned language model training method or information processing method are implemented.

[0050] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned language model training method or information processing method.

[0051] According to the seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of the above-mentioned language model training method or information processing method.

[0052] The language model training method provided in this embodiment can improve the performance of the language model in processing complex contexts and cross-document reasoning. After obtaining the document set of the initial language model, the document set can be screened for related documents with similarity to form a related document set; thereafter, in order to enhance the model's ability to understand cross-document texts, a document relationship graph corresponding to the related document set can be constructed based on the related documents in the related document set and the similarity between the related documents; the similarity between the related documents is reflected through the document relationship graph. On this basis, the document splicing path can be determined in the document relationship graph, and the target related documents can be extracted from the related document set according to the document splicing path for splicing processing to obtain the target context information; finally, the initial language model can be trained based on the target context information until the target language model that meets the training stop condition is obtained. The implementation model can be pre-trained on a sequence of related documents, so that the language model can read and reason outside the document boundary, thereby effectively improving the performance of the model in complex context tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a schematic diagram of a language model training method provided by an embodiment of this specification;

[0054] Figure 2 is a flow chart of a language model training method provided by an embodiment of this specification;

[0055] Figure 3 is a flow chart of an information processing method provided by an embodiment of this specification;

[0056] Figure 4 is a processing flow chart of an information processing method provided by an embodiment of this specification;

[0057] Figure 5 is a structural diagram of a language model training device provided by an embodiment of this specification;

[0058] Figure 6 is a structural diagram of an information processing device provided by an embodiment of this specification;

[0059] Figure 7 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0060] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0061] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0062] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0063] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0064] First, the terms involved in one or more embodiments of this specification are explained.

[0065] Large Language Model: LLM is an important technology in the field of natural language processing (NLP). It can generate natural language text or understand the meaning of language text by using a deep learning model trained with a large amount of text data.

[0066] A greedy algorithm is an algorithm that takes the best or optimal (i.e. most favorable) choice in the current state at each step, hoping to lead to the best or optimal result in the world. Every choice made by a greedy algorithm is a local optimal choice in a certain sense.

[0067] In this specification, a language model training method is provided. One or more embodiments of this specification also relate to an information processing method, a language model training device, an information processing device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0068] In practical applications, in order to make the language model provide better prediction performance, standard pre-training method (Standard Pretraining) and retrieval-based pre-training method (kNN Pretraining) are usually used to pre-train it. Among them, the standard pre-training method randomly shuffles the documents and splices them into a fixed-length context window for language modeling training. Although this method is simple, it cannot guarantee the coherence of the context, resulting in poor performance of the model when processing complex tasks. The retrieval-based pre-training method puts each document and its retrieved related documents in the same context for training. This method can improve the semantic understanding ability of the model to a certain extent, but there is a problem of data duplication, which may cause the model to overfit. Therefore, an effective solution is urgently needed to solve the above problems.

[0069] See also Figure 1As shown in the schematic diagram, the language model training method provided in this embodiment can improve the performance of the language model in processing complex contexts and cross-document reasoning. After obtaining the document set of the initial language model, related documents with similarity can be screened in the document set to form a related document set; thereafter, in order to enhance the model's ability to understand cross-document texts, a document relationship graph corresponding to the related document set can be constructed based on the related documents in the related document set and the similarity between the related documents; the similarity relationship between the related documents is reflected through the document relationship graph. On this basis, the document splicing path can be determined in the document relationship graph, and the target related documents can be extracted from the related document set according to the document splicing path for splicing processing to obtain the target context information; finally, the initial language model can be trained based on the target context information until the target language model that meets the training stop condition is obtained. The implementation model can be pre-trained on a related document sequence, so that the language model can read and reason outside the document boundary, thereby effectively improving the performance of the model in complex context tasks.

[0070] See also Figure 2 , Figure 2 A flow chart of a language model training method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0071] Step S202: obtaining a document set of an initial language model, and screening related documents from the document set to form a related document set.

[0072] The language model training method provided in this embodiment can be applied to the pre-training stage of a large language model of any structure. For example, the language model can be an LLaMA model (Large Language Model Meta AI, large-scale language model), etc. This embodiment does not make any limitations here.

[0073] Specifically, the initial language model specifically refers to the language model that needs to be pre-trained at the current stage. It can be a language model deployed in a question-answering interaction scenario. After the user submits the question information, it can query the relevant documents in the knowledge base and generate answer information matching the question information based on the relevant documents, thereby feeding back to the user the answer information that meets the user's question-answering needs. Correspondingly, the document collection specifically refers to the collection of documents required to be used by the initial language model in the pre-training stage. The documents contained in the collection can come from different data sources and can correspond to different business fields, such as mathematical knowledge fields, film knowledge fields, transaction knowledge fields, etc. This embodiment does not make any restrictions here. Correspondingly, the associated document specifically refers to the documents with similar relationships screened out from the document collection, which are used to form an associated document collection. It can be understood that the associated documents contained in the associated document collection are all similar, which is used to implement the subsequent pre-training of the language model so that it can learn the ability to read and reason outside the document boundary.

[0074] Based on this, in order to improve the performance of the language model in processing complex contexts and cross-document reasoning, after obtaining the document collection of the initial language model, the document collection can be screened for related documents with similarity to form a related document collection; thereafter, in order to enhance the model's ability to understand cross-document texts, the document relationship graph corresponding to the related document collection can be constructed based on the related documents in the related document collection and the similarity between the related documents; the similarity between the related documents can be reflected through the document relationship graph. On this basis, the document splicing path can be determined in the document relationship graph, and the target related documents can be extracted from the related document collection according to the document splicing path for splicing processing to obtain the target context information; finally, the initial language model can be trained based on the target context information until the target language model that meets the training stop condition is obtained.

[0075] Furthermore, after obtaining the document set of the initial language model, in order to select documents with high similarity to form a related document set, thereby improving the model's ability to understand cross-documents, the related documents can be screened by calculating cosine similarity. In this embodiment, the specific implementation is as follows:

[0076] The document embedding vectors corresponding to the initial documents in the document collection are constructed by using a text retrieval model; the cosine similarity between the document embedding vectors corresponding to the initial documents is calculated; and the related documents are screened in the document collection according to the cosine similarity calculation result to form a related document collection.

[0077] Specifically, the text retrieval model specifically refers to a model that can construct an embedding vector for each initial document, which can be a Contriever model, or other models that can construct an embedding vector can be selected, and this embodiment does not make any limitation here. Correspondingly, the document embedding vector specifically refers to the vector expression obtained after embedding the initial document using the text retrieval model. Correspondingly, the cosine similarity specifically refers to the value obtained after calculating the similarity of each initial text according to the document embedding vector.

[0078] Based on this, after determining the document set that needs to be pre-trained for the initial language model, in order to screen out similar and non-duplicate documents for model pre-training, the construction of the associated document set can be completed by calculating the cosine similarity. In this process, the text retrieval model can be used to first construct the document embedding vectors corresponding to each initial document in the document set; on this basis, the cosine similarity between the document embedding vectors corresponding to each initial document can be calculated; thereafter, the cosine similarity calculation result can be compared with the preset similarity threshold, and documents greater than the similarity threshold are selected as associated documents, and the associated document set is formed, which can ensure that the associated documents contained in the associated document set are documents with higher similarity, thereby improving the contextual understanding and reasoning capabilities of the model.

[0079] In specific implementation, when the Contriever model is used to embed documents in a document collection, each document di can be first mapped to an embedding vector E(di) through the Contriever model, and then the similarity between documents is calculated by using cosine similarity, which can be achieved by the formula s(di,dj)=cos(E(di),E(dj)), where di and dj represent two documents whose similarity needs to be calculated.

[0080] In summary, by calculating the cosine similarity to screen similar documents to form a related document set, similar documents can be used to complete the pre-training process in the model pre-training stage, so that the model can have strong prediction capabilities based on cross-document text processing.

[0081] In addition, in order to improve the efficiency of building the associated document set, the construction of the associated document set can be completed by using the approximate nearest neighbor search and the inverted file index. In this embodiment, the specific implementation is as follows:

[0082] The document embedding vectors corresponding to each initial document are quantized using a preset quantizer, and a document index is constructed based on the quantized document embedding vectors; the document collection is searched according to a preset search strategy and the document index, and related documents are determined based on the search results and a related document collection is formed.

[0083] Specifically, the quantizer specifically refers to a unit that quantizes the document embedding vector of each initial document. Correspondingly, the document index specifically refers to index information for searching similar documents based on the quantized document embedding vector. Correspondingly, the search strategy specifically refers to a strategy for setting a search for k similar documents.

[0084] Based on this, in order to achieve the purpose of efficient retrieval and quickly determine the documents with higher similarity to form an associated document set, the document embedding vectors corresponding to each initial document can be quantized using a preset quantizer to build a document index based on the quantized document embedding vectors; on this basis, the document collection can be searched according to the preset search strategy and document index, and then the associated documents with higher similarity can be determined based on the search results to form an associated document set, so that they can be used for pre-training of the initial language model later.

[0085] In practical applications, when using approximate nearest neighbor search and inverted file index (FAISS) to build a collection of related documents, you can first build a FAISS index, and use the product quantization method to quantize and index the document embedding vector. Then perform similarity retrieval. During this process, you can perform an approximate nearest neighbor search in the FAISS index to quickly find the top k most similar documents for each document, and then form a collection of related documents for use.

[0086] In the specific implementation, first, the product quantization of the document embedding vector can be performed; specifically, configure the PQ quantizer: use the faiss.ProductQuantizer class in the FAISS library to configure the product quantizer. It is necessary to specify the dimension of the quantization center (dsub) and the number of sub-quantizers (M), so that the entire vector space can be divided into multiple subspaces, and each subspace is quantized independently. Train the PQ quantizer: Use a set of training data (which can be part of the entire data set) to train the PQ quantizer. The training process will find the optimal quantization center for each subspace. Quantize document embedding: Quantize the document's embedding vector through the trained PQ quantizer to obtain the quantized encoding.

[0087] Secondly, build the FAISS index; specifically, select the index type: the index based on product quantization usually uses the IVFPQ (Inverted File with Product Quantization) index type in FAISS. This index combines the inverted index (IVF) and product quantization (PQ) to achieve efficient approximate nearest neighbor search. Create an index: Use the faiss.IndexIVFPQ class of FAISS to create an index. You need to specify the quantizer (that is, the PQ quantizer trained in step 1), the number of cluster centers in the inverted list (nlist), and the number of nearest neighbors retained under each cluster center. Add data to the index: Add the quantized document embedding vector to the created FAISS index.

[0088] Finally, perform an approximate nearest neighbor search; specifically, set the search parameters: before searching, set the nprobe parameter as needed, which determines the number of nearest neighbors to be explored under each cluster center during the search, thereby affecting the search speed and accuracy. Perform a search: Use FAISS's search function (such as index.search) to perform an approximate nearest neighbor search on a given query vector. The query vector can be any new document embedding vector, which also needs to be quantized by the PQ quantizer. Get results: The search function will return the index and distance of the top k most similar documents for each query vector. This constructs a collection of associated documents for pre-training of the initial language model.

[0089] For example, determine the document set corresponding to the pre-trained language model - LLaMA model, and the document set contains m documents. By using the Contriever model to embed each document contained in the document set, the embedding vector E corresponding to each document is obtained. Then, by calculating the cosine similarity between the embedding vectors E corresponding to each document, according to the cosine similarity calculation result, n documents with similarity are selected from the m documents to form an associated document set (n<m). For example, the n documents can be document 1, document 2... document n of associated image processing knowledge. The LLaMA model can be pre-trained in the future so that the model can have better context understanding and reasoning capabilities.

[0090] In summary, retrieval by building an index to search for similar documents can improve the efficiency of building related documents, thereby saving model pre-training time.

[0091] Step S204: construct a document relationship graph corresponding to the associated document set according to the associated documents in the associated document set and the similarities between the associated documents.

[0092] Specifically, after determining the associated document set composed of the associated documents with correlation as described above, in order to further select highly relevant documents for model pre-training, the strength of the correlation between documents can be reflected by constructing a document relationship graph, and then the highly relevant documents can be screened for model pre-training through the document relationship graph. Therefore, the document relationship graph corresponding to the associated document set can be constructed based on the associated documents in the associated document set and the similarity between the associated documents, for subsequent use. Among them, the document relationship graph specifically refers to an undirected weighted graph constructed by using associated documents as graph nodes and the similarity between associated documents as graph edges.

[0093] Furthermore, when constructing the document relationship graph, considering that the document relationship graph will be used as the basis for constructing the model input context information in the future, it is necessary to make full use of the documents and the similarities between the documents to complete the construction. In this embodiment, the specific implementation method is as follows:

[0094] Graph nodes are constructed according to the associated documents in the associated document set, and graph edges are constructed according to the similarities between the associated documents in the associated document set; connectivity relationships between the graph nodes are established based on the graph edges, and a document relationship graph corresponding to the associated document set is generated according to the connectivity relationship establishment result.

[0095] Based on this, when constructing a document relationship graph, you can first draw graph nodes according to the associated documents in the associated document set, and draw graph edges according to the similarity between the associated documents in the associated document set. When drawing the graph edges, the weight of the edge is determined by the similarity between the associated documents. Then, you can combine the graph edges to establish the connectivity relationship between the graph nodes. On this basis, you can generate a document relationship graph corresponding to the associated document set based on the results of the connectivity relationship establishment, so that you can subsequently screen and splice highly relevant documents.

[0096] Continuing with the above example, after obtaining n documents related to image processing knowledge, n graph nodes can be drawn based on each document. At the same time, graph edges can be constructed based on the similarities between each of the n documents. On this basis, the n graph nodes can be connected based on the graph edges to obtain an undirected weighted graph, so that a set number of documents can be screened out for splicing by subsequently analyzing the undirected weighted graph, thereby pre-training the LLaMA model.

[0097] In summary, by constructing a document relationship graph to reflect the relevance between related documents, it is convenient to subsequently screen out highly relevant documents for splicing and use, thereby improving the pre-training effect of the initial language model.

[0098] Step S206, determining a document splicing path in the document relationship graph, and extracting target associated documents from the associated document set according to the document splicing path for splicing processing to obtain target context information.

[0099] Specifically, after constructing the document relationship graph as described above, since the document relationship graph can reflect the strength of the correlation between each related document, the document splicing path can be found in the document relationship graph by traversing the document relationship graph. The highly relevant related documents can be determined through the document splicing path. At this time, the target related documents can be extracted from the related document collection according to the document splicing path for splicing processing, and then the target context information that can be input into the model for processing can be obtained. On this basis, the language model is trained using highly relevant documents, so that the model can learn context understanding and reasoning capabilities from related document sequences.

[0100] Among them, the document splicing path specifically refers to traversing the document relationship graph, and finding a link consisting of nodes corresponding to documents with higher relevance according to the traversal results. The path consists of multiple graph nodes, and the associated documents corresponding to each graph node are highly relevant documents. Correspondingly, the target associated document is the associated document involved in the document splicing path. Correspondingly, the target context information specifically refers to the text obtained after splicing the target associated documents associated with the document splicing path.

[0101] Furthermore, when determining the document splicing path in the document relationship graph, considering that the path will determine the relevance between similar documents in the target context information, in order to ensure that the target context information is constructed using strongly associated documents, thereby improving the model's cross-document understanding and reasoning capabilities, a greedy algorithm can be used to determine the document splicing path. In this embodiment, the specific implementation method is as follows:

[0102] Determine the i-th graph node in the document relationship graph, and construct an initial document path corresponding to the i-th graph node, wherein i is a positive integer, i starts at 1, and the maximum value of i is the number of graph nodes in the document relationship graph; select the j-th graph node from the neighbor graph nodes connected to the i-th graph node, and update the initial document path to an intermediate document path based on the j-th graph node, wherein j is a positive integer, j starts at 1, and the maximum value of j is the number of neighbor graph nodes; determine whether there are unvisited neighbor graph nodes in the neighbor graph nodes connected to the i-th graph node; if not, j is incremented by 1, and the step of selecting the j-th graph node in the neighbor graph nodes connected to the i-th graph node is executed; if so, i is incremented by 1, and the step of determining the i-th graph node in the document relationship graph is executed, until i is equal to the number of graph nodes, a plurality of intermediate document paths are obtained, and the intermediate document path that meets the document splicing condition is extracted from the plurality of intermediate document paths as the document splicing path.

[0103] Specifically, the i-th graph node specifically refers to the graph node with the least number of edges in the document relationship graph, and the initial document path specifically refers to the initial path constructed with the i-th graph node as the path starting point; accordingly, the j-th graph node specifically refers to the unvisited neighbor node among the neighbor nodes of the i-th graph node. Accordingly, the intermediate document path specifically refers to the path obtained after adding the j-th graph node to the initial document path.

[0104] Based on this, when determining the document splicing path, in order to be able to mine the path composed of nodes corresponding to highly correlated documents from the document relationship graph, we can determine the i-th graph node with the lowest degree in the document relationship graph from the graph dimension, and construct the initial document path corresponding to the i-th graph node, where i is a positive integer, i starts from 1, and the maximum value of i is the number of graph nodes in the document relationship graph.

[0105] Then, the jth graph node that has not been visited can be selected from the neighbor graph nodes connected to the i-th graph node, and the jth graph node can be added to the initial document path to obtain the intermediate document path, where j is a positive integer, j starts from 1, and the maximum value of j is the number of neighbor graph nodes.

[0106] On this basis, it can be determined whether there are unvisited neighbor graph nodes in the neighbor graph nodes connected to the i-th graph node; if not, it means that the document path corresponding to the i-th graph node can still be expanded, so j can be incremented by 1, and the step of selecting the j-th graph node in the neighbor graph nodes connected to the i-th graph node is executed; if so, it means that the expansion path of the i-th graph node has reached the upper limit, so other lowest degree nodes except the i-th graph node can be selected for path construction, so i can be incremented by 1, and the step of determining the i-th graph node in the document relationship graph is executed. Until i is equal to the number of graph nodes, it means that the paths corresponding to all graph nodes in the document relationship graph have been constructed, so multiple intermediate document paths can be obtained, and then the intermediate document paths that meet the document splicing conditions can be extracted from the multiple intermediate document paths as the document splicing path. In specific implementation, when selecting the document splicing path, the longest intermediate document path can be selected as the document splicing path.

[0107] In practical applications, considering that the document relationship graph is an undirected weighted graph, in order to fully mine the undirected weighted graph on this basis, and then quickly and accurately determine the target related documents by analyzing the graph, the greedy algorithm can be used to solve the maximum traveling salesman problem to mine the maximum traveling salesman path (document splicing path). In other words, the path can be initialized first: starting from the node with the lowest degree (the node with the smallest number of edge connections to other nodes), the initial path is constructed; then the path is extended: by selecting the highest weighted and unvisited neighbor node of the current node, it is added to the initial path, thereby updating the initial path; and so on, when all neighbor nodes have been visited, the current path construction is completed. At this time, the next lowest degree node can be selected, and the path extension is continued until all nodes have completed the path construction. The maximum traveling salesman path, that is, the document splicing path, can be determined from the constructed path. On this basis, documents can be selected for splicing according to the document splicing path to obtain the target context information used for model pre-training.

[0108] In summary, by using a greedy algorithm to solve the maximum traveling salesman path problem, it is possible to maximize the continuous access to related documents on the document relationship graph, thereby improving the coherence of the input context.

[0109] Furthermore, after obtaining the document splicing path, since the path can not only reflect the required document content but also has directionality, the target associated documents can be spliced ​​according to the direction information. In this embodiment, the specific implementation method is as follows:

[0110] Determine the splicing graph nodes included in the document splicing path, extract the target associated document from the associated document set based on the splicing graph nodes; splice the target associated document according to the path direction information of the document splicing path, and obtain the target context information that matches the initial language model input condition.

[0111] Specifically, the splicing graph node specifically refers to the graph node included in the document splicing path. Correspondingly, the path direction information specifically refers to the information that characterizes the relationship before and after document splicing. Based on this, after obtaining the document splicing path, the splicing graph nodes included in the document splicing path can be determined; on this basis, the target associated documents can be extracted from the associated document set based on the splicing graph nodes; and then, in order to ensure the consistency of the splicing results, the target associated documents can be spliced ​​according to the path direction information of the document splicing path, and then the target context information that matches the input conditions of the initial language model can be obtained according to the splicing results, so as to be used for pre-training the initial language model later.

[0112] Continuing with the above example, after obtaining an undirected weighted graph based on n documents, the greedy algorithm can be further used to process the undirected weighted graph to mine the maximum weight path. Specifically, the graph node with the lowest degree can be selected in the undirected weighted graph, and then used as the starting node to construct the initial path. On this basis, the highest weighted and unvisited neighbor node of the graph node can be selected and added to the initial path to construct an intermediate path. And so on, until all neighbor nodes are visited, the graph node with the lowest degree other than the graph node is selected for path mining. Finally, by traversing each node in the graph, the maximum weight path can be generated, which contains document 1, document 2, document 3 and document 5. On this basis, documents 1, document 2, document 3 and document 5 are spliced ​​along the maximum weight path to obtain a fixed-size input context for subsequent model pre-training.

[0113] In summary, since the document splicing path is obtained based on document relationship graph mining, when document splicing is performed based on this path to generate context information, the coherence of the context can be guaranteed, and the uniqueness of each target-related document can also be guaranteed, thereby solving the problem of data duplication and improving the model training effect.

[0114] Step S208: training the initial language model based on the target context information until a target language model that meets a training stop condition is obtained.

[0115] Specifically, after obtaining the target context information by splicing the target related documents as mentioned above, further, since the target context information is spliced ​​based on highly relevant related documents, the initial language model is trained based on the target context information until the target language model that meets the training stop conditions is obtained. This can effectively improve the context understanding and reasoning capabilities of the target language model, thereby significantly improving the performance of the target language model in complex tasks.

[0116] Furthermore, in the model training process, in order to obtain a target language model that meets the deployment requirements, model detection can be performed during the training phase. In this embodiment, the specific implementation is as follows:

[0117] The target context information is input into the initial language model for processing to obtain predicted text information; the initial language model is optimized based on the benchmark text information and the predicted text information corresponding to the target context information; whether the optimized initial language model satisfies the training stop condition is detected; if not, candidate associated documents are screened in the document set to form a candidate associated document set, and the candidate associated document set is used as the associated document set, and a document relationship graph corresponding to the associated document set is constructed according to the associated documents in the associated document set and the similarities between the associated documents; if so, the optimized initial language model is used as the target language model.

[0118] Specifically, the predicted text information refers to the predicted information obtained after the initial language model processes the target context information. Correspondingly, the reference text information refers to the real information corresponding to the target context information. By optimizing the model with the predicted text information and the reference text information, the model can learn the knowledge required for the training purpose, thereby meeting the business deployment requirements.

[0119] Based on this, after obtaining the target context information, the target context information can be input into the initial language model for processing, and the text information can be predicted according to the processing result; thereafter, the benchmark text information corresponding to the target context information can be obtained first, and the initial language model can be further optimized in combination with the benchmark text information and the predicted text information; after the optimization is completed, it can be detected whether the optimized initial language model meets the training stop condition; if not, it means that the current optimized initial language model cannot meet the deployment condition, so candidate associated documents can be screened in the document collection to form a candidate associated document set, and the candidate associated document set is used as the associated document set, and the document relationship graph corresponding to the associated document set is constructed according to the associated documents in the associated document set and the similarity between the associated documents, until a model that meets the conditions is obtained, the optimized initial language model can be used as the target language model to be deployed in business scenarios.

[0120] In practical applications, when the generated target context information is used to pre-train the initial language model, the trained initial language model can adopt the same model architecture as the existing pre-training pipeline, and the AdamW optimizer and cosine learning rate scheduling can be used to complete the model optimization; the context understanding and reasoning capabilities of the model can also be improved by increasing the length of the context information in the training batch.

[0121] Using the above example, after obtaining the context information of Document 1, Document 2, Document 3 and Document 5, the LLaMA model can be pre-trained based on the context information. A trained LLaMA model can have better cross-document understanding and reasoning capabilities. For example, when a user submits a question {What steps does image sharpening include}, after inputting the question into the trained LLaMA model, the LLaMA model can first determine multiple documents associated with the question, and then generate an answer based on the question raised by the user {1. Select an image; 2. Select a sharpening method; 3. Apply a sharpening filter...}, and finally feedback the answer to the user.

[0122] The language model training method provided in this embodiment can improve the performance of the language model in processing complex contexts and cross-document reasoning. After obtaining the document set of the initial language model, the document set can be screened for related documents with similarity to form a related document set; thereafter, in order to enhance the model's ability to understand cross-document texts, a document relationship graph corresponding to the related document set can be constructed based on the related documents in the related document set and the similarity between the related documents; the similarity between the related documents is reflected through the document relationship graph. On this basis, the document splicing path can be determined in the document relationship graph, and the target related documents can be extracted from the related document set according to the document splicing path for splicing processing to obtain the target context information; finally, the initial language model can be trained based on the target context information until the target language model that meets the training stop condition is obtained. The implementation model can be pre-trained on a sequence of related documents, so that the language model can read and reason outside the document boundary, thereby effectively improving the performance of the model in complex context tasks.

[0123] See also Figure 3 , Figure 3 A flowchart of an information processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0124] Step S302, obtaining question information submitted by the client, and searching a business knowledge base for multiple answer documents based on the question information;

[0125] Step S304, concatenating the multiple answer documents, and inputting the concatenation result and the question information into a target language model for processing to obtain answer information, wherein the target language model is trained by the above method;

[0126] Step S306: Feedback the answer information to the client.

[0127] The description of the target language model in the information processing method provided in this embodiment can refer to the description in the above embodiments, and this embodiment will not be described in detail here.

[0128] The following combination Figure 4 , taking the application of the information processing method and language model training method provided in this specification in the question-answering interaction scenario as an example, the information processing method and language model training method are further explained. Among them, Figure 4 A processing flow chart of an information processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0129] Step S402, obtaining a document set of an initial language model, and constructing a document embedding vector corresponding to each initial document in the document set using a text retrieval model.

[0130] Step S404, calculating the cosine similarity between the document embedding vectors corresponding to the initial documents.

[0131] Step S406: Filter related documents in the document set according to the cosine similarity calculation result to form a related document set.

[0132] Step S408: constructing graph nodes according to the associated documents in the associated document set, and constructing graph edges according to the similarities between the associated documents in the associated document set.

[0133] Step S410, establishing a connectivity relationship between the graph nodes based on the graph edges, and generating a document relationship graph corresponding to the associated document set according to the connectivity relationship establishment result.

[0134] Step S412: using a greedy algorithm to determine a document splicing path in the document relationship graph.

[0135] Step S414: determine the splicing graph nodes included in the document splicing path, and extract target associated documents from the associated document set based on the splicing graph nodes.

[0136] Step S416, splicing the target associated documents according to the path direction information of the document splicing path to obtain target context information matching the initial language model input condition.

[0137] Step S418: Train the initial language model based on the target context information until a target language model that meets the training stop condition is obtained, and deploy the target language model.

[0138] Step S420, when question information submitted by the client is received, multiple answer documents are searched in the business knowledge base based on the question information;

[0139] Step S422, concatenate multiple answer documents, and input the concatenation result and question information into the target language model for processing to obtain answer information.

[0140] Step S424, feeding back the answer information to the client.

[0141] In summary, in order to improve the performance of the language model in processing complex contexts and cross-document reasoning, after obtaining the document collection of the initial language model, the document collection can be screened for related documents with similarity to form a related document collection; thereafter, in order to enhance the model's ability to understand cross-document texts, the document relationship graph corresponding to the related document collection can be constructed based on the related documents in the related document collection and the similarity between the related documents; the similarity between the related documents can be reflected through the document relationship graph. On this basis, the document splicing path can be determined in the document relationship graph, and the target related documents can be extracted from the related document collection according to the document splicing path for splicing processing to obtain the target context information; finally, the initial language model can be trained based on the target context information until the target language model that meets the training stop condition is obtained. The implementation model can be pre-trained on the relevant document sequence, so that the language model can read and reason outside the document boundary, thereby effectively improving the performance of the model in complex context tasks.

[0142] Corresponding to the above method embodiment, this specification also provides a language model training device embodiment, Figure 5 FIG. 1 is a schematic diagram showing the structure of a language model training device provided by an embodiment of the present specification. Figure 5 As shown, the device comprises:

[0143] The acquisition module 502 is configured to acquire a document set of an initial language model, and select related documents from the document set to form a related document set;

[0144] A construction module 504 is configured to construct a document relationship graph corresponding to the associated document set according to the associated documents in the associated document set and the similarities between the associated documents;

[0145] A splicing module 506 is configured to determine a document splicing path in the document relationship graph, and extract target associated documents from the associated document set according to the document splicing path for splicing processing to obtain target context information;

[0146] The training module 508 is configured to train the initial language model based on the target context information until a target language model that meets a training stop condition is obtained.

[0147] In an optional embodiment, the acquisition module 502 is further configured to:

[0148] The document embedding vectors corresponding to the initial documents in the document collection are constructed by using a text retrieval model; the cosine similarity between the document embedding vectors corresponding to the initial documents is calculated; and the related documents are screened in the document collection according to the cosine similarity calculation result to form a related document collection.

[0149] In an optional embodiment, the acquisition module 502 is further configured to:

[0150] The document embedding vectors corresponding to each initial document are quantized using a preset quantizer, and a document index is constructed based on the quantized document embedding vectors; the document collection is searched according to a preset search strategy and the document index, and related documents are determined based on the search results and a related document collection is formed.

[0151] In an optional embodiment, the construction module 504 is further configured to:

[0152] Graph nodes are constructed according to the associated documents in the associated document set, and graph edges are constructed according to the similarities between the associated documents in the associated document set; connectivity relationships between the graph nodes are established based on the graph edges, and a document relationship graph corresponding to the associated document set is generated according to the connectivity relationship establishment result.

[0153] In an optional embodiment, the splicing module 506 is further configured to:

[0154] Determine the i-th graph node in the document relationship graph, and construct an initial document path corresponding to the i-th graph node, wherein i is a positive integer, i starts at 1, and the maximum value of i is the number of graph nodes in the document relationship graph; select the j-th graph node from the neighbor graph nodes connected to the i-th graph node, and update the initial document path to an intermediate document path based on the j-th graph node, wherein j is a positive integer, j starts at 1, and the maximum value of j is the number of neighbor graph nodes; determine whether there are unvisited neighbor graph nodes in the neighbor graph nodes connected to the i-th graph node; if not, j is incremented by 1, and the step of selecting the j-th graph node in the neighbor graph nodes connected to the i-th graph node is executed; if so, i is incremented by 1, and the step of determining the i-th graph node in the document relationship graph is executed, until i is equal to the number of graph nodes, a plurality of intermediate document paths are obtained, and the intermediate document path that meets the document splicing condition is extracted from the plurality of intermediate document paths as the document splicing path.

[0155] In an optional embodiment, the splicing module 506 is further configured to:

[0156] Determine the splicing graph nodes included in the document splicing path, extract the target associated document from the associated document set based on the splicing graph nodes; splice the target associated document according to the path direction information of the document splicing path, and obtain the target context information that matches the initial language model input condition.

[0157] In an optional embodiment, the training module 508 is further configured to:

[0158] The target context information is input into the initial language model for processing to obtain predicted text information; the initial language model is optimized based on the benchmark text information and the predicted text information corresponding to the target context information; whether the optimized initial language model satisfies the training stop condition is detected; if not, candidate associated documents are screened in the document set to form a candidate associated document set, and the candidate associated document set is used as the associated document set, and a document relationship graph corresponding to the associated document set is constructed according to the associated documents in the associated document set and the similarities between the associated documents; if so, the optimized initial language model is used as the target language model.

[0159] The above is a schematic scheme of a language model training device of this embodiment. It should be noted that the technical scheme of the language model training device and the technical scheme of the above-mentioned language model training method belong to the same concept, and the details not described in detail in the technical scheme of the language model training device can be referred to the description of the technical scheme of the above-mentioned language model training method.

[0160] Corresponding to the above method embodiment, this specification also provides an information processing device embodiment, Figure 6 FIG. 1 shows a schematic diagram of the structure of an information processing device provided by an embodiment of the present specification. Figure 6 As shown, the device comprises:

[0161] The information acquisition module 602 is configured to acquire question information submitted by the client, and query a plurality of answer documents in the business knowledge base based on the question information;

[0162] The model prediction module 604 is configured to concatenate the multiple answer documents and input the concatenation result and the question information into a target language model for processing to obtain answer information, wherein the target language model is trained by the above method;

[0163] The feedback information module 606 is configured to feed back the answer information to the client.

[0164] The above is a schematic scheme of an information processing device of this embodiment. It should be noted that the technical scheme of the information processing device and the technical scheme of the above information processing method belong to the same concept, and the details not described in detail in the technical scheme of the information processing device can be referred to the description of the technical scheme of the above information processing method.

[0165] Figure 7 The block diagram of a computing device 700 according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0166] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0167] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0168] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

[0169] Among them, the processor 720 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned language model training method or information processing method.

[0170] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned language model training method or information processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned language model training method or information processing method.

[0171] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned language model training method or information processing method.

[0172] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned language model training method or information processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned language model training method or information processing method.

[0173] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the above-mentioned language model training method or information processing method.

[0174] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned language model training method or information processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned language model training method or information processing method.

[0175] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0176] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0177] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0178] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0179] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well.

Claims

1. A language model training method, characterized in that: include: Obtaining a document set of an initial language model, and screening related documents from the document set to form a related document set; Constructing a document relationship graph corresponding to the associated document set according to the associated documents in the associated document set and the similarities between the associated documents; Determine a document splicing path in the document relationship graph, and extract target associated documents from the associated document set according to the document splicing path for splicing processing to obtain target context information; The initial language model is trained based on the target context information until a target language model that meets a training stop condition is obtained.

2. The language model training method according to claim 1, characterized in that: The step of selecting related documents from the document set to form a related document set includes: Constructing a document embedding vector corresponding to each initial document in the document collection using a text retrieval model; Calculate the cosine similarity between the document embedding vectors corresponding to each initial document; According to the cosine similarity calculation result, related documents are selected from the document collection to form a related document collection.

3. The language model training method according to claim 2, characterized in that: After the step of using the text retrieval model to construct the document embedding vector corresponding to each initial document in the document collection is executed, the method further includes: Quantize the document embedding vectors corresponding to each initial document using a preset quantizer, and construct a document index based on the quantized document embedding vectors; The document collection is searched according to a preset search strategy and the document index, and related documents are determined based on the search results to form a related document collection.

4. The language model training method according to claim 1, characterized in that: The step of constructing a document relationship graph corresponding to the associated document set according to the associated documents in the associated document set and the similarities between the associated documents includes: Constructing graph nodes according to the associated documents in the associated document set, and constructing graph edges according to the similarities between the associated documents in the associated document set; A connectivity relationship between the graph nodes is established based on the graph edges, and a document relationship graph corresponding to the associated document set is generated according to the connectivity relationship establishment result.

5. The language model training method according to claim 1, characterized in that: Determining the document splicing path in the document relationship graph includes: Determine the i-th graph node in the document relationship graph, and construct an initial document path corresponding to the i-th graph node, wherein i is a positive integer, i starts at 1, and the maximum value of i is the number of graph nodes in the document relationship graph; Selecting a j-th graph node from the neighboring graph nodes connected to the i-th graph node, and updating the initial document path to an intermediate document path based on the j-th graph node, wherein j is a positive integer, j starts at 1, and the maximum value of j is the number of neighboring graph nodes; Determine whether there is an unvisited neighbor graph node among the neighbor graph nodes connected to the i-th graph node; If not, j is incremented by 1, and the step of selecting the jth graph node from the neighboring graph nodes connected to the i-th graph node is executed; If so, i is incremented by 1, and the step of determining the i-th graph node in the document relationship graph is executed until i is equal to the number of graph nodes, and multiple intermediate document paths are obtained, and the intermediate document paths that meet the document splicing conditions are extracted from the multiple intermediate document paths as the document splicing paths.

6. The language model training method according to claim 1, characterized in that: The step of extracting target associated documents from the associated document set according to the document splicing path and performing splicing processing to obtain target context information includes: Determine a splicing graph node included in the document splicing path, and extract a target associated document from the associated document set based on the splicing graph node; The target associated documents are spliced ​​according to the path direction information of the document splicing path to obtain target context information that matches the initial language model input condition.

7. The language model training method according to claim 1, characterized in that: The training of the initial language model based on the target context information until a target language model that satisfies a training stop condition is obtained includes: Inputting the target context information into the initial language model for processing to obtain predicted text information; Optimizing the initial language model based on the reference text information corresponding to the target context information and the predicted text information; Check whether the optimized initial language model meets the training stop condition; If not, screening candidate associated documents from the document set to form a candidate associated document set, taking the candidate associated document set as the associated document set, and executing the step of constructing a document relationship graph corresponding to the associated document set according to the associated documents in the associated document set and the similarities between the associated documents; If so, the optimized initial language model is used as the target language model.

8. An information processing method, characterized in that: include: Obtaining question information submitted by a client, and searching a business knowledge base for multiple answer documents based on the question information; Splicing the multiple answer documents, and inputting the splicing result and the question information into a target language model for processing to obtain answer information, wherein the target language model is trained by the method according to any one of claims 1 to 7; The answer information is fed back to the client.

9. A language model training device, characterized in that: include: An acquisition module is configured to acquire a document set of an initial language model, and select related documents from the document set to form a related document set; A construction module, configured to construct a document relationship graph corresponding to the associated document set according to the associated documents in the associated document set and the similarities between the associated documents; A splicing module is configured to determine a document splicing path in the document relationship graph, and extract target associated documents from the associated document set according to the document splicing path for splicing processing to obtain target context information; The training module is configured to train the initial language model based on the target context information until a target language model that meets a training stop condition is obtained.

10. An information processing device, characterized in that: include: An information acquisition module is configured to acquire question information submitted by a client, and query a plurality of answer documents in a business knowledge base based on the question information; A model prediction module, configured to concatenate the multiple answer documents, and input the concatenation result and the question information into a target language model for processing to obtain answer information, wherein the target language model is trained by the method according to any one of claims 1 to 7; The feedback information module is configured to feed back the answer information to the client.

11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 8 are implemented.

12. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 8.

13. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Document retrieval method and device based on large language model and graph network model

    CN116431837A

  • Question and answer data generation method and device, computer equipment and storage medium

    CN117493508A

  • Enhanced searching using fine-tuned machine learning models

    US20240281446A1

Cited By

  • Data processing method and device, electronic equipment and readable storage medium

    CN120763335A

  • Language model training method and apparatus, and information processing method and apparatus

    WO2026045027A1