Language model training method and apparatus, and information processing method and apparatus

Abstract

Provided are a language model training method and apparatus, and an information processing method and apparatus. The language model training method comprises: acquiring a document collection of an initial language model, and selecting associated documents from the document collection to form an associated document collection (S202); on the basis of the associated documents in the associated document collection and the similarity between the associated documents, constructing a document relationship graph corresponding to the associated document collection (S204); determining a document concatenation path in the document relationship graph, and on the basis of the document concatenation path, extracting target associated documents from the associated document collection and performing concatenation processing on the extracted target associated documents, in order to obtain target context information (S206); and on the basis of the target context information, training the initial language model until a target language model meeting a training stopping condition is obtained (S208).

Description

Language model training methods and devices, information processing methods and devices

[0001] This application claims priority to Chinese Patent Application No. 202411191440.X, filed on August 28, 2024, entitled "Language Model Training Method and Apparatus, Information Processing Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments in this specification relate to the field of machine learning technology, and in particular to language model training methods and apparatus, and information processing methods and apparatus. Background Technology

[0003] With the development of computer and internet technologies, large language models are being applied in an increasing number of scenarios, and their capabilities are gradually being enhanced to meet diverse user needs. Currently, large language models deployed in business applications are primarily implemented through pre-training on large-scale document data. When performing document prediction tasks, these models typically only utilize contextual information from document prefixes. However, as the model scale increases and application scenarios diversify, large language models still face numerous challenges when handling complex cross-document reasoning, instruction execution, and reading comprehension tasks; for example, the model may not accurately follow instructions, has limited reasoning ability on conditional documents, and suffers from high variance in context learning. Existing technologies typically employ standard pre-training methods or retrieval-based pre-training methods to improve model understanding to avoid these problems. However, both methods cannot guarantee contextual coherence during the model pre-training phase and suffer from data duplication, leading to poor model performance on complex tasks and overfitting issues. Therefore, an effective solution is urgently needed to address these problems. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a language model training method. One or more embodiments of this specification also relate to an information processing method, a language model training apparatus, an information processing device, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a language model training method is provided, comprising:

[0006] Obtain the document set of the initial language model, and filter related documents from the document set to form a related document set;

[0007] Based on the associated documents in the associated document set and the similarity between the associated documents, a document relationship graph corresponding to the associated document set is constructed;

[0008] The document splicing path is determined in the document relationship graph, and the target related document is extracted from the related document set according to the document splicing path and spliced ​​to obtain the target context information;

[0009] The initial language model is trained based on the target context information until a target language model that meets the training stopping condition is obtained.

[0010] Optionally, the step of filtering related documents from the document set to form a related document set includes:

[0011] The document embedding vectors corresponding to each initial document in the document set are constructed using a text retrieval model;

[0012] Calculate the cosine similarity between the document embedding vectors corresponding to each initial document;

[0013] Based on the cosine similarity calculation results, related documents are selected from the document set to form a related document set.

[0014] Optionally, after the step of constructing the document embedding vectors corresponding to each initial document in the document set using the text retrieval model is performed, the method further includes:

[0015] The document embedding vectors corresponding to each initial document are quantized using a preset quantizer, and a document index is constructed based on the quantized document embedding vectors.

[0016] The document set is searched according to the preset search strategy and the document index, and related documents are determined and formed into a related document set based on the search results.

[0017] Optionally, constructing a document relationship graph corresponding to the associated document set based on the associated documents in the associated document set and the similarity between the associated documents includes:

[0018] Graph nodes are constructed based on the associated documents in the associated document set, and graph edges are constructed based on the similarity between the associated documents in the associated document set;

[0019] Based on the graph edges, establish the connectivity relationships between the graph nodes, and generate a document relationship graph corresponding to the associated document set based on the results of establishing the connectivity relationships.

[0020] Optionally, determining the document concatenation path in the document relationship graph includes:

[0021] In the document relationship graph, determine the i-th graph node and construct the initial document path corresponding to the i-th graph node, where i is a positive integer, starting from 1, and the maximum value of i is the number of graph nodes in the document relationship graph;

[0022] Select the j-th graph node from the neighboring graph nodes connected to the i-th graph node, and update the initial document path to the intermediate document path based on the j-th graph node, where j is a positive integer, starting from 1, and the maximum value of j is the number of neighboring graph nodes;

[0023] Determine whether there are any unvisited neighboring graph nodes among the neighboring graph nodes connected to the i-th graph node;

[0024] If not, j is incremented by 1, and the step of selecting the j-th graph node from the neighboring graph nodes connected to the i-th graph node is performed;

[0025] If so, i is incremented by 1, and the step of determining the i-th graph node in the document relationship graph is executed until i is equal to the number of graph nodes. Multiple intermediate document paths are obtained, and the intermediate document path that satisfies the document splicing condition is extracted from the multiple intermediate document paths as the document splicing path.

[0026] Optionally, the step of extracting the target related document from the related document set according to the document splicing path and performing splicing processing to obtain the target context information includes:

[0027] Determine the splicing graph nodes contained in the document splicing path, and extract the target associated document from the associated document set based on the splicing graph nodes;

[0028] The target associated documents are concatenated according to the path direction information of the document concatenation path to obtain target context information that matches the input conditions of the initial language model.

[0029] Optionally, training the initial language model based on the target context information until a target language model that meets the training stopping condition is obtained includes:

[0030] The target context information is input into the initial language model for processing to obtain predicted text information;

[0031] The initial language model is optimized based on the baseline text information corresponding to the target context information and the predicted text information.

[0032] Check whether the optimized initial language model meets the training stopping condition;

[0033] If not, select candidate related documents from the document set to form a candidate related document set, and use the candidate related document set as the related document set, and perform the step of constructing a document relationship graph corresponding to the related document set based on the related documents in the related document set and the similarity between the related documents;

[0034] If so, the optimized initial language model will be used as the target language model.

[0035] According to a second aspect of the embodiments of this specification, an information processing method is provided, comprising:

[0036] Obtain the question information submitted by the client, and query multiple answer documents in the business knowledge base based on the question information;

[0037] The multiple answer documents are concatenated, and the concatenation result and the question information are input into a target language model for processing to obtain the answer information. The target language model is trained using the above method.

[0038] The answer information is then sent back to the client.

[0039] According to a third aspect of the embodiments of this specification, a language model training apparatus is provided, comprising:

[0040] The acquisition module is configured to acquire a document set of the initial language model and filter related documents in the document set to form a related document set;

[0041] The construction module is configured to construct a document relationship graph corresponding to the associated document set based on the associated documents in the associated document set and the similarity between the associated documents;

[0042] The splicing module is configured to determine the document splicing path in the document relationship graph, and extract the target related document in the related document set according to the document splicing path for splicing processing to obtain the target context information;

[0043] The training module is configured to train the initial language model based on the target context information until a target language model that meets the training stopping condition is obtained.

[0044] According to a fourth aspect of the embodiments of this specification, an information processing apparatus is provided, comprising:

[0045] The information acquisition module is configured to acquire question information submitted by the client and query multiple answer documents in the business knowledge base based on the question information;

[0046] The model prediction module is configured to concatenate the multiple answer documents and input the concatenation result and the question information into a target language model for processing to obtain the answer information. The target language model is trained using the above method.

[0047] The feedback module is configured to send the answer information back to the client.

[0048] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0049] Memory and processor;

[0050] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the above-described language model training method or information processing method.

[0051] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described language model training method or information processing method.

[0052] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described language model training method or information processing method.

[0053] The language model training method provided in this embodiment aims to improve the performance of the language model when handling complex contexts and cross-document reasoning. After obtaining the initial document set for the language model, it selects similar related documents from the document set to form a related document set. Subsequently, to enhance the model's ability to understand cross-document text, a document relationship graph corresponding to the related document set is constructed based on the related documents in the related document set and the similarity between them. This graph reflects the similarity relationships between related documents. Based on this, document concatenation paths can be determined in the document relationship graph, and target related documents are extracted from the related document set according to these paths and concatenated to obtain target context information. Finally, the initial language model can be trained based on this target context information until a target language model that meets the training stopping condition is obtained. This allows the model to be pre-trained on relevant document sequences, enabling the language model to read and reason outside document boundaries, thereby effectively improving the model's performance in complex context tasks. Attached Figure Description

[0054] Figure 1 is a schematic diagram of a language model training method provided in one embodiment of this specification;

[0055] Figure 2 is a flowchart of a language model training method provided in one embodiment of this specification;

[0056] Figure 3 is a flowchart of an information processing method provided in one embodiment of this specification;

[0057] Figure 4 is a flowchart of the processing procedure of an information processing method provided in one embodiment of this specification;

[0058] Figure 5 is a schematic diagram of the structure of a language model training device provided in one embodiment of this specification;

[0059] Figure 6 is a schematic diagram of the structure of an information processing device provided in one embodiment of this specification;

[0060] Figure 7 is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0061] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0062] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0063] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0064] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0065] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0066] Large Language Model (LLM) is an important technique in the field of Natural Language Processing (NLP). It is a deep learning model trained on a large amount of text data that can generate natural language text or understand the meaning of language text.

[0067] A greedy algorithm is an algorithm that makes the best or optimal choice at each step, aiming to achieve a globally optimal result. Each choice made by a greedy algorithm is, in some sense, a locally optimal choice.

[0068] This specification provides a language model training method. One or more embodiments of this specification also relate to an information processing method, a language model training apparatus, an information processing device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0069] In practical applications, to improve the predictive performance of language models, standard pretraining and retrieval-based pretraining (kNNPretraining) are commonly used. Standard pretraining involves randomly shuffling documents and piecing them together into a fixed-length context window for language modeling training. While simple, this method fails to guarantee contextual coherence, leading to poor model performance on complex tasks. Retrieval-based pretraining, on the other hand, trains each document and its related retrieved documents within the same context. This approach can improve the model's semantic understanding to some extent, but it suffers from data duplication, potentially causing overfitting. Therefore, an effective solution is urgently needed to address these issues.

[0070] Referring to the schematic diagram in Figure 1, the language model training method provided in this embodiment, in order to improve the performance of the language model in handling complex contexts and cross-document reasoning, can, after obtaining the document set of the initial language model, select related documents with similarity to form a related document set; subsequently, to enhance the model's ability to understand cross-document text, a document relationship graph corresponding to the related document set can be constructed based on the related documents in the related document set and the similarity between related documents; the document relationship graph reflects the similarity relationships between related documents. On this basis, document splicing paths can be determined in the document relationship graph, and target related documents can be extracted from the related document set according to the document splicing paths for splicing processing to obtain target context information; finally, the initial language model can be trained based on the target context information until a target language model that meets the training stopping condition is obtained. This enables the model to be pre-trained on relevant document sequences, allowing the language model to read and reason outside document boundaries, thereby effectively improving the model's performance in complex context tasks.

[0071] Referring to Figure 2, Figure 2 shows a flowchart of a language model training method according to an embodiment of this specification, which specifically includes the following steps.

[0072] Step S202: Obtain the document set of the initial language model, and filter related documents in the document set to form a related document set.

[0073] The language model training method provided in this embodiment can be applied to the pre-training stage of large language models with arbitrary structures, such as LLaMA models (Large Language Model Meta AI), etc. This embodiment does not impose any limitations on this.

[0074] Specifically, the initial language model refers to the language model that needs to be pre-trained at this stage. It can be a language model deployed in a question-and-answer interaction scenario. After a user submits a question, it can query relevant documents in a knowledge base and generate answer information matching the question based on those documents, thus providing the user with an answer that meets their question-and-answer needs. Correspondingly, the document set refers to the collection of documents used by the initial language model during the pre-training stage. These documents can come from different data sources and correspond to different business domains, such as mathematical knowledge, film knowledge, or transaction knowledge; this embodiment does not impose any limitations. Correspondingly, the associated documents refer to documents with similar relationships selected from the document set to form an associated document set. This can be understood as all associated documents in the associated document set having similarities, used for subsequent pre-training of the language model, enabling it to learn the ability to read and reason outside document boundaries.

[0075] Therefore, to improve the performance of language models in handling complex contexts and cross-document reasoning, after obtaining the initial document set for the language model, similar related documents can be selected from the document set to form a related document set. Subsequently, to enhance the model's ability to understand cross-document text, a document relationship graph corresponding to the related document set can be constructed based on the related documents in the related document set and the similarity between them. This document relationship graph reflects the similarity relationships between related documents. Based on this, document concatenation paths can be determined in the document relationship graph, and target related documents can be extracted from the related document set according to these paths and concatenated to obtain target context information. Finally, the initial language model can be trained based on the target context information until a target language model that meets the training stopping condition is obtained.

[0076] Furthermore, after obtaining the document set for the initial language model, in order to select documents with high similarity to form a set of related documents, thereby improving the model's cross-document understanding ability, related documents can be filtered by calculating cosine similarity. In this embodiment, the specific implementation is as follows:

[0077] The document embedding vectors corresponding to each initial document in the document set are constructed using a text retrieval model; the cosine similarity between the document embedding vectors corresponding to each initial document is calculated; and related documents are selected from the document set based on the cosine similarity calculation results to form a related document set.

[0078] Specifically, the text retrieval model refers to a model capable of constructing embedding vectors for each initial document. This can be a Contriever model or any other model capable of constructing embedding vectors; this embodiment does not impose any limitations. Correspondingly, the document embedding vector refers to the vector representation obtained after embedding the initial document using the text retrieval model. Correspondingly, the cosine similarity refers to the value obtained by calculating the similarity between each initial text based on the document embedding vector.

[0079] Therefore, after determining the document set for pre-training the initial language model, in order to select similar and non-repeating documents for model pre-training, the cosine similarity can be used to construct the associated document set. In this process, a text retrieval model can first be used to construct document embedding vectors corresponding to each initial document in the document set. Based on this, the cosine similarity between the document embedding vectors of each initial document can be calculated. Then, the cosine similarity calculation result can be compared with a preset similarity threshold. Documents with similarity greater than the threshold are selected as associated documents and form an associated document set. This ensures that the associated documents in the associated document set are all documents with higher similarity, thereby improving the model's contextual understanding and reasoning capabilities.

[0080] In practice, when using the Contriever model to perform embedding on documents in a document set, each document di can be mapped to an embedding vector E(di) using the Contriever model. Then, the similarity between documents can be calculated using cosine similarity, which can be achieved by the formula s(di,dj)=cos(E(di),E(dj)), where di and dj represent two documents whose similarity needs to be calculated.

[0081] In summary, by using cosine similarity calculation to select similar documents to form a set of related documents, similar documents can be used to complete the pre-training process during the model pre-training stage. This enables the model to have strong predictive ability while processing text across documents.

[0082] Furthermore, to improve the efficiency of constructing the associated document set, an approximate nearest neighbor search and a reversed file index can be used. In this embodiment, the specific implementation is as follows:

[0083] The document embedding vectors corresponding to each initial document are quantized using a preset quantizer, and a document index is constructed based on the quantized document embedding vectors. The document set is then searched according to a preset search strategy and the document index, and related documents are determined and a related document set is formed based on the search results.

[0084] Specifically, a quantizer is a unit that quantizes the document embedding vector of each initial document. Correspondingly, a document index refers to the index information constructed based on the quantized document embedding vectors for searching for similar documents. Correspondingly, a search strategy refers to the strategy used to set the search criteria for the k closest documents.

[0085] Therefore, in order to achieve efficient retrieval and quickly identify documents with higher similarity as related document sets, we can first use a preset quantizer to quantize the document embedding vectors corresponding to each initial document, so as to build a document index based on the quantized document embedding vectors. On this basis, we can search the document set according to the preset search strategy and document index, and then determine the related documents with higher similarity as related document sets based on the search results, so as to use them for pre-training the initial language model.

[0086] In practical applications, when constructing a set of related documents using Approximate Nearest Neighbor Search and the Inverted Document Index (FAISS), a FAISS index can be built first. This involves quantizing and indexing the document embedding vectors using product quantization. Similarity retrieval is then performed. During this process, an approximate nearest neighbor search can be performed within the FAISS index to quickly find the k most similar documents for each document, which are then used to form a set of related documents.

[0087] In practice, the first step is to perform product quantization on the document embedding vectors. Specifically, configure the PQ quantizer using the `faiss.ProductQuantizer` class from the FAISS library. You need to specify the dimension of the quantization center (dsub) and the number of sub-quantizers (M), which divides the entire vector space into multiple subspaces, each quantized independently. Train the PQ quantizer using a set of training data (which can be a portion of the entire dataset). The training process will find the optimal quantization center for each subspace. Quantize the document embeddings using the trained PQ quantizer to obtain the quantized encoding.

[0088] Next, construct the FAISS index; specifically, choose the index type: product quantization-based indexes in FAISS typically use the IVFPQ (Inverted File with Product Quantization) index type. This index combines the inverted index (IVF) and product quantization (PQ) to achieve efficient approximate nearest neighbor search. Create the index: Use the `faiss.IndexIVFPQ` class of FAISS to create the index. You need to specify the quantizer (i.e., the PQ quantizer trained in step one), the number of cluster centers in the inverted list (nlist), and the number of nearest neighbors to retain under each cluster center. Add data to the index: Add the quantized document embedding vectors to the created FAISS index.

[0089] Finally, an approximate nearest neighbor search is performed. Specifically, search parameters are set: before the search, the `nprobe` parameter is set as needed, which determines the number of nearest neighbors to explore at each cluster center, thus affecting search speed and accuracy. The search is executed: an approximate nearest neighbor search is performed on a given query vector using FAISS's search function (e.g., `index.search`). The query vector can be any new document embedding vector, which also needs to be quantized by a PQ quantizer. Results are obtained: the search function returns the indices and distances of the k most similar documents for each query vector. This constructs a set of related documents for pre-training of the initial language model.

[0090] For example, we determine the document set corresponding to the pre-trained language model—LLaMA—containing m documents. We then use a Contriever model to embed each document in the document set, obtaining an embedding vector E for each document. Next, we calculate the cosine similarity between the embedding vectors E for each document. Based on the cosine similarity calculation results, we select n documents with similarity from the m documents to form an associated document set (n < m). For example, the n documents could be document 1, document 2, ..., document n, which relate to image processing knowledge. This set can then be used to pre-train the LLaMA model, enabling it to have better contextual understanding and reasoning capabilities.

[0091] In summary, searching for similar documents by building an index can improve the efficiency of building related documents, thereby saving model pre-training time.

[0092] Step S204: Construct a document relationship graph corresponding to the associated document set based on the associated documents in the associated document set and the similarity between the associated documents.

[0093] Specifically, after identifying the relevant document set, to further select highly relevant documents for model pre-training, a document relationship graph can be constructed to represent the strength of the relationship between documents. This graph can then be used to filter highly relevant documents for model pre-training. Therefore, a document relationship graph can be constructed based on the relevant documents in the document set and the similarity between them, for subsequent use. Specifically, the document relationship graph refers to an undirected weighted graph constructed using relevant documents as graph nodes and the similarity between relevant documents as graph edges.

[0094] Furthermore, when constructing the document relationship graph, considering that it will serve as the foundation for building the model's input context information, it is necessary to fully utilize the documents and the similarities between them to complete the construction. In this embodiment, the specific implementation is as follows:

[0095] Graph nodes are constructed based on the associated documents in the associated document set, and graph edges are constructed based on the similarity between the associated documents in the associated document set; connectivity relationships are established between the graph nodes based on the graph edges, and a document relationship graph corresponding to the associated document set is generated based on the connectivity relationship establishment results.

[0096] Based on this, when constructing a document relationship graph, we can first draw graph nodes based on the related documents in the related document set, and draw graph edges based on the similarity between related documents in the related document set. When drawing graph edges, the weight of the edge is determined by the similarity between related documents. Then, we can combine the graph edges to establish the connectivity between graph nodes. Based on this, we can generate the document relationship graph corresponding to the related document set according to the results of establishing the connectivity, so as to filter and stitch together highly relevant documents in the future.

[0097] Following the previous example, after obtaining n documents with related image processing knowledge, we can draw n graph nodes based on each document. At the same time, we can construct graph edges based on the similarity between the documents in the n documents. Based on this, we can connect the n graph nodes based on the graph edges to obtain an undirected weighted graph. This undirected weighted graph can then be analyzed to select a set number of documents for splicing, thereby pre-training the LLaMA model.

[0098] In summary, constructing a document relationship graph to represent the relevance between related documents facilitates the subsequent selection of highly relevant documents for concatenation, thereby improving the pre-training effect of the initial language model.

[0099] Step S206: Determine the document splicing path in the document relationship graph, and extract the target related document in the related document set according to the document splicing path for splicing processing to obtain the target context information.

[0100] Specifically, after constructing the document relationship graph, since the document relationship graph can reflect the strength of the correlation between various related documents, the document relationship graph can be traversed to find the document splicing path. The document splicing path can be used to identify highly related related documents. At this time, the target related documents can be extracted from the related document set according to the document splicing path and spliced ​​to obtain the target context information that can be input into the model for processing. Based on this, the language model can be trained using highly related documents, so that the high-level model can learn context understanding and reasoning ability from related document sequences.

[0101] Specifically, the document concatenation path refers to traversing the document relationship graph and finding links composed of nodes corresponding to documents with higher relevance based on the traversal results. This path consists of multiple graph nodes, and the associated documents corresponding to each graph node are highly relevant documents. Correspondingly, the target associated documents are the associated documents involved in the document concatenation path. Correspondingly, the target context information specifically refers to the text obtained after concatenating the target associated documents related to the document concatenation path.

[0102] Furthermore, when determining the document concatenation path in the document relationship graph, considering that this path will determine the association between similar documents in the target context information, and in order to ensure that the target context information is constructed using strongly related documents, thereby improving the model's cross-document understanding and reasoning ability, a greedy algorithm can be used to determine the document concatenation path. In this embodiment, the specific implementation is as follows:

[0103] In the document relationship graph, determine the i-th graph node and construct the initial document path corresponding to the i-th graph node, where i is a positive integer, starting from 1, and the maximum value of i is the number of graph nodes in the document relationship graph; select the j-th graph node from the neighboring graph nodes connected to the i-th graph node, and update the initial document path to an intermediate document path based on the j-th graph node, where j is a positive integer, starting from 1, and the maximum value of j is the number of neighboring graph nodes; determine whether there are any unvisited neighboring graph nodes among the neighboring graph nodes connected to the i-th graph node; if not, increment j by 1, and execute the step of selecting the j-th graph node from the neighboring graph nodes connected to the i-th graph node; if yes, increment i by 1, and execute the step of determining the i-th graph node in the document relationship graph, until i equals the number of graph nodes, to obtain multiple intermediate document paths, and extract the intermediate document path that satisfies the document concatenation condition from the multiple intermediate document paths as the document concatenation path.

[0104] Specifically, the i-th graph node refers to the graph node with the fewest edges in the document relationship graph, and the initial document path refers to the initial path constructed using the i-th graph node as the starting point. Correspondingly, the j-th graph node refers to the unvisited neighbor node among the neighbor nodes of the i-th graph node. Correspondingly, the intermediate document path refers to the path obtained by adding the j-th graph node to the initial document path.

[0105] Based on this, when determining the document splicing path, in order to mine the path composed of nodes corresponding to highly relevant documents from the document relationship graph, we can determine the i-th graph node with the lowest degree from the graph dimension in the document relationship graph and construct the initial document path corresponding to the i-th graph node, where i is a positive integer, i starts from 1, and the maximum value of i is the number of graph nodes in the document relationship graph.

[0106] Then, the unvisited j-th graph node can be selected from the neighboring graph nodes connected to the i-th graph node, and the j-th graph node can be added to the initial document path to obtain the intermediate document path. Here, j is a positive integer, starting from 1, and the maximum value of j is the number of neighboring graph nodes.

[0107] Based on this, it can be determined whether there are any unvisited neighboring graph nodes among the neighboring graph nodes connected to the i-th graph node. If not, it means that the document path corresponding to the i-th graph node can be expanded, so j can be incremented by 1, and the step of selecting the j-th graph node among the neighboring graph nodes connected to the i-th graph node can be executed. If yes, it means that the expansion path of the i-th graph node has reached its upper limit, so other lowest-degree nodes besides the i-th graph node can be selected to build paths, so i can be incremented by 1, and the step of determining the i-th graph node in the document relationship graph can be executed. Until i equals the number of graph nodes, it means that the paths corresponding to all graph nodes in the document relationship graph have been constructed, so multiple intermediate document paths can be obtained. Then, the intermediate document path that meets the document concatenation conditions can be extracted from the multiple intermediate document paths as the document concatenation path. In specific implementation, the longest intermediate document path can be selected as the document concatenation path when selecting the document concatenation path.

[0108] In practical applications, considering that the document relationship graph is an undirected weighted graph, and in order to fully explore this undirected weighted graph and quickly and accurately determine the target related documents through graph analysis, a greedy algorithm can be used to solve the Maximum Traveling Salesman Problem (MTS) to mine the MTS path (document concatenation path). That is, the path can be initialized first: starting from the lowest degree node (the node with the fewest edges connected to other nodes), an initial path is constructed; then the path is expanded: by selecting the highest-weighted unvisited neighbor node of the current node and adding it to the initial path, the initial path is updated; this process continues until all neighbor nodes have been visited, completing the current path construction. At this point, the next lowest-degree node can be selected to continue path expansion until all nodes have completed path construction. The MTS path, i.e., the document concatenation path, can then be determined from the constructed paths. Based on this, documents can be selected and concatenated according to the document concatenation path to obtain the target context information used for model pre-training.

[0109] In summary, by employing a greedy algorithm to solve the maximum traveling salesman path problem, we can maximize the continuous access of related documents on the document relationship graph, thereby improving the coherence of the input context.

[0110] Furthermore, after obtaining the document concatenation path, since the path not only reflects the required document content but also has directionality, the target related documents can be concatenated according to the direction information. In this embodiment, the specific implementation method is as follows:

[0111] The splicing graph nodes contained in the document splicing path are determined, and the target associated documents are extracted from the associated document set based on the splicing graph nodes; the target associated documents are spliced ​​according to the path direction information of the document splicing path to obtain the target context information that matches the input conditions of the initial language model.

[0112] Specifically, the splicing graph nodes refer to the graph nodes contained in the document splicing path. Correspondingly, path direction information refers to information representing the relationship between the documents before and after splicing. Based on this, after obtaining the document splicing path, the splicing graph nodes contained in the path can be determined. On this basis, target related documents can be extracted from the set of related documents based on the splicing graph nodes. Then, to ensure the coherence of the splicing result, the target related documents can be spliced ​​according to the path direction information of the document splicing path. This allows us to obtain target context information matching the input conditions of the initial language model, which can then be used for subsequent pre-training of the initial language model.

[0113] Following the previous example, after obtaining an undirected weighted graph constructed from n documents, a greedy algorithm can be used to process the graph and discover the path with the maximum weight. Specifically, the lowest-degree node in the graph can be selected first and used as the starting node to construct an initial path. Then, the highest-weight, unvisited neighbor node of that node can be selected and added to the initial path to construct an intermediate path. This process continues until all neighbor nodes have been visited. Then, the lowest-degree node (excluding the original node) is selected for further path mining. Finally, by traversing each node in the graph, the path with the maximum weight can be generated, containing document 1, document 2, document 3, and document 5. By concatenating these documents along the path with the maximum weight, a fixed-size input context can be obtained for subsequent model pre-training.

[0114] In summary, since the document concatenation path is obtained based on document relationship graph mining, the generation of context information by concatenating documents based on this path can ensure the continuity of the context and the uniqueness of each target-related document, thereby solving the data duplication problem and improving the model training effect.

[0115] Step S208: Train the initial language model based on the target context information until a target language model that meets the training stopping condition is obtained.

[0116] Specifically, after obtaining the target context information by splicing the target related documents, since the target context information is based on splicing highly related documents, training the initial language model based on the target context information until a target language model that meets the training stopping condition can effectively improve the context understanding and reasoning ability of the target language model, thereby significantly improving the performance of the target language model in complex tasks.

[0117] Furthermore, in order to obtain a target language model that meets deployment requirements during model training, model detection can be performed during the training phase. In this embodiment, the specific implementation is as follows:

[0118] The target context information is input into the initial language model for processing to obtain predicted text information; the initial language model is optimized based on the baseline text information corresponding to the target context information and the predicted text information; it is then checked whether the optimized initial language model meets the training stopping condition; if not, candidate related documents are selected from the document set to form a candidate related document set, and the candidate related document set is used as the related document set, and the step of constructing a document relationship graph corresponding to the related document set is performed based on the related documents in the related document set and the similarity between related documents; if yes, the optimized initial language model is used as the target language model.

[0119] Specifically, predicted text information refers to the predicted information obtained after the initial language model processes the target context information. Correspondingly, baseline text information refers to the real information corresponding to the target context information. Optimizing the model by combining predicted and baseline text information allows the model to learn the knowledge required for the training objective, thereby meeting business deployment needs.

[0120] Based on this, after obtaining the target context information, it can be input into the initial language model for processing. The text information can be predicted based on the processing results. Then, the baseline text information corresponding to the target context information can be obtained first, and the initial language model can be further optimized by combining the baseline text information and the predicted text information. After optimization, it can be checked whether the optimized initial language model meets the training stopping condition. If not, it means that the current optimized initial language model cannot meet the deployment conditions. Therefore, candidate related documents can be selected from the document set to form a candidate related document set, and the candidate related document set can be used as the related document set. Then, the step of constructing the document relationship graph corresponding to the related document set based on the related documents in the related document set and the similarity between the related documents is executed until a model that meets the conditions is obtained. Then, the optimized initial language model can be used as the target language model for deployment in business scenarios.

[0121] In practical applications, when pre-training the initial language model using the generated target context information, the initial language model being trained can adopt the same model architecture as the existing pre-training pipeline, and model optimization can be completed using the AdamW optimizer and cosine learning rate scheduling; the contextual understanding and reasoning ability of the model can also be improved by increasing the length of the context information in the training batch.

[0122] Continuing with the previous example, after obtaining the contextual information from concatenated documents 1, 2, 3, and 5, the LLaMA model can be pre-trained based on this contextual information. A well-trained LLaMA model can then possess better cross-document understanding and reasoning capabilities. For instance, if a user submits the question "{What steps are involved in image sharpening?}", by inputting this question into the trained LLaMA model, the model can first identify multiple documents related to the question. By concatenating these documents, it can then combine them with the user's question to generate an answer: {1. Select an image; 2. Select a sharpening method; 3. Apply a sharpening filter…}, and finally, the answer can be returned to the user.

[0123] The language model training method provided in this embodiment aims to improve the performance of the language model when handling complex contexts and cross-document reasoning. After obtaining the initial document set for the language model, it selects similar related documents from the document set to form a related document set. Subsequently, to enhance the model's ability to understand cross-document text, a document relationship graph corresponding to the related document set is constructed based on the related documents in the related document set and the similarity between them. This graph reflects the similarity relationships between related documents. Based on this, document concatenation paths can be determined in the document relationship graph, and target related documents are extracted from the related document set according to these paths and concatenated to obtain target context information. Finally, the initial language model can be trained based on this target context information until a target language model that meets the training stopping condition is obtained. This allows the model to be pre-trained on relevant document sequences, enabling the language model to read and reason outside document boundaries, thereby effectively improving the model's performance in complex context tasks.

[0124] Referring to Figure 3, Figure 3 shows a flowchart of an information processing method provided according to an embodiment of this specification, which specifically includes the following steps.

[0125] Step S302: Obtain the question information submitted by the client, and query multiple answer documents in the business knowledge base based on the question information;

[0126] Step S304: The multiple answer documents are concatenated, and the concatenation result and the question information are input into the target language model for processing to obtain the answer information. The target language model is trained using the above method.

[0127] Step S306: The answer information is fed back to the client.

[0128] The description of the target language model in the information processing method provided in this embodiment can be found in the descriptions in the above embodiments, and will not be repeated here.

[0129] The following description, in conjunction with Figure 4, uses the application of the information processing method and language model training method provided in this specification in a question-and-answer interaction scenario as an example to further illustrate the information processing method and language model training method. Figure 4 shows a flowchart of the processing procedure of an information processing method according to an embodiment of this specification, specifically including the following steps.

[0130] Step S402: Obtain the document set of the initial language model, and use the text retrieval model to construct the document embedding vector corresponding to each initial document in the document set.

[0131] Step S404: Calculate the cosine similarity between the document embedding vectors corresponding to each initial document.

[0132] Step S406: Based on the cosine similarity calculation results, filter related documents in the document set to form a related document set.

[0133] Step S408: Construct graph nodes based on the associated documents in the associated document set, and construct graph edges based on the similarity between the associated documents in the associated document set.

[0134] Step S410: Establish the connectivity between the graph nodes based on the graph edges, and generate a document relationship graph corresponding to the associated document set based on the connectivity establishment result.

[0135] Step S412: Use a greedy algorithm to determine the document splicing path in the document relationship graph.

[0136] Step S414: Determine the splicing graph nodes contained in the document splicing path, and extract the target related document from the related document set based on the splicing graph nodes.

[0137] Step S416: Concatenate the target associated documents according to the path direction information of the document concatenation path to obtain the target context information that matches the input conditions of the initial language model.

[0138] Step S418: Train the initial language model based on the target context information until a target language model that meets the training stopping condition is obtained, and then deploy the target language model.

[0139] Step S420: Upon receiving the question information submitted by the client, query multiple answer documents in the business knowledge base based on the question information;

[0140] Step S422: Concatenate multiple answer documents, and input the concatenation result and question information into the target language model for processing to obtain answer information.

[0141] Step S424: Send the answer information back to the client.

[0142] In summary, to improve the performance of language models in handling complex contexts and cross-document reasoning, after obtaining the initial document set for the language model, similar related documents can be selected from the document set to form a related document set. Subsequently, to enhance the model's ability to understand cross-document text, a document relationship graph corresponding to the related document set can be constructed based on the related documents in the related document set and the similarity between them. This document relationship graph reflects the similarity relationships between related documents. Based on this, document concatenation paths can be determined in the document relationship graph, and target related documents can be extracted from the related document set according to these paths for concatenation to obtain target context information. Finally, the initial language model can be trained based on this target context information until a target language model that meets the training stopping condition is obtained. This allows the model to be pre-trained on relevant document sequences, enabling the language model to read and reason outside document boundaries, thereby effectively improving the model's performance in complex context tasks.

[0143] Corresponding to the above method embodiments, this specification also provides embodiments of a language model training device. Figure 5 shows a schematic diagram of the structure of a language model training device provided in one embodiment of this specification. As shown in Figure 5, the device includes:

[0144] The acquisition module 502 is configured to acquire a document set of the initial language model and filter related documents in the document set to form a related document set;

[0145] The construction module 504 is configured to construct a document relationship graph corresponding to the associated document set based on the associated documents in the associated document set and the similarity between the associated documents;

[0146] The splicing module 506 is configured to determine the document splicing path in the document relationship graph, and extract the target related document in the related document set according to the document splicing path for splicing processing to obtain the target context information;

[0147] Training module 508 is configured to train the initial language model based on the target context information until a target language model that meets the training stopping condition is obtained.

[0148] In an optional embodiment, the acquisition module 502 is further configured to:

[0149] The document embedding vectors corresponding to each initial document in the document set are constructed using a text retrieval model; the cosine similarity between the document embedding vectors corresponding to each initial document is calculated; and related documents are selected from the document set based on the cosine similarity calculation results to form a related document set.

[0150] In an optional embodiment, the acquisition module 502 is further configured to:

[0151] The document embedding vectors corresponding to each initial document are quantized using a preset quantizer, and a document index is constructed based on the quantized document embedding vectors. The document set is then searched according to a preset search strategy and the document index, and related documents are determined and a related document set is formed based on the search results.

[0152] In an optional embodiment, the building module 504 is further configured to:

[0153] Graph nodes are constructed based on the associated documents in the associated document set, and graph edges are constructed based on the similarity between the associated documents in the associated document set; connectivity relationships are established between the graph nodes based on the graph edges, and a document relationship graph corresponding to the associated document set is generated based on the connectivity relationship establishment results.

[0154] In an optional embodiment, the splicing module 506 is further configured to:

[0155] In the document relationship graph, determine the i-th graph node and construct the initial document path corresponding to the i-th graph node, where i is a positive integer, starting from 1, and the maximum value of i is the number of graph nodes in the document relationship graph; select the j-th graph node from the neighboring graph nodes connected to the i-th graph node, and update the initial document path to an intermediate document path based on the j-th graph node, where j is a positive integer, starting from 1, and the maximum value of j is the number of neighboring graph nodes; determine whether there are any unvisited neighboring graph nodes among the neighboring graph nodes connected to the i-th graph node; if not, increment j by 1, and execute the step of selecting the j-th graph node from the neighboring graph nodes connected to the i-th graph node; if yes, increment i by 1, and execute the step of determining the i-th graph node in the document relationship graph, until i equals the number of graph nodes, to obtain multiple intermediate document paths, and extract the intermediate document path that satisfies the document concatenation condition from the multiple intermediate document paths as the document concatenation path.

[0156] In an optional embodiment, the splicing module 506 is further configured to:

[0157] The splicing graph nodes contained in the document splicing path are determined, and the target associated documents are extracted from the associated document set based on the splicing graph nodes; the target associated documents are spliced ​​according to the path direction information of the document splicing path to obtain the target context information that matches the input conditions of the initial language model.

[0158] In an optional embodiment, the training module 508 is further configured to:

[0159] The target context information is input into the initial language model for processing to obtain predicted text information; the initial language model is optimized based on the baseline text information corresponding to the target context information and the predicted text information; it is then checked whether the optimized initial language model meets the training stopping condition; if not, candidate related documents are selected from the document set to form a candidate related document set, and the candidate related document set is used as the related document set, and the step of constructing a document relationship graph corresponding to the related document set is performed based on the related documents in the related document set and the similarity between related documents; if yes, the optimized initial language model is used as the target language model.

[0160] The above is an illustrative scheme of a language model training device according to this embodiment. It should be noted that the technical solution of this language model training device and the technical solution of the language model training method described above belong to the same concept. For details not described in detail in the technical solution of the language model training device, please refer to the description of the technical solution of the language model training method described above.

[0161] Corresponding to the above method embodiments, this specification also provides an information processing apparatus embodiment. Figure 6 shows a schematic diagram of the structure of an information processing apparatus provided in one embodiment of this specification. As shown in Figure 6, the apparatus includes:

[0162] The information acquisition module 602 is configured to acquire question information submitted by the client and query multiple answer documents in the business knowledge base based on the question information;

[0163] The model prediction module 604 is configured to concatenate the multiple answer documents and input the concatenation result and the question information into a target language model for processing to obtain the answer information. The target language model is trained using the above method.

[0164] The feedback information module 606 is configured to send the answer information back to the client.

[0165] The above is an illustrative scheme of an information processing device according to this embodiment. It should be noted that the technical solution of this information processing device and the technical solution of the information processing method described above belong to the same concept. For details not described in detail in the technical solution of the information processing device, please refer to the description of the technical solution of the information processing method described above.

[0166] Figure 7 shows a structural block diagram of a computing device 700 according to one embodiment of this specification. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0167] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0168] In one embodiment of this specification, the aforementioned components of the computing device 700, as well as other components not shown in FIG. 7, may be interconnected, for example, via a bus. It should be understood that the block diagram of the computing device shown in FIG. 7 is merely illustrative and not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0169] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.

[0170] The processor 720 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned language model training method or information processing method.

[0171] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solution of the language model training method or information processing method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the language model training method or information processing method described above.

[0172] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described language model training method or information processing method.

[0173] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the language model training method or information processing method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the language model training method or information processing method described above.

[0174] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described language model training method or information processing method.

[0175] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solution of the language model training method or information processing method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the language model training method or information processing method described above.

[0176] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0177] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0178] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0179] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0180] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described in this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification.

Claims

1. A language model training method, characterized in that, include: Obtain the document set of the initial language model, and filter related documents from the document set to form a related document set; Based on the associated documents in the associated document set and the similarity between the associated documents, a document relationship graph corresponding to the associated document set is constructed; The document splicing path is determined in the document relationship graph, and the target related document is extracted from the related document set according to the document splicing path and spliced ​​to obtain the target context information; The initial language model is trained based on the target context information until a target language model that meets the training stopping condition is obtained.

2. The language model training method according to claim 1, characterized in that, The step of filtering related documents from the document set to form a related document set includes: The document embedding vectors corresponding to each initial document in the document set are constructed using a text retrieval model; Calculate the cosine similarity between the document embedding vectors corresponding to each initial document; Based on the cosine similarity calculation results, related documents are selected from the document set to form a related document set.

3. The language model training method according to claim 2, characterized in that, After the step of constructing the document embedding vectors corresponding to each initial document in the document set using the text retrieval model is executed, the method further includes: The document embedding vectors corresponding to each initial document are quantized using a preset quantizer, and a document index is constructed based on the quantized document embedding vectors. The document set is searched according to the preset search strategy and the document index, and related documents are determined and formed into a related document set based on the search results.

4. The language model training method according to claim 1, characterized in that, The step of constructing a document relationship graph corresponding to the associated document set based on the associated documents in the associated document set and the similarity between the associated documents includes: Graph nodes are constructed based on the associated documents in the associated document set, and graph edges are constructed based on the similarity between the associated documents in the associated document set; Based on the graph edges, establish the connectivity relationships between the graph nodes, and generate a document relationship graph corresponding to the associated document set based on the results of establishing the connectivity relationships.

5. The language model training method according to claim 1, characterized in that, Determining the document concatenation path in the document relationship graph includes: In the document relationship graph, determine the i-th graph node and construct the initial document path corresponding to the i-th graph node, where i is a positive integer, starting from 1, and the maximum value of i is the number of graph nodes in the document relationship graph; Select the j-th graph node from the neighboring graph nodes connected to the i-th graph node, and update the initial document path to the intermediate document path based on the j-th graph node, where j is a positive integer, starting from 1, and the maximum value of j is the number of neighboring graph nodes; Determine whether there are any unvisited neighboring graph nodes among the neighboring graph nodes connected to the i-th graph node; If not, j is incremented by 1, and the step of selecting the j-th graph node from the neighboring graph nodes connected to the i-th graph node is performed; If so, i is incremented by 1, and the step of determining the i-th graph node in the document relationship graph is executed until i is equal to the number of graph nodes. Multiple intermediate document paths are obtained, and the intermediate document path that satisfies the document splicing condition is extracted from the multiple intermediate document paths as the document splicing path.

6. The language model training method according to claim 1, characterized in that, The step of extracting the target related document from the related document set according to the document concatenation path and performing concatenation processing to obtain target context information includes: Determine the splicing graph nodes contained in the document splicing path, and extract the target associated document from the associated document set based on the splicing graph nodes; The target associated documents are concatenated according to the path direction information of the document concatenation path to obtain target context information that matches the input conditions of the initial language model.

7. The language model training method according to claim 1, characterized in that, The step of training the initial language model based on the target context information until a target language model that meets the training stopping condition is obtained includes: The target context information is input into the initial language model for processing to obtain predicted text information; The initial language model is optimized based on the baseline text information corresponding to the target context information and the predicted text information. Check whether the optimized initial language model meets the training stopping condition; If not, select candidate related documents from the document set to form a candidate related document set, and use the candidate related document set as the related document set, and perform the step of constructing a document relationship graph corresponding to the related document set based on the related documents in the related document set and the similarity between the related documents; If so, the optimized initial language model will be used as the target language model.

8. An information processing method, characterized in that, include: Obtain the question information submitted by the client, and query multiple answer documents in the business knowledge base based on the question information; The multiple answer documents are concatenated, and the concatenation result and the question information are input into a target language model for processing to obtain the answer information. The target language model is trained by the method described in any one of claims 1 to 7. The answer information is then sent back to the client.

9. A language model training device, characterized in that, include: The acquisition module is configured to acquire a document set of the initial language model and filter related documents in the document set to form a related document set; The construction module is configured to construct a document relationship graph corresponding to the associated document set based on the associated documents in the associated document set and the similarity between the associated documents; The splicing module is configured to determine the document splicing path in the document relationship graph, and extract the target related document in the related document set according to the document splicing path for splicing processing to obtain the target context information; The training module is configured to train the initial language model based on the target context information until a target language model that meets the training stopping condition is obtained.

10. An information processing device, characterized in that, include: The information acquisition module is configured to acquire question information submitted by the client and query multiple answer documents in the business knowledge base based on the question information; The model prediction module is configured to concatenate the multiple answer documents and input the concatenation result and the question information into a target language model for processing to obtain answer information. The target language model is trained by the method described in any one of claims 1 to 7. The feedback module is configured to send the answer information back to the client.

11. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

13. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Images