Large language model training method, question answering method, device, medium, and product

Abstract

The present disclosure relates to the technical field of artificial intelligence and provides a large language model training method, a question answering method, a device, a medium, and a product. The training method comprises: acquiring long text training data, the sequence length of the long text training data being greater than the maximum length of an input text sequence of a pre-trained large language model; increasing a rotation angle base for rotary position embedding of the pre-trained large language model, to obtain a modified pre-trained large language model; and using the long text training data to train the modified pre-trained large language model, to obtain a trained large language model. In the present embodiment, the pre-trained large language model is trained by means of acquiring the long text training data and increasing the rotation angle base for the rotary position embedding, to implement length extension of the input text sequence, so that the trained large language model can process a long text sequence, thereby improving the completeness and accuracy of answers of the large language model to questions depending on long text and depending on comparison of multiple documents.

Description

Training method of large language model, question and answer method, device, medium and product

[0001] The present disclosure claims priority to Chinese Patent Application No. 202411282695.7, filed on September 13, 2024, with the Chinese Patent Office, entitled "Training method of large language model, question and answer method, device, medium and product", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to the field of artificial intelligence, and in particular, to a training method of a large language model, a question and answer method, a device, a medium and a product. BACKGROUND

[0003] Retrieval-Augmented Generation (RAG) technology is a very extensive application scheme of a large language model. Relevant materials are retrieved, then assembled according to a preset instruction, input to a large language model for understanding, and finally answer the user's question. It is commonly used in various knowledge question and answer assistants. Compared with the direct answer of a large language model, the answer of the RAG method utilizes the knowledge of search, which can ensure the timeliness of knowledge and reduce the problem of inaccurate results caused by self-invention of a large language model.

[0004] The RAG scheme in the related art is to split the collected documents into multiple text blocks (chunks) according to a specific blocking manner, and store them in a vector library, a search engine library (for example, an Elasticsearch library) and other databases. When a question raised by a user is obtained, a plurality of libraries are queried in parallel to retrieve a plurality of text blocks most relevant to the question, a ranking model is called to rank the text blocks retrieved by multiple channels, and the text blocks ranked in the front are input into a large language model to obtain an answer to the question based on the inference of the large language model. However, since the length of the text sequence input into the large language model is limited, the reasoning effect of the question is not good for the summary of a long document or the comparison and summary of multiple documents. SUMMARY

[0005] The present disclosure provides a training method of a large language model, a question and answer method, a device, a medium and a product to expand the maximum text sequence length input into a large language model, and improve the completeness and accuracy of the answer of a large language model to questions that depend on long text and comparison of multiple documents.

[0006] In a first aspect, the embodiments of the present disclosure provide a training method of a large language model, including: obtaining long text training data, a sequence length of the long text training data being greater than a maximum length of an input text sequence of a pre-trained large language model; increasing a rotation angle base of rotation position encoding of the pre-trained large language model to obtain a modified pre-trained large language model; training the modified pre-trained large language model by using the long text training data to obtain a trained large language model.

[0007] In a second aspect, the embodiments of the present disclosure provide a question and answer method based on a large language model, including: obtaining question information and task instructions, querying a plurality of text blocks related to the question information; concatenating the question information, the task instructions and the plurality of text blocks to obtain a concatenated text, a sequence length of the concatenated text being greater than a maximum length of an input text sequence of a pre-trained large language model; inputting the concatenated text into the trained large language model to obtain reply information output by the trained large language model; wherein the trained large language model is trained by using the training method in the embodiments of the present disclosure.

[0008] In a third aspect, the embodiments of the present disclosure provide an electronic device, including a memory, a processor and a computer program stored in the memory, the processor implements any of the above methods when executing the computer program.

[0009] In a fourth aspect, the embodiments of the present disclosure provide a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to implement any of the above methods.

[0010] In a fifth aspect, the embodiments of the present disclosure provide a computer program product, the computer program product includes a computer program, the computer program is executed by a processor to implement any of the above methods.

[0011] Compared with the prior art, the present disclosure has the following advantages:

[0012] The present disclosure provides a large language model training method, a question and answer method, a device, a medium and a product. Long text training data is obtained, and a sequence length of the long text training data is greater than a maximum length of an input text sequence of a pre-trained large language model. A rotation angle base of rotation position encoding of the pre-trained large language model is increased to obtain a modified pre-trained large language model. The modified pre-trained large language model is trained using the long text training data to obtain a trained large language model. In this embodiment, the pre-trained large language model is trained by obtaining the long text training data and increasing the rotation angle base of the rotation position encoding. Since the rotation position encoding generates a position encoding vector through mathematical transformation, increasing the rotation angle base of the rotation position encoding can change the way of mathematical transformation, and then affect the generation of the position encoding vector, so that the trained large language model can effectively capture the position information when processing longer sequences, thereby allowing the large language model to process longer input text sequences, realizing the length expansion of the input text sequence, and improving the completeness and accuracy of the large language model in answering the problems of long text dependence and multi-document comparison dependence.

[0013] The above description is only a summary of the technical solutions of the present disclosure. In order to enable a clearer understanding of the technical means of the present disclosure, the contents of the specification can be implemented, and in order to enable the above and other purposes, features and advantages of the present disclosure to be more apparent and easy to understand, the specific embodiments of the present disclosure are described below. BRIEF DESCRIPTION OF DRAWINGS

[0014] In the drawings, like reference numerals designate like elements or components throughout the several views, unless otherwise specified. The drawings are not necessarily to scale, the drawings are only intended to depict some embodiments in accordance with the present disclosure and should not be interpreted as limiting the scope of the present disclosure.

[0015] FIG. 1 is a schematic diagram of splicing of input text blocks of a large language model in the related art.

[0016] FIG. 2 is a schematic diagram of splicing of input text blocks of a large language model according to an embodiment of the present disclosure.

[0017] FIG. 3 is a flowchart of a large language model training method according to an embodiment of the present disclosure.

[0018] FIG. 4 is a flowchart of a large language model training method according to an embodiment of the present disclosure.

[0019] FIG. 5 is a flowchart of a large language model-based question and answer method according to an embodiment of the present disclosure.

[0020] FIG. 6 is a flowchart of a large language model-based question and answer method according to an embodiment of the present disclosure.

[0021] FIG. 7 is a structural block diagram of a training device of a large language model according to an embodiment of the present disclosure.

[0022] FIG. 8 is a structural block diagram of a question and answer device based on a large language model according to an embodiment of the present disclosure.

[0023] FIG. 9 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In the following, only certain exemplary embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the concept or scope of the present disclosure. Therefore, the drawings and the description are considered to be exemplary in nature, rather than limiting.

[0025] To facilitate understanding of the technical solutions of the embodiments of the present disclosure, the related technologies of the embodiments of the present disclosure are described below. The following related technologies can be combined with the technical solutions of the embodiments of the present disclosure in any manner as optional solutions, which all belong to the protection scope of the embodiments of the present disclosure.

[0026] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0027] In the related technologies, the RAG technology retrieves a plurality of text blocks related to the user input question through the retrieval component. Since the input text (also referred to as input context) sequence length of the open source large language model is limited, generally 8k-32k, wherein k=1024, the length of the input text sequence is at most 32x1024 tokens, when the sequence length of the plurality of text blocks exceeds 32k, the large language model cannot be input, therefore, the sorting model is used to sort the plurality of text blocks, and the several text blocks with the highest sorting are selected to be concatenated with the task instruction and the question information and then input to the large language model for processing.

[0028] FIG. 1 is a concatenation diagram of input text blocks of a large language model in the related art. Among them, E-text block 1, E-text block 2…E-text block N represent N text blocks related to the problem information of the user input obtained by querying in a text keyword recall manner. V-text block 1, V-text block 2…V-text block M represent M text blocks related to the problem information of the user input obtained by querying in a vector recall manner, the text blocks are sorted by using a ranking model, the top 3 text blocks are selected, R-text block 1, R-text block 2 and R-text block 3, the selected text blocks are concatenated with task instructions and problem information of the user input, and input into the large language model for processing to obtain reply information. Among them, the task instruction (Instruction) is a guide or command required by the large language model to execute a specific task, indicating the specific task the user wants to complete or the output type required. For example, you are an xxx expert, please answer the user's question xxx according to the query xxx according to the requirement xxx.

[0029] Since the length of the input text sequence of the large language model is limited, only a few text blocks ranked at the top can be selected in the related art. For problems that require summary of long documents or comparison and summary of multiple documents, the reasoning result of the large language model is not complete and accurate enough.

[0030] In the embodiments of the present disclosure, first, long text training data is obtained, and the sequence length of the long text training data is greater than the maximum length of the input text sequence of the pre-trained large language model. Specifically, long context training corpus is collected, for example, data of a network data set or book document data, and then cleaned to a preset length, such as 128k (k=1024). Then, the rotation angle base of the rotation position encoding of the pre-trained large language model is increased to obtain a modified pre-trained large language model. Specifically, the rotation angle base of the rotation position encoding (i.e. the base value of the rotation position encoding) is modified. The rotation angle base is generally set to 10000 (ten thousand) by default, and can be modified to 1000000 (one million). Since the rotation position encoding generates a position encoding vector through mathematical transformation, increasing the rotation angle base of the rotation position encoding can change the way of mathematical transformation, and then affect the generation of the position encoding vector, so that the large language model trained can effectively capture the position information when processing longer sequences, thereby allowing the large language model to process longer input text sequences. The modified pre-trained large language model is trained using long text training data to obtain a trained large language model, i.e. a general large language model.

[0031] Then, the general large language model is fine-tuned, preset field long text training data is obtained, the trained large language model is trained by using the preset field long text training data, and the large language model corresponding to the preset field is obtained.

[0032] Finally, the position is expanded. Specifically, in the configuration file of the large language model corresponding to the preset field (i.e. the first large language model), the maximum position number (the maximum position number represents the maximum text sequence length that the large language model can process) of the large language model is increased to obtain the large language model after position expansion (i.e. the second large language model). For example, using the rotation position encoding expansion method YaRN (Yet another RoPE extensioN) method, the code of the model inference framework vllm, in the configuration file, if the maximum length of the context of the large language model (original_max_position_embeddings) is 32k, the position is expanded by length extrapolation, the extrapolation factor (factor) is set to 8 times, and the maximum length of the input text sequence of the modified large language model is 32k*8=256k.

[0033] Since the large language model in the embodiment of the present disclosure is trained by using long text data, the rotation angle base of the rotation position coding is increased, and the position of the input text sequence length of the large language model is expanded, so that the text blocks can be sorted according to the relevance scores without the need for a sorting model. Since the large language model is more sensitive to the text closer to the task instruction and closer to the problem information part, and since the text blocks obtained by the vector recall mode are more accurate than the text blocks obtained by the text keyword recall mode, the text blocks E-text block 1, E-text block 2…E-text block N obtained by the text keyword recall mode are sorted in descending order of the relevance scores with the problem information, and the text blocks V-text block 1, V-text block 2…V-text block M obtained by the vector recall mode are sorted in ascending order of the relevance scores with the problem information. The text blocks E-text block 1, E-text block 2…E-text block N are spliced after the task instruction, and the text blocks V-text block 1, V-text block 2…V-text block M are spliced before the problem information. FIG. 2 is a splicing diagram of the input text blocks of the large language model according to an embodiment of the present disclosure. In addition, in order to make the large language model more easily distinguish the content at different positions (task instruction, text block, problem information), prefix tokens Pre-Tokens and suffix tokens Post-Tokens are inserted as the first placeholder and the second placeholder, respectively. The number of placeholders of different large language models is different, for example, Pre-Tokens can be 400, and Post-Tokens can be 100. Finally, the spliced text is input into the large language model with long context (for example, supporting a maximum of 256k), and the reply information is obtained. In this example, 30 text blocks can be obtained by each of the two recall modes, and a total of 60 text blocks are obtained, which greatly increases the number of input text blocks.

[0034] In the embodiment, the large language model is trained by using long text data, the rotation angle base of the rotation position coding is increased, and the position of the input text sequence length of the large language model is expanded, thereby increasing the input text sequence length of the large language model and improving the completeness and accuracy of the large language model in answering problems of long text dependence and multi-document comparison dependence. Further, the multi-path recall reverse sorting mode is used for text splicing, which removes the sorting model, making the RAG system more simple and efficient and more accurate. In addition, by inserting placeholders, the large language model can more easily distinguish the content at different positions, and the accuracy of the large language model in answering is improved.

[0035] The embodiment of the present disclosure provides a training method of a large language model. The method in the embodiment can be applied to servers, terminal devices, platforms, devices, etc. with computing and processing capabilities. The server can be a server cluster or a single server, and can be a server deployed in the cloud or a local server.

[0036] As shown in FIG. 3 is a flowchart of a training method of a large language model according to an embodiment of the present disclosure, comprising:

[0037] Step S301, obtaining long text training data, the sequence length of the long text training data is greater than the maximum length of the input text sequence of the pre-trained large language model.

[0038] Wherein, the large language model (Large Language Model, LLM) refers to a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of language text. The specific structure of the large language model can be based on the structure of the Transformer model.

[0039] At present, the sequence length (number of input token) of the input text of the open source pre-trained large language model is generally 8k-32k. Collect long text training corpus, for example, network dataset data or book document data, which includes data of multiple fields, and then clean to a preset length, such as 128k (k = 1024).

[0040] Step S302, increase the rotation angle base of the rotation position encoding of the pre-trained large language model to obtain a modified pre-trained large language model.

[0041] The rotation angle base of the rotation position encoding of the pre-trained large language model is modified, that is, the base value of the rotation position encoding, which is a hyperparameter of the large language model. The rotation position encoding generates a position encoding vector through mathematical transformation. Increasing the rotation angle base of the rotation position encoding can change the way of mathematical transformation, and then affect the generation of the position encoding vector, so that the trained large language model can still effectively capture the position information when processing longer sequences, thereby allowing the large language model to process longer input text sequences. For example, the base value of the rotation position encoding is modified from 10000 to 1000000.

[0042] Step S303, training the modified pre-trained large language model using the long text training data to obtain a trained large language model.

[0043] Training the modified pre-trained large language model using the long text training data can be trained in an unsupervised manner, until the training end condition is met, to obtain a trained large language model, and the maximum length of the input text sequence of the obtained large language model is greater than the maximum length of the input text sequence of the open source pre-trained large language model.

[0044] The method for training a large language model provided in the embodiments of the present disclosure includes: obtaining long text training data, the sequence length of the long text training data being greater than the maximum length of an input text sequence of a pre-trained large language model; increasing a rotation angle base of rotation position encoding of the pre-trained large language model to obtain a modified pre-trained large language model; training the modified pre-trained large language model using the long text training data to obtain a trained large language model. In the embodiments, the pre-trained large language model is trained by obtaining the long text training data and increasing the rotation angle base of the rotation position encoding. Since the rotation position encoding generates a position encoding vector through mathematical transformation, increasing the rotation angle base of the rotation position encoding can change the way of mathematical transformation and further affect the generation of the position encoding vector, so that the trained large language model can effectively capture position information when processing longer sequences, thereby allowing the large language model to process longer input text sequences, realizing length expansion of the input text sequence, and improving the completeness and accuracy of the large language model in answering problems of long text dependency and multi-document comparison dependency.

[0045] In an implementation manner, the method for training a large language model further includes: obtaining long text training data of a preset field, and training the trained large language model using the long text training data of the preset field to obtain a first large language model corresponding to the preset field.

[0046] In actual application, the large language model trained using data of multiple fields is a general large language model, and the general large language model learns general rules of language and extensive knowledge. In order to improve the accuracy of information processing of the large language model for a preset field, the large language model is fine-tuned using long text data of the preset field.

[0047] The preset field can be determined according to a specific field to which the large language model is applied.

[0048] In the embodiments, the large language model is fine-tuned using long text training data of a preset field, which can improve the completeness and accuracy of the large language model in processing long text data of the preset field.

[0049] In an implementation manner, the method for training a large language model further includes: increasing the maximum position number of the first large language model in a configuration file of the first large language model corresponding to the preset field to obtain a second large language model, the maximum length of an input text sequence of the second large language model being greater than the maximum length of an input text sequence of the first large language model.

[0050] In actual applications, the sequence length of the input data of the large language model can be further expanded before inference using the language model. Specifically, in the configuration file of the large language model corresponding to the preset field, the maximum position number (the maximum position number represents the maximum text sequence length that the large language model can process) of the large language model is increased, that is, the value of the parameter original_max_position_embeddings is increased to a preset multiple, for example, from 32k to 32k*8=256k.

[0051] In this embodiment, the input text sequence length of the large language model is expanded in the configuration file, further increasing the maximum length of the input text sequence of the large language model, and improving the completeness and accuracy of the answers of the large language model on long text dependency and multi-document comparison dependency.

[0052] The embodiments of the present disclosure provide a training method of a large language model. The method in the embodiments can be applied to servers, terminal devices, platforms, apparatuses, etc. with computing and processing capabilities. The server can be a server cluster or a single server, and can be a server deployed in the cloud or a local server.

[0053] As shown in FIG. 4, the flowchart of the training method of the large language model according to an embodiment of the present disclosure includes the following steps:

[0054] In step S401, long text training data is obtained, and the sequence length of the long text training data is greater than the maximum length of the input text sequence of the pre-trained large language model.

[0055] At present, the sequence length (the number of input tokens) of the input text of the open-source pre-trained large language model is generally 8k-32k. The long text training corpus is collected, for example, network dataset data or book document data, which includes data of multiple fields, and then cleaned to a preset length, such as 128k (k=1024).

[0056] In step S402, the base value of the rotation angle of the rotation position encoding of the pre-trained large language model is increased to obtain a modified pre-trained large language model.

[0057] The base value of the rotation angle of the rotation position encoding of the pre-trained large language model is modified, that is, the base value of the rotation position encoding, which is a hyperparameter of the large language model. Increasing the base value of the rotation position encoding can increase the input text sequence length of the large language model.

[0058] In step S403, the long text training data is used to train the modified pre-trained large language model to obtain a trained large language model.

[0059] The modified pre-trained large language model is trained using long text training data, which can be optionally trained in an unsupervised manner until the training end condition is met, to obtain a trained large language model.

[0060] In step S404, long text training data of a preset field is obtained, and the trained large language model is trained using the long text training data of the preset field to obtain a first large language model corresponding to the preset field.

[0061] The large language model trained using data of multiple fields is a general large language model, which learns general rules of language and extensive knowledge. In order to improve the accuracy of information processing of the large language model for the preset field, the long text data of the preset field is used to fine-tune the large language model.

[0062] In step S405, the maximum position number of the first large language model is increased in the configuration file of the first large language model corresponding to the preset field to obtain a second large language model.

[0063] The maximum length of the input text sequence of the second large language model is greater than the maximum length of the input text sequence of the first large language model.

[0064] In this embodiment, the pre-trained large language model is trained by obtaining long text training data and increasing the rotation angle base of the rotation position encoding, the length of the input text sequence is expanded, so that the trained large language model can process long text sequences, and the completeness and accuracy of the large language model in answering problems of long text dependence and multi-document comparison dependence are improved. Moreover, the long text training data of the preset field is used to fine-tune the large language model, which can improve the completeness and accuracy of the large language model in processing long text data of the preset field.

[0065] The embodiment of the present disclosure provides a question and answer method based on a large language model. The method in this embodiment can be applied to servers, terminal devices, platforms, devices, etc. with computing and processing capabilities. The server can be a server cluster or a single server, and can be a server deployed in the cloud or a local server.

[0066] As shown in FIG. 5, the flowchart of the question and answer method based on the large language model according to an embodiment of the present disclosure includes the following steps:

[0067] In step S501, the question information and task instructions are obtained, and a plurality of text blocks related to the question information are queried.

[0068] The problem information can be a question input by a user or information called from a preset storage area. The large language model needs to reason for the problem information to obtain reply information corresponding to the problem information.

[0069] The task instruction is a guide or command for the large language model to perform a specific task, indicating a specific task that the user of the large language model wants to complete or a required output type. For example, you are an xxx expert, please answer the user's question xxx according to the xxx query and according to the xxx requirement.

[0070] The multiple text blocks related to the problem information can be queried in different databases through multiple recall methods, and each text block corresponds to a relevance score representing the degree of relevance to the problem information. The higher the score, the higher the degree of relevance.

[0071] The relevance calculation can use the term frequency-inverse document frequency (TF-IDF) method, the text matching algorithm bm25, etc. The text is segmented and then the relevance score is calculated.

[0072] In step S502, the problem information, the task instruction, and the multiple text blocks are spliced to obtain a spliced text.

[0073] The length of the spliced text is greater than the maximum length of the input text sequence of the open-source pre-trained large language model.

[0074] In step S503, the spliced text is input into the trained large language model to obtain reply information output by the trained large language model.

[0075] The trained large language model is obtained by training using the training method in the above embodiments. Since the large language model in this embodiment is trained by obtaining long text training data and increasing the rotation angle base of the rotation position encoding, and the length of the input text sequence of the large language model is positionally expanded, the trained large language model can process long text data.

[0076] The method for question and answer based on the large language model provided in the embodiments of the present disclosure includes obtaining question information and task instructions, querying a plurality of text blocks related to the question information, splicing the question information, the task instructions, and the plurality of text blocks to obtain spliced text, inputting the spliced text into a trained large language model to obtain reply information output by the trained large language model, and the trained large language model is obtained by using the training method in the embodiments of the present disclosure. In the embodiments, the trained large language model is obtained by obtaining long text training data and increasing the rotation angle base of the rotation position coding, so that the length of the input text sequence can be expanded, and the completeness and accuracy of the large language model in answering the problems of long text dependence and multi-document comparison dependence can be improved.

[0077] In an implementation manner, the splicing of the question information, the task instructions, and the plurality of text blocks to obtain the spliced text includes: sorting the plurality of text blocks according to the relevance scores with the question information, and adding the sorted plurality of text blocks between the task instructions and the question information to obtain the spliced text.

[0078] Each text block corresponds to a relevance score representing the degree of relevance with the question information, and the text blocks are sorted in descending order of the relevance scores, and the sorted text blocks are spliced with the task instructions and the question information in the order of task execution, sorted text blocks, and question information to obtain the spliced text, which is input into the large language model to obtain the reply information.

[0079] In the embodiments, the plurality of text blocks are sorted when the spliced text is constructed, which can improve the accuracy of the output result of the large language model. Moreover, the sorting is performed according to the relevance scores, and compared with related technologies, the sorting model is not needed, and the system structure can be simplified.

[0080] In an implementation manner, the sorting of the plurality of text blocks according to the relevance scores with the question information and the adding of the sorted plurality of text blocks between the task instructions and the question information to obtain the spliced text includes: the plurality of text blocks include a plurality of first text blocks obtained by querying in a text keyword recall manner and a plurality of second text blocks obtained by querying in a vector recall manner, the plurality of first text blocks are sorted in descending order of the relevance scores with the question information, and the plurality of second text blocks are sorted in ascending order of the relevance scores with the question information; and the sorted plurality of first text blocks and the sorted plurality of second text blocks are added between the task instructions and the question information to obtain the spliced text.

[0081] The plurality of text blocks are obtained by querying in the vector recall manner (querying in a vector library), and the plurality of text blocks are obtained by querying in the text keyword recall manner (querying by using a search engine library, for example, an Elasticsearch library).

[0082] Since the large language model is more sensitive to the text closer to the task instruction and closer to the problem information part, and since the text block obtained according to the vector recall mode is more accurate than the text obtained according to the text keyword recall mode, the text blocks obtained according to the text keyword recall mode are sorted in descending order of the relevance score with the problem information, and the text blocks obtained according to the vector recall mode are sorted in ascending order of the relevance score with the problem information. Then, the text blocks obtained according to the text keyword recall mode are spliced after the task instruction, and the text blocks obtained according to the vector recall mode are spliced before the problem information to obtain spliced text. The spliced text obtained by inputting the large language model can improve the accuracy of the output result.

[0083] In an implementation manner, after the problem information, the task instruction and the plurality of text blocks are spliced to obtain the spliced text, the question and answer method based on the large language model further includes: adding a first placeholder after the task instruction in the spliced text and adding a second placeholder before the problem information in the spliced text to obtain a processed spliced text, the first placeholder is used to indicate a position corresponding to the task instruction, and the second placeholder is used to indicate a position corresponding to the problem information.

[0084] In order to make the large language model more easily distinguish the content in different positions (task instruction, text block, problem information), the first placeholder Pre-Tokens and the second placeholder Post-Tokens are inserted respectively. The number of placeholders of different large language models is different. For example, Pre-Tokens can be 400, and Post-Tokens can be 100. Finally, the spliced text is input to the large language model with long context (for example, supporting a maximum of 256k) to obtain the reply information.

[0085] In the embodiment, the large language model is more easily distinguished by inserting the placeholders, and the accuracy of the large language model is improved.

[0086] The embodiment of the present disclosure provides a question and answer method based on a large language model. The method in the embodiment can be applied to servers, terminal devices, platforms, devices, etc. with computing and processing capabilities. The server can be a server cluster or a single server, and can be a server deployed in the cloud or a local server.

[0087] As shown in FIG. 6, the flowchart of the question and answer method based on the large language model according to an embodiment of the present disclosure includes:

[0088] In step S601, the problem information and the task instruction are obtained.

[0089] Step S602, query a plurality of first text blocks related to the question information according to the text keyword recall mode.

[0090] Step S603, query a plurality of second text blocks related to the question information according to the vector recall mode.

[0091] Step S604, sort the plurality of first text blocks according to the relevance score from high to low, and sort the plurality of second text blocks according to the relevance score from low to high.

[0092] Step S605, add the sorted plurality of first text blocks and the sorted plurality of second text blocks between the task instruction and the question information to obtain a spliced text.

[0093] Wherein, the sequence length of the spliced text is greater than the maximum length of the input text sequence of the pre-trained large language model.

[0094] Step S606, add a first placeholder after the task instruction in the spliced text, and add a second placeholder before the question information in the spliced text to obtain a processed spliced text.

[0095] Wherein, the first placeholder is used to indicate the position corresponding to the task instruction, and the second placeholder is used to indicate the position corresponding to the question information.

[0096] Step S607, input the processed spliced text into the trained large language model to obtain the reply information output by the trained large language model.

[0097] Wherein, the trained large language model is trained by using the training method in the above embodiment.

[0098] In this embodiment, the trained large language model is trained by obtaining long text training data and increasing the rotation angle base of the rotation position encoding, and the input text sequence length of the trained large language model is position expanded, which can realize the length expansion of the input text sequence and improve the completeness and accuracy of the large language model in answering the problem of long text dependence and multi-document comparison dependence.

[0099] Corresponding to the application scenario and method of the method provided by the embodiments of the present disclosure, the embodiments of the present disclosure also provide a training device of a large language model. As shown in FIG. 7, it is a structural block diagram of a training device of a large language model according to an embodiment of the present disclosure, and the device comprises:

[0100] The data acquisition module 701 is configured to acquire long text training data, and the sequence length of the long text training data is greater than the maximum length of the input text sequence of the pre-trained large language model.

[0101] The parameter modification module 702 is configured to increase a rotation angle base of the rotation position encoding of the pre-trained large language model to obtain a modified pre-trained large language model.

[0102] The model training module 703 is configured to train the modified pre-trained large language model by using the long text training data to obtain a trained large language model.

[0103] The training device for the large language model provided in the embodiments of the present disclosure acquires long text training data, the sequence length of the long text training data is greater than the maximum length of the input text sequence of the pre-trained large language model, increases the rotation angle base of the rotation position encoding of the pre-trained large language model to obtain a modified pre-trained large language model, and trains the modified pre-trained large language model by using the long text training data to obtain a trained large language model. In the embodiments, the pre-trained large language model is trained by acquiring the long text training data and increasing the rotation angle base of the rotation position encoding. Since the rotation position encoding generates the position encoding vector by mathematical transformation, increasing the rotation angle base of the rotation position encoding can change the way of mathematical transformation, and further affect the generation of the position encoding vector, so that the trained large language model can effectively capture the position information when processing longer sequences, thereby allowing the large language model to process longer input text sequences, realizing the length expansion of the input text sequence, and improving the completeness and accuracy of the answer of the large language model on the problem of long text dependence and multi-document comparison dependence.

[0104] In an implementation manner, the training device for the large language model is further configured to acquire long text training data of a preset field, and train the trained large language model by using the long text training data of the preset field to obtain a first large language model corresponding to the preset field.

[0105] In an implementation manner, the training device for the large language model is further configured to increase the maximum position number of the first large language model in a configuration file of the first large language model corresponding to the preset field to obtain a second large language model, and the maximum length of the input text sequence of the second large language model is greater than the maximum length of the input text sequence of the first large language model.

[0106] The functions of each module in the embodiments of the present disclosure can be referred to the corresponding description in the above method, and have the corresponding beneficial effects, which will not be repeated here.

[0107] Corresponding to the application scenarios and methods of the method provided in the embodiments of the present disclosure, the embodiments of the present disclosure further provide a large language model-based question and answer device. As shown in FIG. 8, a structural block diagram of a large language model-based question and answer device according to an embodiment of the present disclosure, the device comprises:

[0108] The information obtaining module 801 is configured to obtain question information and task instructions, and query a plurality of text blocks related to the question information.

[0109] The text concatenating module 802 is configured to concatenate the question information, the task instructions, and the plurality of text blocks to obtain a concatenated text, and the sequence length of the concatenated text is greater than the maximum length of an input text sequence of the pre-trained large language model.

[0110] The information processing module 803 is configured to input the concatenated text into the trained large language model to obtain reply information output by the trained large language model, and the trained large language model is obtained by using the training method in the embodiments of the present disclosure.

[0111] The question and answer device based on the large language model provided in the embodiments of the present disclosure obtains question information and task instructions, queries a plurality of text blocks related to the question information, concatenates the question information, the task instructions, and the plurality of text blocks to obtain a concatenated text, the sequence length of the concatenated text is greater than the maximum length of an input text sequence of the pre-trained large language model, inputs the concatenated text into the trained large language model to obtain reply information output by the trained large language model, and the trained large language model is obtained by using the training method in the embodiments of the present disclosure. In the embodiments, the trained large language model is obtained by obtaining long text training data and increasing the rotation angle base of the rotation position encoding, the length of the input text sequence is expanded, and the completeness and accuracy of the large language model in answering the problems of long text dependence and multi-document comparison dependence are improved.

[0112] In an implementation manner, the text concatenating module 802 is configured to sort the plurality of text blocks according to the relevance scores of the question information, and add the sorted plurality of text blocks between the task instructions and the question information to obtain the concatenated text.

[0113] In an implementation manner, the text concatenating module 802 is specifically configured to: the plurality of text blocks include a plurality of first text blocks obtained by querying in a text keyword recall manner and a plurality of second text blocks obtained by querying in a vector recall manner, sort the plurality of first text blocks according to the relevance scores of the question information from high to low, and sort the plurality of second text blocks according to the relevance scores of the question information from low to high; and add the sorted plurality of first text blocks and the sorted plurality of second text blocks between the task instructions and the question information to obtain the concatenated text.

[0114] In an implementation manner, the large language model based question answering apparatus is further configured to: after concatenating the question information, the task instruction and the plurality of text blocks to obtain a concatenated text, adding a first placeholder after the task instruction in the concatenated text and adding a second placeholder before the question information in the concatenated text to obtain a processed concatenated text, the first placeholder being used to indicate a position corresponding to the task instruction, and the second placeholder being used to indicate a position corresponding to the question information.

[0115] The functions of the modules in the embodiments of the present disclosure can be referred to the corresponding description in the above method, and have the corresponding beneficial effects, which will not be repeated here.

[0116] FIG. 9 is a block diagram of an electronic device for implementing the embodiments of the present disclosure. As shown in FIG. 9, the electronic device includes a memory 910 and a processor 920, and the memory 910 stores a computer program executable on the processor 920. The processor 920 implements the method in the above embodiments when executing the computer program. The number of the memory 910 and the processor 920 can be one or more.

[0117] The electronic device further includes:

[0118] A communication interface 930 for communicating with external devices and transmitting data.

[0119] If the memory 910, the processor 920 and the communication interface 930 are independently implemented, the memory 910, the processor 920 and the communication interface 930 can be connected to each other through a bus and complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in FIG. 9, but it does not mean that there is only one bus or only one type of bus.

[0120] Optionally, in specific implementation, if the memory 910, the processor 920 and the communication interface 930 are integrated on a chip, the memory 910, the processor 920 and the communication interface 930 can complete communication between them through an internal interface.

[0121] The embodiments of the present disclosure provide a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method provided in the embodiments of the present disclosure.

[0122] The embodiment of the present disclosure provides a computer program product, the computer program product comprises a computer program, and the computer program is executed by a processor to implement the method provided in the embodiment of the present disclosure.

[0123] The embodiment of the present disclosure further provides a chip, the chip comprising a processor, the processor is used for calling and running instructions stored in a memory, so that a communication device installed with the chip executes the method provided in the embodiment of the present disclosure.

[0124] The embodiment of the present disclosure further provides a chip, comprising an input interface, an output interface, a processor and a memory, the input interface, the output interface, the processor and the memory are connected through an internal connection path, the processor is used for executing code in the memory, and when the code is executed, the processor is used for executing the method provided in the embodiment.

[0125] It should be understood that the processor described above can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It should be noted that the processor can be a processor supporting an advanced RISC machine (ARM) architecture.

[0126] Further, the memory can optionally include a read-only memory and a random access memory. The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memory. The non-volatile memory can include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory, among others. The volatile memory can include a random access memory (RAM), which is used as an external cache. By way of example, and not limitation, many forms of RAM are available. For example, a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a Sync link DRAM (SLDRAM), and a direct Rambus RAM (DR RAM), among others.

[0127] In the above-described embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the present disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium.

[0128] In the description of the disclosure, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the disclosure. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, different embodiments or examples described in the specification and the features of different embodiments or examples can be combined and combined by those skilled in the art without contradiction, if necessary.

[0129] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the disclosure, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0130] Any process or method described in the flowchart or otherwise described herein can be understood as representing a module, a segment or a portion of code including one or more executable instructions for implementing specific logical functions or processes. And the scope of the preferred embodiments of the disclosure includes additional implementations, in which the functions can be performed in the order shown or discussed, including in a substantially simultaneous manner or in reverse order according to the functions involved.

[0131] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions, or in conjunction with these instruction execution systems, devices or apparatus.

[0132] It should be understood that parts of the disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-described embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above-described embodiment method can be completed by a program instructing the relevant hardware, which can be stored in a computer-readable storage medium, which includes one or a combination of the steps of the method embodiment when executed.

[0133] In addition, each function unit in each embodiment of the present disclosure can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module. When the integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0134] The above is only an exemplary embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art can easily think of various changes or replacements within the technical range disclosed in the present disclosure, and these should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A method of training a large language model, wherein, The method comprises: obtaining long text training data, the sequence length of the long text training data being greater than the maximum length of the input text sequence of the pre-trained large language model; increasing the rotation angle base of the rotation position coding of the pre-trained large language model to obtain a modified pre-trained large language model; training the modified pre-trained large language model using the long text training data to obtain a trained large language model.

2. The method of claim 1, wherein, The method further comprises: obtaining long text training data of a preset field, and training the trained large language model using the long text training data of the preset field to obtain a first large language model corresponding to the preset field.

3. The method of claim 2, wherein, The method further comprises: increasing the maximum position number of the first large language model in the configuration file of the first large language model corresponding to the preset field to obtain a second large language model, the maximum length of the input text sequence of the second large language model being greater than the maximum length of the input text sequence of the first large language model.

4. A method of question answering based on a large language model, wherein, The method comprises: obtaining problem information and task instructions, and querying a plurality of text blocks related to the problem information; splicing the problem information, the task instructions, and the plurality of text blocks to obtain a spliced text, the sequence length of the spliced text being greater than the maximum length of the input text sequence of the pre-trained large language model; inputting the spliced text into the trained large language model to obtain reply information output by the trained large language model; wherein the trained large language model is trained using the training method of any one of claims 1-3.

5. The method of claim 4, wherein, The method further comprises: sorting the plurality of text blocks according to the relevance scores with the problem information, and adding the sorted plurality of text blocks between the task instructions and the problem information to obtain the spliced text.

6. The method of claim 5, wherein, The method further comprises: The plurality of text blocks comprise a plurality of first text blocks obtained by querying in a text keyword recall manner and a plurality of second text blocks obtained by querying in a vector recall manner, the plurality of first text blocks are sorted according to the relevance scores with the problem information from high to low, and the plurality of second text blocks are sorted according to the relevance scores with the problem information from low to high; adding the sorted plurality of first text blocks and the sorted plurality of second text blocks between the task instructions and the problem information to obtain the spliced text.

7. The method according to any one of claims 4-6, wherein, After the problem information, the task instructions, and the plurality of text blocks are spliced to obtain the spliced text, the method further comprises: adding a first placeholder after the task instructions in the spliced text and adding a second placeholder before the problem information in the spliced text to obtain a processed spliced text, the first placeholder being used to indicate the position corresponding to the task instructions, and the second placeholder being used to indicate the position corresponding to the problem information.

8. An electronic device, comprising: The electronic device includes a memory, a processor, and a computer program stored on the memory, and the processor implements the method of any one of claims 1-7 when executing the computer program.

9. A computer readable storage medium, wherein, The computer readable storage medium stores a computer program, and the computer program implements the method of any one of claims 1-7 when executed by a processor.

10. A computer program product, wherein, The computer program product includes a computer program, and the computer program implements the method of any one of claims 1-7 when executed by a processor.

Images