Method and device for training large language model based on long text

By performing blocked parallel processing of long text and introducing unsupervised training, the problem of insufficient understanding ability of large language models in long text processing is solved, the processing efficiency and accuracy is improved, the context window is expanded, and the execution of downstream tasks is enhanced Effect.

CN119004107APending Publication Date: 2024-11-22ALIPAY (HANGZHOU) INFORMATION TECH CO LTD

Patent Information

Application Number
CN202411123941.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When processing long text, existing large language models are difficult to effectively understand and analyze long text features, resulting in poor processing performance and limited context window length, and the inability to fully understand long text content and reason based on this.

Method used

By chunking and parallel processing of long text, using small encoders to encode text blocks, and introducing unsupervised self-encoding tasks and downstream tasks, the model's ability to retain original information of text blocks and the execution effect of downstream tasks is enhanced, and the model is improved Accuracy of long text context understanding.

Benefits of technology

The processing efficiency and accuracy of the large language model for long text is improved, the context processing window is expanded, the model's ability to capture long text semantics and the decoder's ability to interpret text block representations is enhanced, and the execution effect of downstream tasks is ensured.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004107A_ABST
    Figure CN119004107A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a method for training a large language model based on a long text, the large language model comprises a pre-trained target decoder, and the method comprises the following steps: carrying out text segmentation on a target long text to obtain a plurality of text blocks; and inputting each text block into a text encoder for encoding to obtain an encoding representation corresponding to each text block. And inputting each coding representation into a mapping network for processing to obtain a mapping representation mapped to the embedding space of the target decoder. The mapping representation of the multiple text blocks and the text representation corresponding to the first prompt text are input into a target decoder, a first answer text is obtained, and the first prompt text is used for indicating the large language model and resuming the target long text according to the mapping representation. And determining a first loss according to the first answer text and the target long text. A text encoder and the mapping network are updated according to a predicted loss, where the predicted loss includes the first loss.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of large language model training and long text expansion, and in particular, to a method and device for training a large language model based on long text. Background Art

[0002] In recent years, Large Language Models (LLMs) have shown impressive performance and have attracted great attention from academia and industry. This is mainly due to the wide applicability of LLMs in multiple application fields, including general language understanding, healthcare services, personalized teaching, and recommendation systems.

[0003] However, as LLM gradually moves towards the era of long text processing, how to effectively model and analyze context with long text characteristics has become a key research topic, with the goal of enabling LLM to accurately identify, deeply understand and respond appropriately to key semantics in long texts. In some application scenarios, this challenge of long text understanding is particularly prominent. For example, when users instruct LLM to summarize the content of a long report or extract key information from a book through question-answer interaction, LLM's long text processing capability is particularly important.

[0004] Therefore, we hope to have a solution that can improve the long text processing capability of LLM through technical means, so that it can understand the long text context efficiently and accurately. Summary of the invention

[0005] One or more embodiments of this specification describe a method and device for training a large language model based on long text. In a large language model with a decoder, a small encoder is introduced to process the blocks of the long text input sequence (hereinafter also referred to as text blocks) in parallel, and an unsupervised training mechanism is introduced. Through the auto-encoder task and the downstream task, the encoder's ability to retain the original information of the text block and promote the execution of downstream tasks are enhanced. In this way, by processing text blocks in parallel, the processing efficiency of long texts is improved, and through unsupervised training, the accuracy of the model's understanding of the long text context is improved, ensuring the execution effect of downstream tasks.

[0006] According to a first aspect, a method for training a large language model based on a long text is provided, wherein the large language model includes a pre-trained target decoder, and the method includes:

[0007] Perform text segmentation on the target long text to obtain multiple text blocks.

[0008] Each text block is input into the text encoder for encoding to obtain the encoding representation corresponding to each text block.

[0009] Each encoded representation is input into a mapping network for processing to obtain a mapping representation mapped to the embedding space of the target decoder.

[0010] The mapping representations of the multiple text blocks and the text representation corresponding to the first prompt text are input into the target decoder to obtain a first answer text, wherein the first prompt text is used to indicate the large language model to restate the target long text according to the mapping representation.

[0011] A first loss is determined according to the first answer text and the target long text.

[0012] The text encoder and the mapping network are updated according to the prediction loss, wherein the prediction loss includes the first loss.

[0013] In one implementation, the target decoder is fine-tuned according to the prediction loss.

[0014] In one implementation, the predicted loss further includes a second loss, and the method further includes:

[0015] further inputting a text representation corresponding to a second text into the target decoder to obtain a second answer text, wherein the second text includes a second prompt text and a second natural language question, and the second prompt text is used to instruct the large language model to answer the second natural language question using the multiple text blocks as background information;

[0016] The second loss is determined based on the label answer text corresponding to the second natural language question and the second answer text.

[0017] In one scenario of the above implementation, the predicted loss is a weighted sum of the first loss and the second loss.

[0018] In one implementation, the text segmentation of the target long text includes:

[0019] The target long text is segmented into a plurality of text blocks by taking a preset text length as a unit segmentation length.

[0020] In one implementation, the text segmentation of the target long text includes:

[0021] The target long text is divided into a plurality of text blocks using a preset text separator as a segmentation point, wherein the text separator includes at least one of the following: a comma, a period, a semicolon, and a line break.

[0022] In one scenario of the above implementation, the text representation is obtained by the following steps:

[0023] For any natural language text, an encoding operation is performed using the embedding layer included in the large language model to obtain a corresponding text representation.

[0024] In one implementation, the first answer text is obtained by the following steps:

[0025] The multiple text blocks are divided into a plurality of sub-texts each including a continuous text block.

[0026] For any sub-text, several corresponding mapping representations are input into the target decoder for restatement to obtain the restated sub-text.

[0027] A first answer text is determined based on the splicing of the restated sub-texts corresponding to the plurality of sub-texts.

[0028] In one implementation, the step of inputting each text block into a text encoder for encoding includes: encoding the input text blocks in a parallel processing manner through the text encoder.

[0029] According to a second aspect, there is provided a device for training a large language model based on a long text, the large language model comprising a pre-trained target decoder, the device comprising:

[0030] The text segmentation module is configured to perform text segmentation on the target long text to obtain multiple text blocks.

[0031] The encoding module is configured to input each text block into a text encoder for encoding to obtain an encoding representation corresponding to each text block.

[0032] The mapping module is configured to input each encoded representation into a mapping network for processing to obtain a mapping representation mapped to the embedding space of the target decoder.

[0033] The restatement module is configured to input the mapping representations of the multiple text blocks and the text representation corresponding to the first prompt text into the target decoder to obtain a first answer text, wherein the first prompt text is used to indicate the large language model, and restate the target long text according to the mapping representation.

[0034] The loss determination module is configured to determine a first loss based on the first answer text and the target long text.

[0035] An updating module is configured to update the text encoder and the mapping network according to a prediction loss, wherein the prediction loss includes the first loss.

[0036] According to a third aspect, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0037] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein an executable code is stored in the memory, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0038] In the method and device provided in the embodiments of the present specification, the long text context is first divided into multiple text blocks, and the encoder is used to encode each text block into a text block representation. Then, through an adapter module (mapping network), the text block representation output by the encoder is aligned with the input embedding space of the decoder in the LLM. On this basis, by setting dual training goals and using unsupervised learning strategies, the encoder, adapter and decoder are updated end-to-end to improve the long text processing performance of the LLM. The dual training goals set can, on the one hand, enhance the decoder's ability to capture semantics in long texts, and on the other hand, improve the decoder in the LLM's ability to interpret text block representations, thereby achieving the goal of expanding the LLM context processing window. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A schematic diagram of a training framework disclosed in an embodiment of this specification;

[0041] Figure 2 A schematic diagram of a long text segmentation architecture provided in an embodiment of this specification;

[0042] Figure 3 A flow chart of a method for training a large language model based on long text according to an embodiment of this specification;

[0043] Figure 4 A schematic diagram of a device for training a large language model based on long text according to an embodiment of the present specification. DETAILED DESCRIPTION

[0044] The solution provided by the embodiments of this specification is described below in conjunction with the accompanying drawings.

[0045] As mentioned above, in the field of large language models (LLMs), how to improve the ability to model and analyze long text context information has become an important research topic. Some existing technologies have many shortcomings in this regard. For example, the model has poor processing performance for long texts and cannot fully understand the content of long texts and make inferences based on them; the model has a limited window length for processing long text contexts, etc.

[0046] In some existing technologies, the size of the context window of LLM is increased by continuously pre-training and fine-tuning the model. For example, the context processing capability is expanded by modifying the position embedding information of LLM. This method often requires additional training process so that the model can support long texts, and usually, the model cannot understand the additional tokens well, and may also generate higher reasoning costs.

[0047] In other existing technologies, long texts are compressed into significantly shorter embeddings through context compression technology, thereby achieving the purpose of processing long texts. However, this method lacks the ability to expand long texts. When the compression rate reaches the limit, as the context text increases, the compressed embedding will also increase. In addition, this method often requires fine-tuning the parameters of LLM to make it suitable for specific application scenarios, and is not suitable for general LLM.

[0048] In order to solve the above problems, a method and device for training a large language model based on long text is proposed in the embodiments of this specification. By segmenting the long text, the long text is converted into text blocks of controllable size. By encoding multiple text blocks in parallel, the processing efficiency of the model for long text can be improved. By introducing unsupervised training tasks, the model's ability to extract original information from text blocks can be enhanced, and the execution effect of the model on downstream tasks can be improved.

[0049] Figure 1 A training framework diagram disclosed in the embodiment of this specification is shown. Figure 1, the long text is divided into multiple text blocks through text segmentation. In this operation, a suitable text segmentation method can be used to ensure that the text blocks are semantically consistent with the long text. After the multiple text blocks obtained by segmentation are processed by the text encoder, their respective encoding representations are generated. The encoding representations are further processed by the mapping network and mapped to the embedding space of the target decoder to obtain their respective corresponding mapping representations. The mapping representations will be used as the input of the target decoder for model training. One of the training tasks is to restate the long text, as shown in the "prompt text #1" in the figure. The prompt text is used to instruct the target decoder to restate the long text in natural language according to the input mapping representation. According to the obtained "answer text #1" and the original long text, the first loss can be determined. Another training task is to answer natural language questions, as shown in the "prompt text #2" in the figure. The prompt text is used to instruct the target decoder to answer the natural language questions in the prompt text with the text block corresponding to the input mapping representation as background information. According to the obtained "answer text #2" and the label answer text corresponding to the second natural language question, the second loss can be determined. Before being input into the target decoder, the above-mentioned prompt texts are processed into text representations that can be received by the target decoder through the embedding layer. Finally, the training loss is determined according to the first loss and the second loss, and the text encoder, the mapping network, and the target decoder are updated accordingly.

[0050] The above is an overview of the technical concept in an exemplary embodiment of this specification. Figure 3 In FIG. 1 , a flow chart of a method for training a large language model based on long text according to an embodiment of the present specification is shown. It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 3 In this embodiment, the large language model includes a pre-trained target decoder, and the method includes at least the following steps: Step S301: Perform text segmentation on the target long text to obtain multiple text blocks. Step S303: Input each text block into the text encoder for encoding to obtain the encoding representation corresponding to each text block. Step S305: Input each encoding representation into the mapping network for processing to obtain a mapping representation mapped to the embedding space of the target decoder. Step S307: Input the mapping representations of the multiple text blocks and the text representation corresponding to the first prompt text into the target decoder to obtain a first answer text, wherein the first prompt text is used to instruct the large language model to restate the target long text according to the mapping representation. Step S309: Determine a first loss based on the first answer text and the target long text. Step S311: Update the text encoder and the mapping network based on the prediction loss, wherein the prediction loss includes the first loss.

[0051] The specific implementation methods of the above steps will be described in detail below with reference to the accompanying drawings.

[0052] In step S301, the target long text is segmented to obtain multiple text blocks.

[0053] For a target long text context C i , which usually contains too much non-key text or noisy tokens, so it is necessary to use a text encoder to encode C i Encoding is performed to extract the core semantics of long texts.

[0054] As mentioned above, in order to achieve higher encoding efficiency, the target long text is segmented before being input into the text encoder. Figure 2 The long text segmentation architecture provided according to this embodiment is shown, and the target long text C i Contains multiple tokens t, which are the smallest processing unit of text. It should be noted that a token can be a word, a character, a subword (such as an affix), or any string with linguistic meaning in a specific context. In the text segmentation of long texts, certain segmentation rules can be used to subdivide long texts into multiple text fragments, which are called text blocks.

[0055] like Figure 2 As shown, white blocks represent words in a long text, and shaded boxes represent text blocks obtained by text segmentation. In one example, different text blocks may contain the same number of words or different numbers of words, for example, text blocks #1 and #2 in the figure both contain 4 words, while text block #3 contains 3 words. In other examples, different text blocks may also overlap with each other, for example, text blocks #4 and #5 in the figure have 2 overlapping words.

[0056] According to one implementation, the target long text is segmented into multiple text blocks using a preset text length as a unit segmentation length. Specifically, the segmentation rule is to pre-set the number of text characters contained in the text block, use the number as the unit segmentation length, and intercept the long text. The long text is evenly segmented into multiple text blocks, and each text block contains a fixed number of characters. For example, 20 characters are set as the unit segmentation length, then in the long text, every 20 characters are intercepted as a text block until the end of the text. The segmentation rule is simple and intuitive, and can be quickly applied to long text segmentation practice.

[0057] According to another implementation, the target long text is segmented into a plurality of text blocks using a preset text delimiter as a segmentation point. Specifically, the text delimiter is predefined in the segmentation rule, and the long text is divided into a plurality of text blocks accordingly. In the long text, the text content between two adjacent text delimiters is completely intercepted to form an independent text block. This text segmentation method can retain the complete semantics of the intercepted text content in the text block, which is conducive to accurately extracting and retaining semantic information in the subsequent text encoding process.

[0058] It should be noted that the text delimiter is a symbol or character string used to mark the beginning or end of a specific text segment in the text. It can be a space, punctuation mark (such as a comma, period, semicolon, quotation mark, etc.), or a special character (such as a tab, newline, etc.). The above examples do not represent a limitation on text delimiters. In reality, there are a variety of text delimiters, which are selected according to specific needs. Examples are not given one by one in the embodiments of this specification.

[0059] Thus, the long text C i By segmenting the text, multiple text blocks are obtained. Each text block contains several tokens.

[0060] As described above, in actual text segmentation practice, various flexibly customized segmentation rules can be used to segment long texts; the content and length of the segmented text blocks can also be flexibly adjusted according to specific needs. This specification embodiment does not make specific limitations here.

[0061] Next, in step S303, each text block is input into a text encoder for encoding to obtain a coding representation corresponding to each text block.

[0062] In this step, the long text C i The text blocks obtained after text segmentation are input into the text encoder, and each text block Extracted as an encoded representation It can be expressed as:

[0063]

[0064] Among them, ψ represents the text encoder, d ψ Represents the latent space dimension of the text encoder.

[0065] It should be noted that in order to make the encoded representation accurately contain the complete semantics of the text block and reflect the relative position relationship between the text blocks, the text encoder is usually trainable. It can simultaneously express the relative position and absolute position information of the tokens contained in the text block. The training tasks and corresponding training objectives will be explained below.

[0066] In addition, the encoding representation It can be a global representation of a text block. For example, a text encoder is used to encode the text in the text block to obtain the encoding of each token, and the encoding of the [CLS] token is taken as the representation of the entire text to represent the overall characteristics of the text block. Other fusion algorithms can also be used to obtain the encoding representation of the text block based on the encoding of each token.

[0067] In some embodiments, the text encoder may be a pre-trained text encoder, which can fully utilize the training accumulation of historical text corpus for the text encoder to obtain a more expressive coding representation and avoid training the text encoder from scratch.

[0068] According to one implementation, the text encoder is used to encode the input text blocks in a parallel processing manner. Since the text blocks input to the text encoder are text fragments obtained after text segmentation, they usually contain limited text characters. Therefore, any small text encoder can be used to process the input text blocks in parallel at the same time to achieve parallel and block processing of long texts. This not only expands the size of the context window, but also improves the efficiency of the text encoder in extracting semantics from long texts.

[0069] The various encoding representations obtained in the above steps will be input into the LLM model with the target decoder in the subsequent steps for semantic understanding and to answer natural language questions based on this. However, considering the possible differences in the embedding space between the text encoder and the target decoder, an "adapter" is required to map the encoding representation to the embedding space of the target decoder so that the encoding representation can be directly understood by the target decoder. Therefore, in step S305, each encoding representation is input into the mapping network for processing to obtain a mapping representation mapped to the embedding space of the target decoder.

[0070] In this step, the encoding representation After being processed by the mapping network, each mapping representation is obtained which can be directly input into the target decoder

[0071] MLP (Multi-layer Perceptron) can be used as a mapping network to map each coded representation obtained in the previous step to an embedding space that can be directly processed by the target decoder. In other embodiments, other network models with representation space conversion functions can also be used as mapping networks, and the embodiments of this specification do not specifically limit the selection of models.

[0072] In step S307, the mapping representations of the multiple text blocks and the text representation corresponding to the first prompt text are input into the target decoder to obtain a first answer text, wherein the first prompt text is used to indicate the large language model to restate the target long text according to the mapping representation.

[0073] In order to ensure that each mapping representation input to the target decoder effectively encodes the feature information of the corresponding text block, during training, the first prompt text is used to instruct the large language model to restate the content of the target long text in natural language based on the input mapping representation. The restated text generated by the target decoder is used as the first answer text, which is used in the loss function to determine the first loss in subsequent steps.

[0074] In some embodiments, before inputting into the target decoder, the first prompt text needs to be encoded into a corresponding text representation. For example, the first prompt text can be encoded using an embedding layer included in the large language model, and the embedding layer corresponds to the target decoder, so that a corresponding text representation can be generated in the embedding space of the target decoder, and the text representation can be directly processed and understood by the target decoder.

[0075] According to one implementation, since the target long text has too many characters, the target decoder cannot fully restate it at one time. Therefore, a sliding method is adopted to restate the context fragments each time according to the mapping representation of a small number of continuous text blocks until all text blocks are processed, thereby completing the restatement of the target long text.

[0076] Specifically, the plurality of text blocks are first divided into a plurality of sub-texts each containing a continuous text block, and each sub-text can be regarded as the amount of text that can be processed by the sliding window at a time. Then, for any sub-text, a plurality of mapping representations corresponding to the sub-text are input into the target decoder for restatement to obtain a restated sub-text. Finally, the first answer text is determined based on the splicing of the restated sub-texts corresponding to the plurality of sub-texts.

[0077] It should be noted that the continuous text blocks contained in the sub-text refer to the continuity between the text blocks. Since the mapping representation corresponding to the text block encodes the relative position and absolute position information of the word unit, in order to ensure the accuracy of the text semantics during the restatement process, when dividing the sub-text, the text blocks need to be continuously collected to ensure that the position information in the mapping representation is not destroyed. On the other hand, since in the text segmentation operation, the collected text blocks are not necessarily continuous on the target long text (this is explained above), the continuous text blocks do not necessarily represent a limitation on the continuity of the original characters in the target long text.

[0078] As for the sliding window mechanism, it can be understood that, according to the sliding window size and sliding step length used in practice, the mapping representations covered by the current sliding window may overlap with the mapping representations included in the previous sliding window processing, and vice versa. Corresponding to the restated sub-texts obtained under different sliding window mechanisms, corresponding different splicing strategies need to be adopted when splicing. For example, each time the sliding window covers 3 mapping representations, and the sliding step length is 2 mapping representations, then in each sliding window processing, there will be overlapping processing of a mapping representation with the previous one. Therefore, in the splicing of the restated sub-text, the repeatedly processed text needs to be spliced ​​and filtered accordingly to ensure that the target decoder can correctly restate the target long text.

[0079] The above steps describe the process of text segmentation, representation extraction, representation mapping, and text restatement of the target long text in some embodiments. After this series of operations, the restatement text (i.e., the first answer text) obtained contains the errors generated by the text encoder during data compression and representation extraction, and the target decoder during the restatement process. In the subsequent training, the goal is to reduce the difference or loss between the target long text and the first answer text, so that the LLM can restore the input text as accurately as possible.

[0080] In step S309, a first loss is determined according to the first answer text and the target long text.

[0081] It should be noted that the first loss is proportional to the text difference between the first answer text and the target long text. In some embodiments, the Next Token Prediction method can be used to predict the next most likely word by analyzing the word sequence that has appeared, and compare it with the word at the corresponding position in the target long text (which can be regarded as a label word, ground-truth token) to calculate the first loss. In a specific example, the first loss can be calculated using a cross-entropy loss function, that is, the probability distribution of the next word predicted by the model is compared with the probability distribution of the label word, and the first loss is determined by calculating the cross-entropy loss function.

[0082] Based on the first loss determined in the above steps, in step S311, the text encoder and the mapping network are updated according to the prediction loss, wherein the prediction loss includes the first loss.

[0083] In this step, the parameters of the text encoder and the mapping network can be adjusted according to the first loss, and the training goal is to minimize the first loss, that is, to reduce the difference between the first answer text and the target long text.

[0084] Furthermore, considering that the trained model will be used to execute downstream tasks, the execution effect of the trained model in downstream tasks can be improved by training the downstream tasks.

[0085] According to one embodiment, the prediction loss also includes a second loss, and the model parameters can be updated based on the loss determined by the difference between the answer text of the target decoder answering the natural language question based on the target long text as background knowledge and the label answer text.

[0086] Specifically, on the basis of inputting the mapping representations corresponding to the multiple text blocks into the target decoder, the text representation corresponding to the second text is also input into the target decoder to obtain a second answer text, wherein the second text includes a second prompt text and a second natural language question, and the second prompt text is used to instruct the large language model to answer the second natural language question with the multiple text blocks as background information; the second loss is determined based on the labeled answer text corresponding to the second natural language question and the second answer text.

[0087] Similar to the determination method of the first loss described above, the second loss is proportional to the text difference between the second answer text and the label answer text. In some embodiments, the Next Token Prediction method can be used to predict the next most likely word, compare it with the word (ground-truth token) at the corresponding position in the label answer text, and calculate the second loss. In a specific example, the second loss can be calculated using a cross-entropy loss function, that is, the probability distribution of the next word predicted by the model is compared with the probability distribution of the label word, and the second loss is determined by calculating the cross-entropy loss function.

[0088] In this embodiment, the training goal is to minimize the second loss, that is, to reduce the difference between the second answer text and the label answer text.

[0089] In one implementation of this embodiment, the predicted loss is the weighted sum of the first loss and the second loss. That is, in one training batch, the above two training tasks are respectively performed to obtain the first loss and the second loss. According to the respective weights of the training tasks, the weighted sum of the first loss and the second loss is calculated as the predicted loss. The loss weight of each training task needs to be defined as needed in specific practice, and is not specifically limited in the embodiments of this specification.

[0090] It should be noted that in this embodiment, answering natural language questions is used as an example to illustrate the method of introducing downstream task training in model training. In other embodiments, the downstream tasks to which the model can be applied are not limited to this, and can also be text generation, intelligent question answering, machine translation, etc. In practical applications, it is necessary to select corresponding training methods and appropriate loss functions to train the model according to different downstream tasks. They are not listed here one by one.

[0091] According to one implementation, the target decoder can also be fine-tuned according to the prediction loss to reduce the sensitivity of the target decoder to the position information contained in the input text block and improve the ability to understand the original text. During training, the text encoder, mapping network and target decoder are adjusted simultaneously to form an end-to-end model update.

[0092] In addition, during the parameter update process of the model, a LoRA-based parameter fine-tuning method can be used to freeze the parameters in the model and add a trainable bypass matrix to some network layers of the model. The bypass matrix can significantly reduce the number of parameters while maintaining the performance of the model. The model can be fine-tuned by fine-tuning the bypass matrix.

[0093] It should be understood that the above steps S303-S311 can be performed repeatedly, through multiple rounds of repeated representation extraction, representation mapping, text restatement or downstream task execution on the text block, to determine the prediction loss, and update the model. The updated model is used as the basis for the next round of training, and the next round of model training is performed. In this way, the text encoder, mapping network, and target decoder will continue to converge, improving the model's ability to understand the context of long texts, thereby ensuring the execution effect of downstream tasks.

[0094] In this specification, the word "first" in terms such as the first prompt text, the first answer text, etc., and the corresponding "second", "third" (if any), etc. in the text are merely for the convenience of distinction and description and do not have any limiting meaning.

[0095] The foregoing describes certain embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0096] Figure 4 Schematic diagram of a device for training a large language model based on long text according to an embodiment of this specification. The device 400 is deployed in a computing device, and the computing device can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. Figure 3 The large language model includes a pre-trained target decoder, and the device 400 includes:

[0097] The text segmentation module 401 is configured to perform text segmentation on the target long text to obtain multiple text blocks.

[0098] The encoding module 402 is configured to input each text block into a text encoder for encoding to obtain an encoding representation corresponding to each text block.

[0099] The mapping module 403 is configured to input each encoded representation into a mapping network for processing to obtain a mapping representation mapped to the embedding space of the target decoder.

[0100] The restatement module 404 is configured to input the mapping representations of the multiple text blocks and the text representation corresponding to the first prompt text into the target decoder to obtain a first answer text, wherein the first prompt text is used to indicate the large language model, and restate the target long text according to the mapping representation.

[0101] The loss determination module 405 is configured to determine a first loss according to the first answer text and the target long text.

[0102] The updating module 406 is configured to update the text encoder and the mapping network according to the prediction loss, wherein the prediction loss includes the first loss.

[0103] According to another embodiment of the present specification, there is also provided a computer program product, including a computer program / instruction, which implements the aforementioned combination when executed by a processor. Figure 3 The steps of the method.

[0104] According to another embodiment, the present specification further provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the aforementioned combination is implemented. Figure 3 The steps of the method.

[0105] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention can be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0106] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for training a large language model based on a long text, wherein the large language model includes a pre-trained target decoder, the method comprising: Perform text segmentation on the target long text to obtain multiple text blocks; Input each text block into a text encoder for encoding to obtain a coding representation corresponding to each text block; Input each encoded representation into a mapping network for processing to obtain a mapping representation mapped to the embedding space of the target decoder; Inputting the mapping representations of the multiple text blocks and the text representation corresponding to the first prompt text into the target decoder to obtain a first answer text, wherein the first prompt text is used to instruct the large language model to restate the target long text according to the mapping representation; Determine a first loss according to the first answer text and the target long text; The text encoder and the mapping network are updated according to the prediction loss, wherein the prediction loss includes the first loss.

2. The method according to claim 1, further comprising: The target decoder is fine-tuned according to the prediction loss.

3. The method according to claim 1, wherein the predicted loss further includes a second loss, and the method further includes: further inputting a text representation corresponding to a second text into the target decoder to obtain a second answer text, wherein the second text includes a second prompt text and a second natural language question, and the second prompt text is used to instruct the large language model to answer the second natural language question using the multiple text blocks as background information; The second loss is determined based on the label answer text corresponding to the second natural language question and the second answer text.

4. The method according to claim 3, wherein: The predicted loss is a weighted sum of the first loss and the second loss.

5. The method according to claim 1, wherein: The text segmentation of the target long text includes: The target long text is segmented into a plurality of text blocks by taking a preset text length as a unit segmentation length.

6. The method according to claim 1, wherein: The text segmentation of the target long text includes: The target long text is divided into a plurality of text blocks using a preset text separator as a segmentation point, wherein the text separator includes at least one of the following: a comma, a period, a semicolon, and a line break.

7. The method according to claim 1 or 3, wherein: The text representation is obtained by the following steps: For any natural language text, an encoding operation is performed using the embedding layer included in the large language model to obtain a corresponding text representation.

8. The method according to claim 1, wherein: The first answer text is obtained by following the steps: Dividing the plurality of text blocks into a plurality of sub-texts each comprising a continuous text block; For any sub-text, inputting its corresponding mapping representations into the target decoder for restatement to obtain a restated sub-text; A first answer text is determined based on the splicing of the restated sub-texts corresponding to the plurality of sub-texts.

9. The method according to claim 1, wherein: The step of inputting each text block into a text encoder for encoding includes: encoding the input text blocks in a parallel processing manner through the text encoder.

10. A device for training a large language model based on a long text, the large language model comprising a pre-trained target decoder, the device comprising: A text segmentation module is configured to segment the target long text into multiple text blocks; An encoding module configured to input each text block into a text encoder for encoding to obtain an encoding representation corresponding to each text block; A mapping module configured to input each encoded representation into a mapping network for processing to obtain a mapping representation mapped to the embedding space of the target decoder; a restatement module configured to input the mapping representations of the multiple text blocks and the text representation corresponding to the first prompt text into the target decoder to obtain a first answer text, wherein the first prompt text is used to indicate the large language model, and restate the target long text according to the mapping representation; a loss determination module, configured to determine a first loss according to the first answer text and the target long text; An updating module is configured to update the text encoder and the mapping network according to a prediction loss, wherein the prediction loss includes the first loss.

11. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

12. A computing device comprising a memory and a processor, characterized in that: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 9 is implemented.

Citation Information

Patent Citations

  • Information record data blocking method based on pre-training language model

    CN117763093A

  • Unsupervised Cross-Domain Data Augmentation for Long-Document Based Prediction and Explanation

    US20240185270A1

Cited By

  • Long text information processing method and system based on large model, medium, product and terminal

    CN121562565A

  • Long text information processing method and system based on large model, medium, product and terminal

    CN121562565B