Text generation method based on large model and training method of text generation model

By introducing memory state and attention mechanisms into the deep learning model and combining the loop mechanism, the problems of reduced long text inference effect and high computational complexity are solved, and high precision and low-cost text generation are achieved.

CN118940730APending Publication Date: 2024-11-12BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411045495.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

When existing deep learning models process long text, the inference effect is significantly reduced and the computational complexity is high, making it difficult to effectively capture long-distance semantic features.

Method used

The text generation method based on the big model is adopted to obtain memory state and embedding features, combine attention mechanism and loop mechanism to update hidden states, generate long-distance semantic transmission, and realize long-distance semantic transmission.

Benefits of technology

Without reducing the accuracy and efficiency of inference, high-precision generation of text of arbitrary length is achieved, the inference cost is reduced, and the generation of infinitely long text is supported.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940730A_ABST
    Figure CN118940730A_ABST
Patent Text Reader

Abstract

The invention provides a text generation method based on a large model, a training method and device of a text generation model, equipment, a medium and a program product, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, natural language processing, large models and the like. The specific implementation scheme of the text generation method comprises the steps of obtaining a memory state for a to-be-processed text; the memory state is generated based on a previous text of the to-be-processed text; taking the embedding feature of the to-be-processed text as an initial hiding state, and processing the memory state and the initial hiding state by adopting a first attention mechanism to obtain an updated hiding state; and based on the updated hidden state, generating a later text for the to-be-processed text.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as deep learning, natural language processing and large models, and in particular to a text generation method based on a large model and a training method, device, equipment, medium and program product for a text generation model. Background Art

[0002] With the development of computer technology and network technology, deep learning models are being used more and more widely, and have made breakthrough progress in various fields.

[0003] In the field of natural language processing, recursive models or transformer models can be used to capture long-range semantic features in text. However, as the length of the inference text increases, the inference effect will decrease significantly, and the inference complexity is high. Summary of the invention

[0004] The present disclosure aims to provide a large model-based text generation method and a training method, device, equipment, medium and program product for a text generation model that can ensure reasoning effect and reduce reasoning complexity.

[0005] According to the first aspect of the present disclosure, a text generation method based on a large model is provided, including: obtaining a memory state for a text to be processed; the memory state is generated based on a previous text of the text to be processed; using the embedded features of the text to be processed as the initial hidden state, and using a first attention mechanism to process the memory state and the initial hidden state to obtain an updated hidden state; and based on the updated hidden state, generating a subsequent text for the text to be processed.

[0006] According to a second aspect of the present disclosure, a training method for a text generation model is provided, comprising: obtaining sample texts, each sample text comprising a plurality of text segments obtained by dividing a long text; for each text segment in the plurality of text segments: obtaining a memory state for each text segment; the memory state is generated based on a previous text of each text segment; using the embedded features of each text segment as an initial hidden state, using an encoding network in a text generation model to process the memory state and the initial hidden state to obtain an updated hidden state; based on the updated hidden state, using an output layer in the text generation model to generate a subsequent text for each text segment; and training the text generation model based on the subsequent texts for the plurality of text segments and the sample texts, wherein the encoding network uses a first attention mechanism to process the memory state and the initial hidden state.

[0007] According to a third aspect of the present disclosure, a text generation device based on a large model is provided, including: a state acquisition module, used to acquire a memory state for a text to be processed; the memory state is generated based on a previous text of the text to be processed; a first state update module, used to use the embedded features of the text to be processed as the initial hidden state, and adopt a first attention mechanism to process the memory state and the initial hidden state to obtain an updated hidden state; and a text generation module, used to generate a subsequent text for the text to be processed based on the updated hidden state.

[0008] According to a fourth aspect of the present disclosure, a training device for a text generation model is provided, comprising: a sample acquisition module for acquiring sample texts, each sample text comprising a plurality of text segments obtained by dividing a long text; a state acquisition module for acquiring a memory state for each of the plurality of text segments, wherein the memory state is generated based on a previous text of each text segment; a first state update module for processing the memory state and the initial hidden state using an encoding network in a text generation model with an embedded feature of each text segment as an initial hidden state to obtain an updated hidden state; a text generation module for generating a subsequent text for each text segment using an output layer in the text generation model based on the updated hidden state; and a model training module for training the text generation model based on the subsequent texts and sample texts for the plurality of text segments, wherein the encoding network processes the memory state and the initial hidden state using a first attention mechanism.

[0009] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large model-based text generation method or the text generation model training method provided by the present disclosure.

[0010] According to the sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the large model-based text generation method or the text generation model training method provided by the present disclosure.

[0011] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program / instruction, wherein the computer program / instruction is stored on at least one of a readable storage medium and an electronic device, and wherein the computer program / instruction, when executed by a processor, implements the large model-based text generation method or the text generation model training method provided in the present disclosure.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 It is a schematic diagram of an application scenario of a text generation method based on a large model and a training method and device for a text generation model according to an embodiment of the present disclosure;

[0015] Figure 2 is a flowchart of a text generation method based on a large model according to an embodiment of the present disclosure;

[0016] Figure 3 is a schematic diagram of the principle of obtaining the updated hidden state according to an embodiment of the present disclosure;

[0017] Figure 4 is a schematic diagram of an implementation principle of a text generation method based on a large model according to an embodiment of the present disclosure;

[0018] Figure 5 is a flowchart of a method for training a text generation model according to an embodiment of the present disclosure;

[0019] Figure 6 is a schematic diagram of the principle of returning the gradient of the objective function according to an embodiment of the present disclosure;

[0020] Figure 7 is a structural block diagram of a text generation device based on a large model according to an embodiment of the present disclosure;

[0021] Figure 8 is a structural block diagram of a training device for a text generation model according to an embodiment of the present disclosure; and

[0022] Fig. 9 It is a block diagram of an electronic device used to implement the large model-based text generation method or the text generation model training method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] In the field of natural language processing, recursive models or transformer models can be used to capture long-range semantic features in text. For example, transformers can be used to generate text. However, due to the limitation of the encoding length of the attention mechanism, only shorter texts can usually be encoded, that is, the semantics of long texts cannot be memorized, and long texts cannot be accurately inferred. For example, long short-term memory networks can be used to generate text. However, since a series of complex gating operations are required for each token, the computational complexity is high and the inference efficiency is low.

[0025] In order to solve the problems existing in the related art, the present disclosure provides a text generation method based on a large model and a training method, device, equipment, medium, and program product of a text generation model. Figure 1 The application scenarios of the method and device provided by the present disclosure are described.

[0026] Figure 1 It is a schematic diagram of application scenarios of a large-model-based text generation method and a text generation model training method and device according to an embodiment of the present disclosure.

[0027] like Figure 1 As shown, the application scenario 100 of this embodiment may include an electronic device 110, which may be various electronic devices with processing functions, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, servers, and the like.

[0028] The electronic device 110 may have a text processing function for processing the input text 120 to predict the subsequent text 130 of the text 120. In one embodiment, the electronic device 110 may also have an intelligent voice function for converting a voice signal provided by a user into the text 120, generating the subsequent text 130 of the text, and converting the subsequent text 130 into a voice signal for playback, so as to achieve intelligent interaction with the user.

[0029] Exemplarily, the electronic device 110 may use a model that combines the attention mechanism and the loop mechanism to encode the text 120. In this way, the encoding length of the attention network is no longer limited to the local part, but can recursively accumulate long-term information, thereby affecting and correcting the subsequent inference process. That is, in the text generation scenario, the text generation model may include the large model that combines the attention mechanism and the loop mechanism.

[0030] Exemplarily, the electronic device 110 may also use the text generation method provided by the present disclosure to process the text 120, thereby generating a subsequent text 130. In this way, the semantics of a long text can be transferred, and an infinitely long text can be generated. Accordingly, the electronic device 110 may use the text generation model 140 provided by the present disclosure to implement the text generation method.

[0031] like Figure 1 As shown, the application scenario 100 may also include a server 150, which may be a background management server that supports the operation of client applications in the electronic device 110. The electronic device 110 may be connected to the server 150 through a network, and the network may include a wired or wireless communication link. The server 150 may also be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0032] For example, the server 150 may train the text generation model 140 using a large amount of long texts, and in response to an acquisition request from the electronic device 110 , send the trained text generation model 140 to the electronic device 110 , so that the electronic device 110 may use the text generation model 140 to generate the subsequent text 130 .

[0033] In one embodiment, the electronic device 110 may also send the text 120 to the server 150 , and the server 150 may process the text 120 using a trained text generation model to obtain a subsequent text 130 .

[0034] It should be noted that the text generation method based on the big model provided in the present disclosure can be executed by the electronic device 110, or by the server 150. Accordingly, the text generation device based on the big model provided in the present disclosure can be set in the electronic device 110, or in the server 150. The training method of the text generation model provided in the present disclosure can be executed by the server 150. Accordingly, the training device of the text generation model provided in the present disclosure can be set in the server 150.

[0035] It should be understood that Figure 1 The number and type of the electronic devices 110 and the server 150 in the embodiment are only for illustration. According to the implementation requirements, there may be any number and type of electronic devices 110 and the server 150.

[0036] The following will be combined Figure 2 to Figure 4 The text generation method based on a large model provided by the present disclosure is described in detail.

[0037] Figure 2 It is a flowchart of a text generation method based on a large model according to an embodiment of the present disclosure.

[0038] like Figure 2 As shown, the large model-based text generation method 200 of this embodiment may include operations S210 to S230.

[0039] In operation S210, a memory state of the text to be processed is obtained.

[0040] According to an embodiment of the present disclosure, the text to be processed may be text input by a user, or may be text obtained by converting a voice provided by a user, and the present disclosure does not limit this.

[0041] The memory state for the text to be processed can be a memory state obtained by processing other texts before the text to be processed is processed using a large model-based text generation method, or it can be a randomly generated memory state. In this embodiment, the memory state can be understood as the memory cells involved in LSTM. The memory state can, for example, express the semantic information of the previous text of the text to be processed. Specifically, the memory state can be generated based on the previous text of the text to be processed. When the text to be processed has no previous text, a randomly generated memory state can be obtained.

[0042] In one embodiment, any text semantic extraction principle may be used to extract semantic information from the previous text of the text to be processed, and the extracted semantic information may be used as the memory state.

[0043] In operation S220, the embedded features of the text to be processed are used as the initial hidden state, and the first attention mechanism is used to process the memory state and the initial hidden state to obtain an updated hidden state.

[0044] According to an embodiment of the present disclosure, the first attention mechanism may be a cross-attention mechanism or a unidirectional attention mechanism.

[0045] For example, in operation S220, the initial hidden state can be used as the query feature, the memory state can be used as the key feature and the value feature, and the calculation can be performed using the cross-attention mechanism, and the calculated result can be used as the updated hidden state.

[0046] For example, in operation S220, the initial hidden state can be used as a query feature, and the concatenated feature obtained by concatenating the initial hidden state and the memory state can be used as the key feature and the value feature. The cross-attention mechanism is used for calculation, and the calculated result is used as the updated hidden state.

[0047] In operation S230 , a subsequent text for the text to be processed is generated based on the updated hidden state.

[0048] After obtaining the updated hidden state, this embodiment may use a fully connected network or a normalized network to process the updated hidden state, thereby obtaining the subsequent text.

[0049] For example, the updated hidden state is set to H. In this embodiment, the softmax function in the following formula (1) can be used to process the updated hidden state, thereby generating the subsequent text Y t . Where W and b are network parameters.

[0050] Y t =Softmax(W·H t +b) Formula (1)

[0051] According to the embodiments of the present disclosure, since the memory state of the previous text generated based on the text to be processed is taken into account when generating text, the semantic information of the previous text of the text to be processed can be combined when generating text, which is conducive to the high-precision generation of long text. Specifically, since the attention mechanism is used when generating text, and the memory state based on the previous text generation is taken into account, the reasoning cost can be reduced compared to the technical solution of using LSTM to generate text; compared to the technical solution of generating text based only on the attention mechanism, longer semantic information can be captured. Overall, the text generation method based on the large model of the embodiment of the present disclosure combines the recursive idea and the attention mechanism, so it can realize the reasoning of text of arbitrary length without reducing the accuracy and generation efficiency.

[0052] In one embodiment, the above operations S220 and S230 may be implemented by using a large model.

[0053] In one embodiment, the above operation S210 can obtain the memory state in the following manner. In response to executing the text generation task for the previous text, the stored memory state is obtained. The stored memory state is generated based on the previous text of the text to be processed. It can be understood that when the text to be processed is a long text, the long text can be divided into multiple text segments, the text to be processed can be each text segment in the multiple text segments, and the previous text of the text to be processed is a text segment in the multiple text segments that is adjacent to the text to be processed and is located before the text to be processed.

[0054] For example, the disclosed embodiment can generate a memory state based on the text to be processed after each text generation task is executed, and store the memory state in a predetermined storage space in association with the identification information of the provider of the text to be processed. When acquiring the memory state for the text to be processed, this embodiment can first determine whether the associated memory state is stored in the predetermined storage space based on the identification information of the provider of the text to be processed. If so, it can be determined that the text generation task for the previous text has been executed, and the most recently stored memory state in the associated memory state can be acquired as the memory state for the text to be processed. In this way, even if the provider provides multiple texts to be processed with an associated relationship for text generation at intervals, the text generation process can also take into account the semantic information of the text to be processed that has been provided, thereby facilitating the improvement of the accuracy of the generated text and the improvement of the text generation experience.

[0055] In one embodiment, if the text generation task for the previous text has not been performed, the randomly generated memory state can be used as the memory state for the text to be processed. This embodiment can randomly generate a memory state in advance and store it in a predetermined storage space. If the associated memory state is not obtained from the predetermined storage space, the randomly generated memory state can be obtained from the predetermined storage space.

[0056] The following will be combined Figure 3 The principle of obtaining the updated hidden state is further expanded and described.

[0057] Figure 3 It is a schematic diagram of the principle of obtaining the updated hidden state according to an embodiment of the present disclosure.

[0058] In one embodiment, the text to be processed can be processed recursively so that the text generation process can learn the hierarchical features in the text to be processed, that is, to capture different abstract levels of the text to be processed in different recursive processes, so that more complex mathematical expressions can be gradually constructed, which is conducive to a better understanding of the text to be processed, thereby facilitating the improvement of the accuracy of the generated subsequent text.

[0059] For example, the acquired memory state may include a state sequence consisting of multiple memory sub-states, and the number of the multiple memory sub-states may be equal to the number of recursive times N for processing the text to be processed. L Equal. Among them, N L The value of can be 2, 4, 6, 8, etc., which is set according to actual needs, and the present disclosure does not limit this. The specific recursive process can be: first, the text to be processed X t Embedding(X t ) as the initial hidden state The first attention mechanism is used to deal with the initial hidden state and N L The first memory substate among the memory substates Processing to get the initial hidden state The next hidden state This next hidden state can then be As the current hidden state, the first attention mechanism is used to and N L The second memory substate in the memory substate Processing to obtain hidden state The next hidden state Similarly, the first attention mechanism is used to focus on the current hidden state and N L The i-th memory substate among the memory substates Process and get the current hidden state The next hidden state Until N L Each memory substate in N memory substates is processed, that is, recursively L times, will recurse N L The next hidden state obtained As the aforementioned updated hidden state.

[0060] In one embodiment, multiple encoding layers connected in sequence (i.e., stacked) may be set to update the hidden state, and the hidden state output by the last encoding layer is used as the updated hidden state. The multiple encoding layers are constructed based on the first attention mechanism. Specifically, Figure 3 As shown, in embodiment 300, a total of N L The encoding layer is used to encode the text to be processed to obtain the updated hidden state. Specifically, this embodiment can first encode the initial hidden state 301 and N L The first substate in the memory substate 311 is used as the input of the first coding layer 321 in the plurality of coding layers connected in sequence, and the first coding layer 321 generates the first updated hidden state 302. Then, the hidden state of the first update can be 302 and N L The second substate in the memory substate 312 is used as the input of the second encoding layer 322 in the plurality of encoding layers connected in sequence, and the second encoding layer 322 generates a second updated hidden state By analogy, we can L -1) encoding layer generates the (Nth L-1) updated hidden states 303 and N L The Nth memory substate L Sub-status 313 is the last coding layer (i.e., the Nth coding layer) among the multiple coding layers connected in sequence. L The input of the Nth coding layer 323) is L The Nth output of the coding layer 323 L Updated hidden states 304 is the updated hidden state obtained in the above operation S220. L A natural number.

[0061] According to an embodiment of the present disclosure, the first attention mechanism may be, for example, a unidirectional attention mechanism. The principle of using the first attention mechanism to update the hidden state may be referred to in the following formula (2). Wherein, Query represents the query feature, KV represents the key feature and the value feature, [,] represents the concat() operation on the feature, N L The expression function of the first attention mechanism adopted by the i-th encoding layer in the encoding layers. That is, when updating the hidden state, the current hidden state is used as the query feature, and the feature obtained by concatenating the i-th memory sub-state and the current hidden state is used as the key feature and value feature.

[0062]

[0063] Figure 4 It is a schematic diagram of the implementation principle of the text generation method based on a large model according to an embodiment of the present disclosure.

[0064] According to an embodiment of the present disclosure, after generating a subsequent text for a to-be-processed text, the memory state can be updated based on the acquired memory state and the hidden state obtained based on the to-be-processed text, so that when processing the subsequent text of the to-be-processed text, the memory state that can express the semantic information of the to-be-processed text can be acquired. In this way, the long-distance transmission of semantic information can be facilitated, and the generation of infinitely long text with high accuracy can be achieved.

[0065] For example, this embodiment may use a second attention mechanism to process the memory state and the initial hidden state to obtain an updated memory state. After obtaining the updated memory state, this embodiment may, for example, store the updated memory state in a predetermined storage space, so that when processing the subsequent text of the text to be processed, the memory state containing the semantic information of the text to be processed can be obtained from the predetermined storage space. For example, in the case where the memory state for the text to be processed is obtained from a predetermined storage space, this embodiment may update the memory state associated with the provider of the text to be processed stored in the predetermined storage space to the obtained updated memory state.

[0066] According to an embodiment of the present disclosure, the second attention mechanism may be, for example, a bidirectional attention mechanism. Since the bidirectional attention mechanism can combine forward and reverse context information, a more comprehensive semantic dependency relationship can be captured. Therefore, by adopting a bidirectional attention mechanism to update the memory state, the memory state can be made to better express long-distance semantic dependencies, and thus help improve the accuracy of the subsequent text generated based on the memory state.

[0067] In one embodiment, a recursive method can be used to obtain each sub-state in the updated memory state, so that the updated memory state can express the hierarchical features of the text to be processed, and the updated memory state can more accurately express the semantic information of the text to be processed. In this way, it is beneficial to improve the processing effect of the text to be processed provided subsequently and improve the accuracy of the generated subsequent text.

[0068] For example, the acquired memory state may include a state sequence consisting of multiple memory sub-states, and the number of the multiple memory sub-states may be equal to the number of recursions of updating the memory state. The number of recursions of updating the memory state may be equal to the number of recursions N of processing the text to be processed. L The process of recursively updating the memory state can be: using the second attention mechanism to update the initial hidden state (which can be understood as the 0th hidden state) and the 1st memory substate in the state sequence Processing is performed to obtain the first memory sub-state in the updated memory state The second attention mechanism is used to obtain the first hidden state in the process of obtaining the updated hidden state. and the second memory substate in the state sequence Processing is performed to obtain the second memory sub-state in the updated memory state Similarly, the second attention mechanism is used to obtain the (i-1)th hidden state in the process of obtaining the updated hidden state. and the i-th memory substate in the state sequence Processing is performed to obtain the i-th memory sub-state in the updated memory state Until the Nth in the updated memory state is obtained L Memory Substate

[0069] According to the embodiment of the present disclosure, the principle of using the second attention mechanism to update the memory state can be seen in the following formula (3). N L The expression function of the second attention mechanism used in the recursion. That is, when updating the memory state, the current memory state is used as the query feature, and the features obtained by concatenating the i-th memory sub-state and the (i-1)-th hidden state obtained in sequence are used as the key feature and value feature.

[0070]

[0071] In one embodiment, multiple encoding layers may be set to update the memory state. The multiple encoding layers are constructed based on the second attention mechanism, and each encoding layer outputs a memory sub-state in the updated memory state.

[0072] In one embodiment, when the text to be processed is a long text, the second attention mechanism can be used to obtain the updated memory state when the text length of the subsequent text to be generated is greater than the predetermined length, without waiting for the entire generation of the subsequent text for the text to be processed. The predetermined length can be set based on the length limit that the attention mechanism can encode, or it can be set according to actual needs, and the present disclosure does not limit this. By setting the predetermined length, the effective transmission of semantic information can be guaranteed, and the contextual semantic information of the text to be processed will not be effectively learned due to the excessive length of the text to be processed, and therefore the accuracy of the generated subsequent text can be guaranteed.

[0073] like Figure 4 As shown, in an embodiment 400, a plurality of first coding layers connected in sequence and a plurality of second coding layers connected in sequence can be used to process the text to be processed to obtain an updated hidden state and an updated memory state. In this embodiment 400, the text to be processed can be, for example, a long text. This embodiment 400 can pre-divide the text to be processed into a plurality of text segments with a sequence. Except for the last text segment in the order, the text length of each text segment can be, for example, N, that is, each text segment can be divided into N tokens. This embodiment can process a plurality of text segments with a sequence in sequence, and update the memory state once after generating the subsequent text of each text segment.

[0074] Specifically, Figure 4As shown, the number of the multiple text segments obtained by division is set to T. When the embodiment 400 performs the text generation task for the text to be processed, the text generation task can be performed for the first text segment 430 first, and the randomly generated memory state can be obtained. The memory state can be represented as a state sequence When performing the text generation task on the first text segment 430, the embedding feature of the first text segment 430 can be determined as the initial hidden state 431, then the first first encoding layer 411 of the stacked multiple first encoding layers is used to perform the initial hidden state based on the first attention mechanism 431 and Memory Substate 441 is processed to obtain the hidden state 432. Then, the second first encoding layer 412 is used to perform the hidden state based on the first attention mechanism. 432 and Memory Substate 442 is processed to obtain the hidden state By analogy, using the Nth L The first encoding layer 413 is based on the first attention mechanism to the hidden state 433 and Memory Substate 443 is processed to obtain the hidden state 434. The hidden state 434 is the updated hidden state obtained by processing the first text segment 430 and the memory state. At the same time, the first second encoding layer 421 of the plurality of second encoding layers can be used to perform the initial hidden state based on the second attention mechanism. 431 and Memory Substate 441 is processed to obtain the first memory sub-state in the updated memory state 451. Then, the second second encoding layer 422 is used to perform the hidden state based on the second attention mechanism. 432 and Memory Substate 442 is processed to obtain the second memory sub-state in the updated memory state 452. And so on, using the Nth L The second encoding layer 423 is based on the second attention mechanism to the hidden state 433 and Memory Substate 443 is processed to obtain the Nth in the updated memory state L Memory Substate 453. From this, we can get the updated memory state The updated memory state can be used as the memory state for the second text segment among the T text segments. The randomly generated memory state can be, for example, a memory state of size M×N L×D tensor, where M is the length of the memorized token and D is the dimension of the semantic features of each memorized token. By expanding the tensor, a state sequence can be obtained, and the size of each memorized sub-state in the state sequence is M×D.

[0075] Similarly, when performing the text generation task on the second text segment 460, the embedding feature of the second text segment 460 can be determined first as the initial hidden state 461, then the first first encoding layer 411 of the stacked multiple first encoding layers is used to perform the initial hidden state based on the first attention mechanism 461 and Memory Substate 451 is processed to obtain the hidden state 462. Then, the second first encoding layer 412 is used to perform the hidden state based on the first attention mechanism. 462 and Memory Substate 452 is processed to obtain the hidden state By analogy, using the Nth L The first encoding layer 413 is based on the first attention mechanism to the hidden state 463 and Memory Substate 453 is processed to obtain the hidden state 464. The hidden state 464 is the updated hidden state obtained by processing the second text segment 460 and the memory state. At the same time, the first second encoding layer 421 of the plurality of second encoding layers can be used to perform the initial hidden state based on the second attention mechanism. 461 and Memory Substate 451 is processed to obtain the first memory sub-state in the updated memory state 471. Then, the second second encoding layer 422 is used to perform the hidden state based on the second attention mechanism. 462 and Memory Substate 452 is processed to obtain the second memory sub-state in the updated memory state 472. And so on, using the Nth L The second encoding layer 423 is based on the second attention mechanism to the hidden state 463 and Memory Substate 453 is processed to obtain the Nth in the updated memory state L Memory substate 473. From this, we can get the updated memory state The updated memory state can be used as the memory state for the third text segment among the T text segments. Similarly, the text generation task can be performed for each of the T text segments.

[0076] According to an embodiment of the present disclosure, when the text generation task is performed for the first text segment 430, the updated hidden state is obtained. After 434, the output layer 480 can be used to update the hidden state 434 is processed to obtain the subsequent text for the first text segment 430, that is, the first subsequent text 491. Similarly, when the text generation task is performed for the second text segment 460, the updated hidden state is obtained. After 464, the output layer 480 can be used to update the hidden state 464 is processed to obtain the subsequent text for the second text segment 460, that is, the second subsequent text 492. By analogy, the subsequent text for the Tth text segment can be obtained.

[0077] According to an embodiment of the present disclosure, the output layer 480 may, for example, use the softmax function described in the above formula (1) to process the updated hidden state, which is not limited in the present disclosure.

[0078] In one embodiment, the size of the hidden state may be the same as the size of the memory sub-state, and the memory sub-state may be understood as a special hidden state.

[0079] Through the large model-based text generation method of the embodiment of the present disclosure, for a long text with a length of L, the upper limit of the inference cost is Θ((N+M).L), which is much lower than the inference cost Θ(L) of the existing mainstream text generation model. 2 ), the inference cost of text generation can be greatly reduced. And by updating the memory state, the technical solution of text generation model processing text of any length for text generation can be realized, and the inference efficiency and inference effect can be ensured not to be reduced.

[0080] In order to facilitate the implementation of the text generation method based on the large model, the present disclosure also provides a training method for the text generation model. Figure 5 The training method is described in detail.

[0081] Figure 5 It is a flowchart of a method for training a text generation model according to an embodiment of the present disclosure.

[0082] like Figure 5 As shown, the text generation model training method 500 of this embodiment may include operations S510 to S550.

[0083] In operation S510, a sample text is acquired.

[0084] According to an embodiment of the present disclosure, each sample text may include, for example, a plurality of text segments obtained by dividing a long text. A long text may refer to a text whose length is greater than a predetermined length, which is similar to the predetermined length described above and will not be described in detail here.

[0085] In operation S520 , for each text segment of the plurality of text segments, a memory state for each text segment is acquired.

[0086] The operation S520 may be to take each text segment as a text to be processed and first obtain the memory state for the text to be processed. It is understandable that the implementation principle of the operation S520 is similar to the implementation principle of the operation S210 described above, and will not be repeated here.

[0087] In operation S530, the embedding feature of each text segment is used as the initial hidden state, and the encoding network in the text generation model is used to process the memory state and the initial hidden state to obtain an updated hidden state.

[0088] Among them, the encoding network can be a network constructed based on the first attention mechanism, that is, the encoding network uses the first attention mechanism to process the memory state and the initial hidden state. The encoding network may include one encoding layer or multiple encoding layers connected in sequence, which is not limited in the present disclosure. The principle of the encoding network processing the memory state and the initial hidden state in operation S530 is similar to the implementation principle of operation S220 described above, and will not be repeated here.

[0089] In operation S540 , based on the updated hidden state, a subsequent text for each text segment is generated using an output layer in the text generation model.

[0090] The output layer may be a network layer constructed based on a normalization function (eg, a softmax function), etc. The implementation principle of operation S540 is similar to the implementation principle of operation S230 described above, and will not be described in detail here.

[0091] In operation S550 , a text generation model is trained based on subsequent texts and sample texts for a plurality of text segments.

[0092] According to an embodiment of the present disclosure, the tth text segment X among the plurality of text segments t For example, the token sequence {x t,1 , x t,2 , ..., x t,N The training goal of the text generation model is to predict the second token in the tth text segment based on the first token in the tth text segment. Thus, for the tth text segment, the generated subsequent text can be, for example, composed of the token sequence {x' t,2, x' t,3 , ..., x' t,N , x' t+1,1 That is, the subsequent text generated for the t-th text segment includes the predicted t-th text segment X t The 2nd to Nth tokens in the text segment X, and the (t+1)th text segment X t+1 In this way, all tokens except the first token in the sample text can be used as the true value when training the text generation model. In this way, self-supervised training of the text generation model can be achieved.

[0093] For example, this embodiment may use the following formula (4) to calculate the loss value generated when the text generation model performs the text generation task for the t-th text segment. Cross Entropy() in formula (4) represents the cross entropy loss function. This embodiment may use the function represented by the following formula (4) as the objective function, and train the text generation model with the goal of minimizing the objective function.

[0094] L t = CrossEntropy({x' t,2 , x' t,3 , ..., x' t,N , x' t+1,1}, {x t,2 , x t,3 ,...x t,N , x t+1,1})

[0095] Formula (4)

[0096] In this embodiment, the loss can be obtained by using formula (4) for multiple text segments, and the sum of the obtained multiple loss values ​​is used as the total loss of the text generation model, and the text generation model is trained with the goal of minimizing the total loss. It can be understood that the cross entropy loss function used when calculating the loss value is only used as an example to facilitate understanding of the present disclosure, and the present disclosure does not limit this.

[0097] According to an embodiment of the present disclosure, the memory state may include a state sequence consisting of multiple memory sub-states. The encoding network may include a first encoding sub-network, which includes multiple first encoding layers connected in sequence and constructed based on a first attention mechanism, for example, the N encoding layers described in the above embodiment 300 may include: Lcoding layers. In this embodiment, the operation S530 described above can be performed, for example, using the following principle: using the embedded features of each text segment as the initial hidden state, using the i-th coding layer in the multiple first coding layers to process the i-th memory sub-state in the state sequence and the current hidden state, and obtaining the next hidden state of the current hidden state. Among them, the updated hidden state is the next hidden state obtained by the last coding layer in the multiple first coding layers. This implementation principle is similar to the implementation principle described in embodiment 300, and will not be repeated here.

[0098] According to an embodiment of the present disclosure, when the text segment currently to be processed is the first text segment among multiple text segments, a randomly generated memory state can be obtained as the memory state for the text segment currently to be processed. When the text segment currently to be processed is a text segment other than the first text segment among multiple text segments, a stored memory state can be obtained. The stored memory state is obtained based on the previous text segment (specifically, it can be the previous text segment) of each text segment among the multiple text segments.

[0099] According to an embodiment of the present disclosure, the encoding network may further include a second encoding subnetwork constructed based on a second attention mechanism. In the process of training the text generation model, the second encoding subnetwork may also be used to process the memory state and the initial hidden state to obtain an updated memory state. The stored memory state is then updated to the updated memory state. The second attention mechanism may be a bidirectional attention mechanism, and the aforementioned first attention mechanism may be a unidirectional attention mechanism.

[0100] According to an embodiment of the present disclosure, the memory state may include a state sequence consisting of multiple memory sub-states, and the updated hidden state is the last hidden state of the multiple hidden states obtained in sequence. The second encoding sub-network may include multiple second encoding layers constructed based on the second attention mechanism, for example, it may include the N described in the above embodiment 400. L The second coding layer. The second coding sub-network is used to process the memory state and the initial hidden state to obtain the updated memory state. For example, the implementation principle can be: the i-th second coding layer among the multiple second coding layers is used to process the i-th memory sub-state in the state sequence and the obtained (i-1)-th hidden state to obtain the i-th memory sub-state in the updated memory state. The implementation principle of obtaining the updated memory state can be specifically referred to the aforementioned embodiment 400, which will not be repeated here.

[0101] According to an embodiment of the present disclosure, when training a text generation model, a function for calculating a loss value can be used as an objective function. The gradient of the objective function is fed back through a back propagation algorithm. Subsequently, the text generation model is trained based on the gradient obtained by the back propagation with the goal of minimizing the objective function. Among them, the back propagation algorithm is specifically based on the chain rule in calculus, and sequentially calculates and stores the gradients of intermediate variables and parameters of each layer in the text generation model related to the objective function along the order from the output layer to the input layer of the text generation model. This embodiment can determine the adjustment direction and adjustment amount of the network parameters in the text generation model based on the calculated and stored gradients, and adjust the network parameters based on this to achieve optimization of the text generation model.

[0102] In one embodiment, a back-propagation through time algorithm may be used to perform gradient backpropagation of the objective function, so as to expand the text generation model by time steps, thereby obtaining the dependency relationship between the model variables and network parameters of the text generation model, and applying back-propagation according to the chain rule to calculate and store the gradient.

[0103] It is understandable that the objective function is related to the difference between the subsequent text of the plurality of text segments and the target text in the sample text, where the target text is the text in the sample text corresponding to the subsequent text. For example, the objective function can be obtained based on formula (4) described above.

[0104] Figure 6 It is a schematic diagram of the principle of returning the gradient of the objective function according to an embodiment of the present disclosure.

[0105] According to an embodiment of the present disclosure, gradient cutoff points can be pre-set for multiple text segments, that is, a target text segment is selected from multiple text segments, and when the gradient is fed back to the target text segment, it will no longer be fed back, so as to prevent too many gradients from being recorded during the feeding back process, resulting in a situation where the video memory explodes.

[0106] Specifically, Figure 6 As shown, in the embodiment 600, the following truncation points {tl, ...tk, ...tK} can be set, that is, the tl-th, tk-th and tK-th text segments in the T text segments 610 are target text segments, tl, ...tk, ...tK∈[1,T]. This embodiment can determine the text segment sequence consisting of the text segment located between two adjacent target text segments and the two adjacent target text segments when returning the target text segment among the multiple text segments. For example, the first text segment sequence {X1, X2, ..., X tl}611, the second text segment sequence {X tl , X tl+1 , ..., X t2}612,...,{X tk , X tk+1 , ..., X t(k+1)}, and the Kth text segment sequence {X t(K-1) , X t(K-1)+1 , ..., X tK}613. It can be understood that the target text segment includes the last text segment X among the T text segments. T , for example, the target text segment X tK Actually, it is text segment X T . The target text segment may also include the first text segment X1, because the gradient will be naturally truncated after being fed back to the first text segment. Subsequently, the embodiment may use the time backpropagation algorithm to feed back the gradient of the objective function in each text segment sequence, and obtain a gradient as the gradient for the target text segment currently fed back. For example, by feeding back the gradient of the objective function in the first text segment sequence 611, a first gradient 621 for text segment X1 may be obtained. By feeding back the gradient of the objective function in the second text segment sequence 612, a first gradient 622 for text segment X1 may be obtained. tl The second gradient 622 of the objective function is returned in the Kth text segment sequence 613, and the gradient of the objective function for the text segment X can be obtained. t(K-1) The Kth gradient is 623.

[0107] Specifically, when returning the gradient of the objective function in a text segment sequence, the sum of the losses generated by the text generation model for the text segments in the text segment sequence can be calculated first. For example, for the kth text segment, the sum of the losses calculated is Then, the loss sum is derived from all network parameters of the text generation model to obtain the derivative GP L This embodiment can be the derivative GP L As a target text segment gradient.

[0108] After obtaining the gradients for all target text segments that are returned, the sum of all the obtained gradients can be used as the gradient 630 obtained by the return. Then, the text generation model is trained based on the gradient. It can be understood that for the last text segment X that is returned T , since there are no two adjacent target text segments returned, the gradient is not returned and calculated.

[0109] According to an embodiment of the present disclosure, in order to achieve truncation of gradient return, during the gradient return process, the embodiment can store the memory state As a parameter. In this way, when the gradient is passed back to It will be automatically truncated when .

[0110] It can be understood that the specific principle of returning the gradient of the objective function is only used as an example to facilitate understanding of the present disclosure, and the present disclosure does not limit this.

[0111] In one embodiment, due to direct gradient truncation, the training effect of the model may be reduced. In order to avoid the reduction of the training effect of the model, the embodiment can also derive the memory state of the target text segment that was last returned to the target text segment currently returned to obtain the gradient of the memory state. Then, based on the gradient and the change amount of the memory state for the target text segment that was last returned, determine the influence of the text segment after the target text segment that was last returned on the gradient for the target text segment that is currently returned. It is understandable that the change amount of the memory state for the target text segment that was last returned can be obtained by returning the gradient of the objective function in the text segment after the target text segment that was last returned. Then, based on the influence amount, adjust the gradient returned by deriving the loss. In this way, the model training effect can be unaffected by the gradient truncation, and the explosion of video memory can be avoided on the basis of ensuring the model training effect.

[0112] For example, the product of the gradient of the memory state and the amount of change of the memory state with respect to the target text segment that was last passed back can be used as the influence amount.

[0113] In one embodiment, you can first set That is, the change in the memory state for the last text segment among the T text segments is zero. This is because there are no other text segments after the last text segment. tk When the gradient of Derivative all network parameters of the text generation model to obtain the derivative GP L In addition, you can also Derivative all network parameters of the text generation model to obtain the derivative GP M Then, the following formula (5) is used to calculate the target text segment X that is currently returned. tk The gradient ΔP k .in, Memory state For target text segment X t(k+1) The amount of change.

[0114]

[0115] Finally, this embodiment can calculate the sum of all the gradients returned AP = ∑ k ΔP k , and get the gradient obtained by backpropagation.

[0116] Based on the text generation method based on the large model provided by the present disclosure, the present disclosure also provides a text generation device based on the large model. Figure 7 The device is described in detail.

[0117] Figure 7 It is a structural block diagram of a text generation device based on a large model according to an embodiment of the present disclosure.

[0118] like Figure 7 As shown, the large model-based text generation device 700 of this embodiment may include a first state acquisition module 710 , a first state update module 720 and a first text generation module 730 .

[0119] The first state acquisition module 710 is used to acquire the memory state for the text to be processed. The memory state is generated based on the previous text of the text to be processed. In one embodiment, the first state acquisition module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0120] The first state updating module 720 is used to use the embedded features of the text to be processed as the initial hidden state, and use the first attention mechanism to process the memory state and the initial hidden state to obtain an updated hidden state. In one embodiment, the first state updating module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0121] The first text generation module 730 is used to generate a subsequent text for the to-be-processed text based on the updated hidden state. In one embodiment, the first text generation module 730 can be used to perform the operation S230 described above, which will not be described in detail here.

[0122] According to an embodiment of the present disclosure, the memory state includes a state sequence consisting of multiple memory sub-states. The above-mentioned first state update module 720 may include: a state determination sub-module, which is used to use the embedded features of the text to be processed as the initial hidden state, and use the first attention mechanism to process the i-th memory sub-state and the current hidden state in the state sequence to obtain the next hidden state of the current hidden state; a first update sub-module, which is used to update the current hidden state to the next hidden state in response to the presence of unprocessed memory vectors in multiple memory vectors, and set i to i+1; and a second update sub-module, which is used to use the next hidden state as the updated hidden state in response to the absence of unprocessed memory vectors in multiple memory vectors, wherein i is a natural number less than or equal to the total number of multiple memory sub-states.

[0123] According to an embodiment of the present disclosure, the above-mentioned first state acquisition module 710 may include: a first acquisition sub-module, used to acquire a stored memory state in response to executing a text generation task for a previous text; the stored memory state is generated based on the previous text; and a second acquisition sub-module, used to acquire a randomly generated memory state in response to not executing a text generation task for a previous text.

[0124] According to an embodiment of the present disclosure, the above-mentioned text generation device 700 may also include: a second state update module, which is used to process the memory state and the initial hidden state using a second attention mechanism in response to the text length of the generated subsequent text being greater than a predetermined length to obtain an updated memory state; and a first storage update module, which is used to update the stored memory state to an updated memory state, wherein the first attention mechanism is a unidirectional attention mechanism and the second attention mechanism is a bidirectional attention mechanism.

[0125] According to an embodiment of the present disclosure, the memory state includes a state sequence consisting of multiple memory sub-states; the updated hidden state is the last hidden state of the multiple hidden states obtained in sequence. The above-mentioned second state update module can be specifically used to: use the second attention mechanism to process the i-th memory sub-state in the state sequence and the obtained (i-1)-th hidden state to obtain the i-th memory sub-state in the updated memory state, where i is a natural number less than or equal to the total number of multiple memory sub-states; the first memory sub-state is obtained by processing the initial hidden state.

[0126] Based on the training method of the text generation model provided by the present disclosure, the present disclosure also provides a training device for the text generation model. Figure 8 The device is described in detail.

[0127] Figure 8 It is a structural block diagram of a training device for a text generation model according to an embodiment of the present disclosure.

[0128] like Figure 8 As shown, the text generation model training device 800 of this embodiment may include a sample acquisition module 810 , a second state acquisition module 820 , a third state update module 830 , a second text generation module 840 and a model training module 850 .

[0129] The sample acquisition module 810 is used to acquire sample texts, each sample text including a plurality of text segments obtained by dividing a long text. In one embodiment, the sample acquisition module 810 can be used to perform the operation S510 described above, which will not be described in detail here.

[0130] The first state acquisition module 820 is used to acquire the memory state of each text segment in the plurality of text segments. The memory state is generated based on the previous text of each text segment. In one embodiment, the first state acquisition module 820 can be used to perform the operation S520 described above, which will not be described in detail here.

[0131] The third state update module 830 is used to use the embedded features of each text segment as the initial hidden state, and use the encoding network in the text generation model to process the memory state and the initial hidden state to obtain an updated hidden state. Among them, the encoding network uses the first attention mechanism to process the memory state and the initial hidden state. In one embodiment, the third state update module 830 can be used to perform the operation S530 described above, which will not be repeated here.

[0132] The first text generation module 840 is used to generate the subsequent text for each text segment based on the updated hidden state using the output layer in the text generation model. In one embodiment, the first text generation module 840 can be used to perform the operation S540 described above, which will not be repeated here.

[0133] The model training module 850 is used to train the text generation model based on the subsequent text and sample text for multiple text segments. In one embodiment, the model training module 850 can be used to perform the operation S550 described above, which will not be repeated here.

[0134] According to an embodiment of the present disclosure, the memory state includes a state sequence consisting of multiple memory sub-states; the encoding network includes a first encoding sub-network; the first encoding sub-network includes multiple first encoding layers connected in sequence and constructed based on a first attention mechanism. The above-mentioned third state update module 830 can be specifically used to: use the embedded features of each text segment as the initial hidden state, use the i-th encoding layer among the multiple first encoding layers to process the i-th memory sub-state and the current hidden state in the state sequence, and obtain the next hidden state of the current hidden state, wherein the updated hidden state is the next hidden state obtained by the last encoding layer among the multiple first encoding layers, and i is a natural number less than or equal to the total number of multiple memory sub-states.

[0135] According to an embodiment of the present disclosure, the above-mentioned second state acquisition module 820 may include: a third acquisition sub-module, used to obtain a randomly generated memory state in response to each text segment being the first text segment among multiple text segments; a fourth acquisition sub-module, used to obtain a stored memory state in response to each text segment being a text segment other than the first text segment among multiple text segments, wherein the stored memory state is obtained based on the previous text segment of each text segment among the multiple text segments.

[0136] According to an embodiment of the present disclosure, the encoding network also includes a second encoding subnetwork constructed based on the second attention mechanism. The training device 800 of the above-mentioned text generation model may also include: a fourth state update module, which is used to process the memory state and the initial hidden state using the second encoding subnetwork to obtain an updated memory state; and a second storage update module, which is used to update the stored memory state to the updated memory state, wherein the first attention mechanism is a unidirectional attention mechanism and the second attention mechanism is a bidirectional attention mechanism.

[0137] According to an embodiment of the present disclosure, the memory state includes a state sequence consisting of multiple memory sub-states; the updated hidden state is the last hidden state of the multiple hidden states obtained in sequence; the second coding subnetwork includes multiple second coding layers constructed based on the second attention mechanism. The above-mentioned fourth state update module can be specifically used to: use the i-th second coding layer among the multiple second coding layers to process the i-th memory sub-state in the state sequence and the obtained (i-1)-th hidden state to obtain the i-th memory sub-state in the updated memory state, where i is a natural number less than or equal to the total number of the multiple memory sub-states; the hidden state processed by the first second coding layer is the initial hidden state.

[0138] According to an embodiment of the present disclosure, the above-mentioned model training module 850 may include: a gradient feedback submodule, which is used to feedback the gradient of the objective function using the time backpropagation algorithm; and a training submodule, which is used to train the text generation model based on the gradient obtained by feedback with the goal of minimizing the objective function, wherein the objective function is related to the difference between the subsequent text of multiple text segments and the target text in the sample text, and the target text is the text corresponding to the subsequent text.

[0139] According to an embodiment of the present disclosure, the above-mentioned gradient feedback submodule may include: a sequence determination unit, which is used to determine, in response to a target text segment among multiple text segments, a text segment sequence consisting of a text segment located between two adjacent target text segments that have been returned and two adjacent target text segments; the target text segment includes the last text segment among the multiple text segments, and there are multiple target text segments; a gradient feedback unit, which is used to use a time backpropagation algorithm to feedback the gradient of the objective function within the text segment sequence, and determine the gradient for the target text segment currently returned; and a gradient determination unit, which is used to determine the gradient obtained by feedback based on the sum of multiple gradients for multiple target text segments.

[0140] According to an embodiment of the present disclosure, the training device of the above-mentioned text generation model may also include: a state determination module, which is used to determine, in response to the target text segment being returned, a memory state obtained from the target text segment that was last returned for the target text segment currently being returned as the target state; a gradient determination module, which is used to determine the gradient of the target state; and an influence determination module, which is used to determine the influence of the text segment after the last target text segment that was last returned among multiple text segments on the gradient of the target text segment currently being returned based on the gradient of the target state and the change of the target state for the target text segment that was last returned, wherein the gradient for the target text segment currently being returned is determined based on the sum of the gradient and the influence obtained by returning the gradient of the target function within the text segment sequence.

[0141] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals. In the technical solution of this disclosure, the user's authorization or consent is obtained before obtaining or collecting user personal information.

[0142] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0143] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement a large model-based text generation method or a text generation model training method of an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0144] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0145] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0146] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a text generation method based on a large model or a training method for a text generation model. For example, in some embodiments, a text generation method based on a large model or a training method for a text generation model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the text generation method based on the large model or the training method for the text generation model described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute a large model-based text generation method or a text generation model training method in any other appropriate manner (eg, by means of firmware).

[0147] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0148] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0149] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0150] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0151] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0152] A computer system may include a client and a server. The client and the server are generally far away from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. Among them, the server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server of a distributed system, or a server combined with a blockchain.

[0153] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0154] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A text generation method based on a large model, comprising: Get the memory status for the text to be processed; The memory state is generated based on the previous text of the text to be processed; Using the embedded features of the text to be processed as the initial hidden state, using the first attention mechanism to process the memory state and the initial hidden state to obtain an updated hidden state; as well as Based on the updated hidden state, a subsequent text for the text to be processed is generated.

2. The method according to claim 1, wherein: The memory state includes a state sequence consisting of multiple memory sub-states; the first attention mechanism is used to process the memory state and the initial hidden state to obtain an updated hidden state, including: Using the embedded features of the text to be processed as the initial hidden state, the first attention mechanism is used to process the i-th memory sub-state and the current hidden state in the state sequence to obtain the next hidden state of the current hidden state; In response to the presence of an unprocessed memory vector in the plurality of memory vectors, updating the current hidden state to the next hidden state, and setting i to i+1; and In response to the absence of an unprocessed memory vector in the plurality of memory vectors, taking the next hidden state as the updated hidden state, Wherein, i is a natural number that is less than or equal to the total number of the plurality of memory sub-states.

3. The method according to claim 1 or 2, wherein: The step of obtaining the memory state of the text to be processed includes: In response to executing a text generation task for the previous text, obtaining a stored memory state; the stored memory state is generated based on the previous text; In response to not having performed a text generation task for the previous text, a randomly generated memory state is obtained.

4. The method according to claim 3, further comprising: In response to the text length of the generated subsequent text being greater than a predetermined length, a second attention mechanism is used to process the memory state and the initial hidden state to obtain an updated memory state; as well as updating the stored memory state to the updated memory state, Among them, the first attention mechanism is a unidirectional attention mechanism, and the second attention mechanism is a bidirectional attention mechanism.

5. The method according to claim 4, wherein: The memory state includes a state sequence consisting of multiple memory sub-states; the updated hidden state is the last hidden state among the multiple hidden states obtained in sequence; The adopting the second attention mechanism to process the memory state and the initial hidden state to obtain the updated memory state includes: The second attention mechanism is used to process the i-th memory sub-state in the state sequence and the obtained (i-1)-th hidden state to obtain the i-th memory sub-state in the updated memory state. Wherein, i is a natural number less than or equal to the total number of the plurality of memory sub-states; the first memory sub-state is obtained by processing the initial hidden state.

6. A method for training a text generation model, comprising: Acquire sample texts, each sample text comprising a plurality of text segments obtained by dividing a long text; For each text segment in the plurality of text segments: Acquire a memory state for each of the text segments; the memory state is generated based on a previous text of each of the text segments; Using the embedding features of each text segment as the initial hidden state, the encoding network in the text generation model is used to process the memory state and the initial hidden state to obtain an updated hidden state; as well as Based on the updated hidden state, using the output layer in the text generation model to generate subsequent text for each text segment; as well as Based on the subsequent texts for the multiple text segments and the sample texts, the text generation model is trained, The encoding network uses a first attention mechanism to process the memory state and the initial hidden state.

7. The method according to claim 6, wherein: The memory state includes a state sequence consisting of multiple memory sub-states; the encoding network includes a first encoding sub-network; the first encoding sub-network includes multiple first encoding layers connected in sequence and constructed based on a first attention mechanism; the encoding network in the text generation model is used to process the memory state and the initial hidden state to obtain an updated hidden state, which includes: Using the embedding features of each text segment as the initial hidden state, using the i-th encoding layer of the multiple first encoding layers to process the i-th memory sub-state and the current hidden state in the state sequence to obtain the next hidden state of the current hidden state, The updated hidden state is the next hidden state obtained by the last coding layer in the multiple first coding layers, and i is a natural number less than or equal to the total number of the multiple memory sub-states.

8. The method according to claim 6 or 7, wherein: The acquiring of the memory status for each text segment comprises: In response to each text segment being a first text segment among the plurality of text segments, obtaining a randomly generated memory state; In response to each of the text segments being other text segments in the plurality of text segments except the first text segment, obtaining a stored memory state, The stored memory state is obtained based on the previous text segment of each text segment in the multiple text segments.

9. The method according to claim 8, wherein: The encoding network also includes a second encoding subnetwork constructed based on a second attention mechanism; the method also includes: Processing the memory state and the initial hidden state using the second encoding subnetwork to obtain an updated memory state; and updating the stored memory state to the updated memory state, Among them, the first attention mechanism is a unidirectional attention mechanism, and the second attention mechanism is a bidirectional attention mechanism.

10. The method according to claim 9, wherein: The memory state includes a state sequence consisting of multiple memory sub-states; the updated hidden state is the last hidden state of the multiple hidden states obtained in sequence; the second encoding sub-network includes multiple second encoding layers constructed based on the second attention mechanism; The adopting the second encoding sub-network to process the memory state and the initial hidden state to obtain an updated memory state comprises: Using the i-th second coding layer among the plurality of second coding layers to process the i-th memory sub-state in the state sequence and the obtained (i-1)-th hidden state to obtain the i-th memory sub-state in the updated memory state, Wherein, i is a natural number less than or equal to the total number of the plurality of memory sub-states; the hidden state processed by the first second encoding layer is the initial hidden state.

11. The method according to claim 6, wherein: The training of the text generation model based on the subsequent texts of the plurality of text segments and the sample texts comprises: Use the back-propagation through time algorithm to propagate the gradient of the objective function; and With the goal of minimizing the objective function, the text generation model is trained based on the gradient obtained by backpropagation. The objective function is related to the difference between the subsequent texts of the multiple text segments and the target text in the sample text, and the target text is the text corresponding to the subsequent text.

12. The method according to claim 11, wherein: The step of using the time back propagation algorithm to return the gradient of the objective function includes: In response to a target text segment among the plurality of text segments transmitted back, determining a text segment sequence consisting of a text segment located between two adjacent target text segments transmitted back and the two adjacent target text segments; the target text segment includes the last text segment among the plurality of text segments, and there are a plurality of target text segments; Using the time back-propagation algorithm to propagate the gradient of the objective function back within the text segment sequence, and determining the gradient for the target text segment currently propagated back; and The returned gradient is determined based on a sum of multiple gradients for multiple target text segments.

13. The method according to claim 12, further comprising: In response to returning the target text segment, determining a memory state acquired for a target text segment that was previously returned before the target text segment currently returned as the target state; determining a gradient of the target state; as well as Based on the gradient of the target state and the change amount of the target state with respect to the last returned target text segment, determining the influence amount of the text segment after the last returned target text segment among the multiple text segments on the gradient with respect to the currently returned target text segment, The gradient for the target text segment currently returned is determined based on the sum of the gradient obtained by returning the gradient of the target function within the text segment sequence and the influence amount.

14. A text generation device based on a large model, comprising: A state acquisition module, used to acquire the memory state of the text to be processed; The memory state is generated based on the previous text of the text to be processed; A first state updating module is used to use the embedding feature of the text to be processed as the initial hidden state, and use a first attention mechanism to process the memory state and the initial hidden state to obtain an updated hidden state; as well as A text generation module is used to generate a subsequent text for the text to be processed based on the updated hidden state.

15. The device according to claim 14, wherein: The memory state includes a state sequence consisting of multiple memory sub-states; the first state update module includes: A state determination submodule, configured to use the embedded features of the text to be processed as the initial hidden state, and use the first attention mechanism to process the i-th memory substate and the current hidden state in the state sequence to obtain the next hidden state of the current hidden state; A first updating submodule, configured to update the current hidden state to the next hidden state and set i to i+1 in response to the presence of an unprocessed memory vector in the plurality of memory vectors; and A second updating submodule is configured to, in response to the absence of an unprocessed memory vector in the plurality of memory vectors, use the next hidden state as the updated hidden state, Wherein, i is a natural number that is less than or equal to the total number of the plurality of memory sub-states.

16. The device according to claim 14 or 15, wherein: The state acquisition module includes: A first acquisition submodule is used to acquire a stored memory state in response to executing a text generation task for the previous text; the stored memory state is generated based on the previous text; The second acquisition submodule is used to acquire a randomly generated memory state in response to the text generation task for the previous text not being performed.

17. The apparatus according to claim 16, further comprising: A second state updating module is configured to process the memory state and the initial hidden state using a second attention mechanism to obtain an updated memory state in response to the length of the generated subsequent text being greater than a predetermined length; as well as a storage update module, configured to update the stored memory state to the updated memory state, Among them, the first attention mechanism is a unidirectional attention mechanism, and the second attention mechanism is a bidirectional attention mechanism.

18. The device according to claim 17, wherein: The memory state includes a state sequence consisting of multiple memory sub-states; the updated hidden state is the last hidden state among the multiple hidden states obtained in sequence; the second state updating module is used to: The second attention mechanism is used to process the i-th memory sub-state in the state sequence and the obtained (i-1)-th hidden state to obtain the i-th memory sub-state in the updated memory state. Wherein, i is a natural number less than or equal to the total number of the plurality of memory sub-states; the first memory sub-state is obtained by processing the initial hidden state.

19. A training device for a text generation model, comprising: A sample acquisition module is used to acquire sample texts, each sample text includes a plurality of text segments obtained by dividing a long text; A state acquisition module, used for acquiring a memory state for each text segment of the plurality of text segments; The memory state is generated based on the previous text of each text segment; A first state updating module is used to use the embedding feature of each text segment as an initial hidden state, and use the encoding network in the text generation model to process the memory state and the initial hidden state to obtain an updated hidden state; A text generation module, configured to generate subsequent text for each text segment using an output layer in the text generation model based on the updated hidden state; as well as A model training module, used for training the text generation model based on the subsequent texts of the multiple text segments and the sample text, The encoding network uses a first attention mechanism to process the memory state and the initial hidden state.

20. The device according to claim 19, wherein The memory state includes a state sequence consisting of multiple memory sub-states; the encoding network includes a first encoding sub-network; the first encoding sub-network includes multiple first encoding layers connected in sequence and constructed based on a first attention mechanism; the first state update module is used to: Using the embedding features of each text segment as the initial hidden state, using the i-th encoding layer of the multiple first encoding layers to process the i-th memory sub-state and the current hidden state in the state sequence to obtain the next hidden state of the current hidden state, The updated hidden state is the next hidden state obtained by the last coding layer in the multiple first coding layers, and i is a natural number less than or equal to the total number of the multiple memory sub-states.

21. The device according to claim 19 or 20, wherein: The state acquisition module includes: A first acquisition submodule, configured to acquire a randomly generated memory state in response to each text segment being the first text segment among the plurality of text segments; a second acquisition submodule, configured to acquire a stored memory state in response to each of the text segments being other text segments in the plurality of text segments except the first text segment, The stored memory state is obtained based on the previous text segment of each text segment in the multiple text segments.

22. The device according to claim 21, wherein The encoding network also includes a second encoding subnetwork constructed based on a second attention mechanism; the device also includes: A second state updating module, configured to process the memory state and the initial hidden state using the second encoding sub-network to obtain an updated memory state; and a storage update module, used to update the stored memory state to the updated memory state, Among them, the first attention mechanism is a unidirectional attention mechanism, and the second attention mechanism is a bidirectional attention mechanism.

23. The device according to claim 22, wherein: The memory state includes a state sequence consisting of multiple memory sub-states; the updated hidden state is the last hidden state of the multiple hidden states obtained in sequence; the second encoding sub-network includes multiple second encoding layers constructed based on the second attention mechanism; the second state update module is used to: Using the i-th second coding layer among the plurality of second coding layers to process the i-th memory sub-state in the state sequence and the obtained (i-1)-th hidden state to obtain the i-th memory sub-state in the updated memory state, Wherein, i is a natural number less than or equal to the total number of the plurality of memory sub-states; the hidden state processed by the first second encoding layer is the initial hidden state.

24. The device according to claim 19, wherein The model training module includes: A gradient feedback submodule, for feedback the gradient of the objective function using a time back-propagation algorithm; and The training submodule is used to train the text generation model based on the gradient obtained by backpropagation with the goal of minimizing the objective function. The objective function is related to the difference between the subsequent texts of the multiple text segments and the target text in the sample text, and the target text is the text corresponding to the subsequent text.

25. The device according to claim 24, wherein: The gradient return submodule includes: a sequence determination unit, configured to determine, in response to a target text segment among the plurality of text segments, a text segment sequence consisting of a text segment located between two adjacent target text segments that have been returned and the two adjacent target text segments; the target text segment includes the last text segment among the plurality of text segments, and there are a plurality of target text segments; A gradient feedback unit, configured to feedback the gradient of the objective function in the text segment sequence using the time backpropagation algorithm to determine the gradient for the target text segment currently being fed back; and The gradient determination unit is used to determine the gradient obtained by the return based on the sum of multiple gradients for multiple target text segments.

26. The apparatus according to claim 25, further comprising: A state determination module, configured to determine, in response to the target text segment being returned, a memory state acquired for the target text segment that was previously returned before the target text segment currently being returned, as a target state; A gradient determination module, used to determine the gradient of the target state; as well as an influence determination module, configured to determine, based on the gradient of the target state and the change amount of the target state with respect to the target text segment that was last returned, an influence amount of a text segment after the target text segment that was last returned among the multiple text segments on the gradient with respect to the target text segment that is currently returned, The gradient for the target text segment currently returned is determined based on the sum of the gradient obtained by returning the gradient of the target function within the text segment sequence and the influence amount.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 13.

29. A computer program product, comprising a computer program / instruction, wherein the computer program / instruction is stored on at least one of a readable storage medium and an electronic device, and when the computer program / instruction is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Citation Information

Cited By

  • Question and answer processing method and device, medium and equipment

    CN119599030A