Text sequence generation method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to a text sequence generation method and device, equipment and a storage medium. The method includes the steps that in response to completion of generation of a first set of text tokens, feature representations of all the text tokens in the first set of text tokens are obtained to obtain a first set of feature representations, and the first set of feature representations are configured to indicate first conditional probability distribution of a target text token; on the basis of each feature representation in the first set of feature representations, regenerating the feature representation of each text token in the first set of text tokens to obtain a second set of feature representations, the second set of feature representations being configured to indicate conditional probability distribution of the target text token; generating a target text token in the text token sequence based on the second group of feature representations; and determining a target text sequence based on the generated text token sequence.

Description

TECHNICAL FIELD

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to a method, apparatus, device and storage medium for text sequence generation. BACKGROUND

[0002] Text generation technology is a key technology in the field of natural language processing, which can automatically generate text content conforming to grammar, structure and semantic rules by using computers. The implementation of text generation technology is mainly based on machine learning models, which simulate natural language generation ability by learning a large number of samples. With the rapid development of the Internet, this technology plays an increasingly important role in automatic summarization, machine translation and other fields. SUMMARY

[0003] In a first aspect of the present disclosure, a method for text sequence generation is provided, comprising: in the generation process of a target text token in a text token sequence, in response to the completion of the generation of a first group of text tokens, obtaining the feature representation of each text token in the first group of text tokens to obtain a first group of feature representations, wherein the first group of feature representations are configured to indicate a first conditional probability distribution of the target text token; based on each feature representation in the first group of feature representations, regenerating the feature representation of each text token in the first group of text tokens to obtain a second group of feature representations, the second group of feature representations being configured to indicate a second conditional probability distribution of the target text token; generating the target text token in the text token sequence based on the second conditional probability distribution indicated by the second group of feature representations; and determining a target text sequence based on the generated text token sequence.

[0004] In a second aspect of the present disclosure, a feature representation acquisition module is provided, configured to, in the generation process of a target text token in a text token sequence, in response to the completion of the generation of a first group of text tokens, obtain the feature representation of each text token in the first group of text tokens to obtain a first group of feature representations, wherein the first group of feature representations are configured to indicate a first conditional probability distribution of the target text token; a feature representation correction module configured to, based on each feature representation in the first group of feature representations, regenerate the feature representation of each text token in the first group of text tokens to obtain a second group of feature representations, the second group of feature representations being configured to indicate a second conditional probability distribution of the target text token; a text token generation module configured to generate the target text token in the text token sequence based on the second conditional probability distribution indicated by the second group of feature representations; and a text sequence determination module configured to determine a target text sequence based on the generated text token sequence.

[0005] In a third aspect of the disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of the disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon a computer program, the computer program being executable by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of the disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method according to the first aspect of the disclosure.

[0008] It is to be understood that the particulars shown herein are by way of example and for purposes of illustrative discussion of the embodiments of the present disclosure only and are not intended to limit the scope of the present disclosure to the particular embodiment illustrated. Other BRIEF DESCRIPTION OF DRAWINGS

[0009] The above-mentioned and other features and advantages of various embodiments of the present disclosure will become more apparent by reference to the following detailed description taken in conjunction with the accompanying drawings. In the drawings, like reference numerals designate like elements, wherein:

[0010] Figure 1 a schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 a flowchart showing an example process for a method for text sequence generation according to some embodiments of the present disclosure;

[0012] Figure 3 a schematic structural block diagram of an apparatus for text sequence generation according to some embodiments of the present disclosure; and

[0013] Figure 4 a block diagram of an electronic device in which one or more embodiments of the present disclosure can be implemented. DETAILED DESCRIPTION

[0014] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure will be thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0015] It should be noted that the titles of any sections / sub-sections provided herein are not limiting. Various embodiments are described throughout this document and any type of embodiment can be included under any section / sub-section. Furthermore, embodiments described in any section / sub-section can be combined with any other embodiments described in the same section / sub-section and / or different section / sub-section in any manner.

[0016] In the description of embodiments of the present disclosure, the term "includes" and its derivatives, such as "including," should be understood in an open, inclusive sense, that is, "including, but not limited to." The term "based on" should be understood as "based at least in part on." The term "one embodiment" or "an embodiment" should be understood as "at least one embodiment." The term "some embodiments" should be understood as "at least some embodiments." Other explicitly and implicitly recited definitions can also be found below. The terms "first," "second," and the like can refer to different or identical objects. Other explicit and implicit definitions can also be found below.

[0017] Data of users, acquisition and / or use of data, etc. can be involved in embodiments of the present disclosure. These aspects all comply with corresponding laws and regulations and relevant provisions. In embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are performed on the premise that the user is aware of and confirms. Accordingly, when implementing embodiments of the present disclosure, the type of data or information that can be involved, the scope of use, the scenario of use, etc. should be informed to the user and authorized by the user in a proper manner according to relevant laws and regulations. The specific informing and / or authorization manner can vary according to actual situations and application scenarios, and the scope of the present disclosure is not limited in this respect.

[0018] In the present specification and embodiments, if personal information processing is involved, it will be processed on the premise of legality (for example, obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and only within the scope prescribed or agreed. Users refuse to process personal information other than the necessary information required for basic functions, which will not affect the user's use of basic functions.

[0019] As briefly described above, the implementation of the text generation technique is mainly based on a machine learning model. In the process of text generation, a commonly used solution is to utilize an autoregressive-based decoder. Such a decoder is capable of predicting the next text token based on the previous text tokens, thereby generating the entire text token sequence step by step. Specifically, the machine learning model will first generate the text token sequence corresponding to the input content step by step through the autoregressive decoder. Once the text token sequence is generated, the machine learning model will further utilize these text tokens to construct the complete text sequence for the input content. This step is the final output of the text generation technique, and the generated text sequence is relevant to the input content and semantically coherent.

[0020] Although the autoregressive-based decoder is widely used in the process of text generation, it also has some drawbacks.

[0021] Specifically, the conventional autoregressive-based decoder first determines the conditional probability of the nth text token in the text token sequence based on the first n-1 text tokens (or the prefix sequence of the nth text token) when calculating the nth text token. Then, the specific content of the nth text token is determined based on the determined conditional probability. In simple terms, this scheme mainly considers the association between the nth text token and the prefix sequence when determining the conditional probability of the nth text token, i.e., attention calculation. However, this scheme ignores the association of the generated prefix sequence. In other words, it does not fully consider the mutual dependence and influence between the first n-1 text tokens. This results in the generation of each text token based on the previous text tokens in the decoding process, so any previous errors will be accumulated and amplified in the subsequent decoding process.

[0022] Due to the above-mentioned problems of cumulative error and ignoring the internal association of the prefix sequence, the conventional autoregressive-based decoder is difficult to achieve the best decoding effect, which may result in the generated text being unsmooth, contradictory, or lacking coherence, thereby affecting the user experience and text quality.

[0023] In view of this, embodiments of the present disclosure provide a scheme for text sequence generation. According to the scheme, first, in the generation process of a target text token in a text token sequence, in response to the completion of the generation of a first set of text tokens, a feature representation of each text token in the first set of text tokens is obtained to obtain a first set of feature representations, where the first set of feature representations is configured to indicate a first conditional probability distribution of the target text token. Then, based on each feature representation in the first set of feature representations, the feature representation of each text token in the first set of text tokens is regenerated to obtain a second set of feature representations, where the second set of feature representations is configured to indicate a second conditional probability distribution of the target text token. Subsequently, based on the second conditional probability distribution indicated by the second set of feature representations, the target text token in the text token sequence is generated. Finally, based on the generated text token sequence, a target text sequence is determined.

[0024] It will be understood more clearly through the following description that, in the process of generating a target text token (for example, an nth text token in a text token sequence), the scheme provided by embodiments of the present disclosure effectively reduces the cumulative error existing in a traditional autoregressive decoder by regenerating the feature representations of a first set of text tokens (for example, the first n-1 text tokens in the text token sequence, that is, a prefix sequence of the target text token), and fully considers the association relationship between the text tokens in the prefix sequence. This not only helps to improve the coherence and rationality of the final output target text sequence, but also significantly improves the quality of the target text sequence and the user experience.

[0025] Various example implementations of the scheme will be described in detail below in combination with the accompanying drawings.

[0026] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. Referring to Figure 1 , the example environment 100 can include an electronic device 120.

[0027] In this example environment 100, a user can interact with the electronic device 120 via a terminal device. As an example, the user can provide input content 110 to the electronic device 120 via the terminal device. After receiving the input content 110, the electronic device 120 can use a machine learning model 121 to generate a corresponding text token sequence 122 by predicting text tokens one by one based on the input content 110. Further, the electronic device 120 generates a target text sequence 130 for the input content 110 based on the text token sequence 122.

[0028] In some embodiments, the terminal device can be any type of mobile terminal, fixed terminal, or portable terminal including a mobile handset, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media player, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a game device, or any combinations of the aforementioned and the like, including accessories and peripherals related thereto, or any combinations thereof. In some embodiments, the terminal device 110 can also be capable of supporting any type of interface to the user (such as "wearable" circuitry, etc.).

[0029] The electronic device 120 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content distribution network, and big data and artificial intelligence platform. The electronic device 120 may, for example, include a computing system / server such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like. The electronic device 120 can provide background services for applications in the terminal device that support content presentation.

[0030] A communication connection can be established between the electronic device 120 and the terminal device. The communication connection can be established by wired or wireless means. The communication connection can include, but is not limited to, a Bluetooth connection, a mobile network connection, a universal serial bus connection, a wireless fidelity connection, and the like, and embodiments of the present disclosure are not limited in this regard. In embodiments of the present disclosure, the electronic device 120 and the terminal device can achieve signaling interaction through the communication connection therebetween.

[0031] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only, and do not imply any limitation on the scope of the present disclosure.

[0032] Figure 2 A flowchart of an example process 200 for a method of text sequence generation according to some embodiments of the present disclosure is shown. The process 200 can be implemented at the electronic device 120.

[0033] Referring to Figure 2In box 210, during the generation of the target text token in the text token sequence 122, the electronic device 120, in response to the completion of the generation of the first set of text tokens, acquires the feature representations of each text token in the first set of text tokens to obtain a first set of feature representations, wherein the first set of feature representations is configured to indicate a first conditional probability distribution of the target text tokens. As an example, the text token sequence 122 may be constructed by a machine learning model 121 in response to the detection of input content 110.

[0034] In some embodiments, the input content 110 may be raw data or information provided by the user to the machine learning model 121. For example, the input content 110 may be data in various forms such as text, images, and audio, depending on the application scenario and task requirements of the machine learning model 121. For example, in a text generation task, the input content 110 may be a text description or a topic, and the machine learning model 121 may generate a corresponding target text sequence 130 based on this content.

[0035] In some embodiments, the machine learning model 121 can construct a text token sequence 122 based on a self-attention mechanism. This machine learning model 121 can predict the next text token based on previous text tokens, thereby progressively generating the entire text token sequence 122. As an example, once the text token sequence 122 is generated, the electronic device 120 can use the machine learning model 121 to construct a complete target text sequence 130 for the input content 110, based on these text tokens, such that the target text sequence 130 is relevant to and semantically coherent with the input content 110.

[0036] In some embodiments, the first set of text tokens may also be referred to as the prefix sequence of the target text token. Once the machine learning model 121 has completed the generation of the first set of text tokens, the electronic device 120 can obtain the feature representation of each text token in this set of text tokens (i.e., the first set of feature representations) and generate the target text token based on the feature representation of this set of text tokens.

[0037] In some embodiments, based on a set of feature representations of the prefix sequence of the nth text token (e.g., the first to the (n-1)th text tokens), the electronic device 120 can determine a conditional probability distribution for the nth text token, where n is a positive integer. This conditional probability distribution can be configured to instruct the machine learning model 121 to determine the text token most likely to be the nth text token (or the next most likely text token). The first conditional probability distribution is also the conditional probability distribution determined by the electronic device 120 based on a first set of feature representations. As an example, the first conditional probability distribution can be configured to instruct the machine learning model 121 to determine the text token most likely to be the target text token.

[0038] To more clearly reveal the principles of the embodiments of this disclosure, the process of determining the first set of feature representations will be described below.

[0039] Assume the first set of text tokens Y n-1 Includes n-1 text tokens y i Let i = 1, 2, ..., n-1. As an example, the first set of text tokens Y... n-1 This can be expressed using Formula 1:

[0040] Y n-1 =(y1,y2,…,y n-1 );

[0041] Formula 1 in the first group of text tokens Y n-1 In the middle, each text token y i Each can be represented by a single feature h. i To express. These features represent h i The combination constitutes the first set of feature representations H. n-1 As an example, the first set of features represents H. n-1 This can be expressed using Formula 2:

[0042] H n-1 =(h1,h2,…,h n-1 );

[0043] Formula 2 serves as an example, with the first set of features representing H. n-1 The feature representation h in i Before generating the target text token, the machine learning model 121 may use a decoder based on a self-attention mechanism to process each of the currently generated text tokens y. i The determined vector features. It should be noted that the embodiments of this disclosure related to the decoder based on the self-attention mechanism will be described in detail below, and therefore will not be repeated here.

[0044] In box 220, electronic device 120 is based on the first set of feature representations H n-1 Each feature in h represents i Regenerate the first set of text tokens Y n-1 The feature representations of each text token are used to obtain the second set of feature representations H. n-1 The second set of features represents H. n-1 ′ is configured as the second conditional probability distribution indicating the target text token.

[0045] Based on the content described above, it is known that machine learning model 121 generates text token y. n-2 At that time, text token y n-1 Not yet generated, therefore, in the first set of text tokens Yn-1 In the text token y n-2 It is actually based on text tokens y1, y2, ..., y n-3 And generated. That is, text token y. n-2 The generation process does not actually take into account the text token y n-1 Accordingly, the text token y n-2 Feature representation h n-2 It also did not take into account the text token y n-1 Feature representation h n-1 And so on, text tokens y1, y2, ..., y n-3 The feature representations are h1, h2, ..., h n-3 It also did not take into account the text token y n-1 Feature representation h n-1 .

[0046] Based on this, embodiments of this disclosure, through block 220, can be based on feature representations h1, h2, ..., h n-1 Regenerate the first set of text tokens Y n-1 Each text token y i The feature representation is used to obtain the second set of feature representations H. n-1 Thus, the second set of features represents H. n-1 The feature representation h in ' i 'Text tokens can be considered y' n-1 Feature representation h n-1 .

[0047] In some embodiments, the electronic device 120 represents H from the first set of features. n-1 In the middle, obtain the first set of text tokens Y. n-1 The feature representation h of at least one first text token i To obtain at least one first feature representation, the electronic device 120 then regenerates the first set of text tokens Y based on each of the at least one first feature representation. n-1 The characteristic representation of the third text token in the sequence. As an example, in text token sequence 122, each of the first text tokens in at least one first text token is located after the third text token.

[0048] In some embodiments, the third text token may refer to the first set of text tokens Y. n-1 Except for the last text token y i Any text token other than y i As an example, electronic device 120 can regenerate the first set of text tokens Y. n-1 Each text token y in iThe feature representation. As an example, the second set of feature representations can be expressed by Equation 3:

[0049] H n-1 h' = (h1', h2', ..., h n-1 ′); Formula 3 Where h1′, h2′, ..., h n-1 ' can be text tokens y1, y2, ..., y n-1 The new feature representation generated by the corresponding one in the middle.

[0050] When generating the target text token (i.e., the nth text token), the text token y n-1 The target text token has already been generated. Therefore, when generating the target text token (i.e., the nth text token), the electronic device 120 can do so based on the feature representations h1, h2, ..., h n-1 For the first set of text tokens Y n-1 Each text token (e.g., y1, y2, ..., y) n-2 The characteristic representation of h) i Make corrections.

[0051] In some embodiments, each first text token is located after a third text token. In this way, for text token y... i Electronic device 120 can utilize features to represent h i+1 To feature representation h n-1 One or more features in the text token y are used to represent the text token y. i Feature representation h i Make corrections. This will allow the text token y to be modified. i Feature representation h i The correction can incorporate as much information as possible from its subsequent text tokens, thereby improving the new feature representation h. i The accuracy of ′.

[0052] In some embodiments, in the text token sequence 122, at least one first text token is adjacent to a third text token, and the last first text token in at least one first text token is the last text token in the currently generated text tokens in the text token sequence 122.

[0053] As an example, two adjacent text tokens can mean that there are no other text tokens between them. For example, suppose in the first set of text tokens Y... n-1 In the text, the third text token is text token y3, and its adjacent text tokens can be text token y2 and text token y4. Among these two text tokens, the one located at text token y... iThen comes the text token adjacent to it, namely text token y4. Furthermore, the last first text token has a special positional attribute. This last first text token is not only the end of this group of first text tokens, but it is also the last text token in the entire "currently generated text token sequence 122". This means that in the currently generated text token stream, no other text token is located after this particular first text token. In this context, at least the first first text token in a sequence of first text tokens can refer to text token y4, and the last first text token can refer to text token y n-1 In this way, the text token y i Feature representation h i The correction can cover its subsequent text tokens as broadly as possible.

[0054] In some embodiments, in the text token sequence 122, each of the at least one first text token can be a plurality of text tokens arranged consecutively or a plurality of text tokens arranged at intervals. Optionally, in embodiments of this disclosure, each of the at least one first text token is a plurality of text tokens arranged consecutively. In this manner, the text token y i Feature representation h i The correction can be based on a comprehensive consideration of information from all subsequent text tokens, which can further improve the new feature representation h. i The accuracy of the feature correction is ensured to guarantee its precision and effectiveness.

[0055] In some embodiments, the electronic device 120 obtains feature representations related to at least one second text token in the first set of text tokens from a first set of feature representations to obtain at least one second feature representation. Then, the electronic device 120 regenerates the feature representation of the third text token based on each of the first feature representations in the at least one set of first feature representations and each of the second feature representations in the at least one set of second feature representations. As an example, in the text token sequence 122, each of the at least one second text token precedes the third text token.

[0056] In some embodiments, each second text token precedes the third text token. In this way, for text token y... i In addition to using features to represent h i+1 To feature representation h n-1 One or more features in the text token y are used to represent the text token y. i Feature representation h i In addition to making corrections, the electronic device 120 can also utilize feature representations h1 to h2. i-1 One or more features in the text token y are used to represent the text token y.i Feature representation h i Corrections are made. Through this two-way correction method, electronic device 120 can more comprehensively incorporate text tokens. i Contextual information, thereby significantly improving the new feature representation h i The accuracy of '' makes the feature expression richer and more precise.

[0057] In some embodiments, in the text token sequence 122, the first second text token in at least one second text token is the first text token in the currently generated text tokens in the text token sequence 122, and the last second text token in at least one second text token is adjacent to the third text token.

[0058] As described above, two adjacent text tokens can mean that there are no other text tokens between them. For example, suppose in the first set of text tokens Y... n-1 In the text, the third text token is text token y3, and its adjacent text tokens can be text token y2 and text token y4. Among these two text tokens, the one located at text token y... i Then, the text token adjacent to it is text token y4. Furthermore, the first second text token has a special positional attribute. The first second text token is not only the beginning of this set of second text tokens, but it is also the first text token in the entire "currently generated text token sequence 122". This means that in the currently generated text token stream, no other text token precedes this particular second text token. In this context, the first second text token in at least one set can refer to text token y1, and the last second text token can refer to text token y4. i-1 In this way, the text token y i Feature representation h i The correction can cover the text tokens before and after it as broadly as possible.

[0059] In some embodiments, in the text token sequence 122, each of the at least one second text token can be a plurality of text tokens arranged consecutively or a plurality of text tokens arranged at intervals. Optionally, in embodiments of this disclosure, each of the at least one second text token is a plurality of text tokens arranged consecutively. In this manner, for text token y i Feature representation h i The correction can comprehensively consider all text tokens before and after it (that is, cover text tokens y1-y). i-1 and text token y i+1 -y n-1Information. By comprehensively considering this contextual information, the expression of new features can be maximized. i The accuracy of ′.

[0060] In some embodiments, the feature representations of each text token in the first set of text tokens are regenerated using a self-attention-based encoder in machine learning model 121.

[0061] As an example, a self-attention-based decoder can be constructed by stacking multiple hidden layers. In such a decoder, each hidden layer has a set of vector features, which can be calculated based on the set of vector features of the previous hidden layer through a self-attention mechanism. Furthermore, the set of vector features of each hidden layer can include vector features related to each text token currently generated. As an example, the electronic device 120 can use a first set of feature representations as input to the self-attention-based decoder, while a second set of feature representations is the output of the self-attention-based decoder.

[0062] As an example: the self-attention mechanism can be represented by Equation 4:

[0063]

[0064] Where Q, K, and V represent Query, Key, and Value, respectively, which are matrices obtained by linear transformation of a set of vector features from the previous hidden layer. k K is the vector dimension of matrix K. T This represents the transpose of matrix K. Indicates to Perform calculations based on the softmax function.

[0065] As an example, Q, K, and V can be calculated using Formula 5:

[0066]

[0067] Among them, a l-1 W is a set of vector features from the previous hidden layer of the l-th hidden layer. Q W K W V It is a parameter matrix.

[0068] As an example, Formula 4 can be further expanded, as shown in Formula 6:

[0069]

[0070] in, Represents text token y mIn the vector features of the (l-1)th hidden layer, m≤n, j represents an index used to traverse all key-value pairs, and τ is another index used to traverse all key vectors in the denominator of the softmax function to calculate the normalization factor.

[0071] In this way, the self-attention-based decoder comprehensively references the vector features of all text tokens in the previous hidden layer when processing vector features in each hidden layer to determine their values. This comprehensive reference mechanism ensures that each vector feature incorporates rich contextual information. More importantly, when the computation of vector features in each hidden layer considers both preceding and following context, these vector features are able to learn more complete semantic information.

[0072] In box 230, electronic device 120 generates target text tokens in text token sequence 122 based on a second conditional probability distribution indicated by a second set of feature representations.

[0073] As an example, electronic device 120 can be based on a set of feature vectors A from the last hidden layer (e.g., L layer) of the encoder. L To determine the second set of feature representations. As an example, electronic device 120 can determine the second conditional probability distribution P(y) of the target text token indicated by the second set of feature representations using Equation 7. n ).

[0074] P(y n ) = softmax(Wh n-1 );

[0075] Formula 7

[0076] Where W is the parameter matrix, h n-1 It can be based on a set of feature vectors A L Vector features in To determine.

[0077] In box 240, electronic device 120 determines target text sequence 130 based on the generated text token sequence 122. As an example, target text sequence 130 may be the target text sequence 130 output by machine learning model 121 for input content 110.

[0078] In some embodiments, text tokens serve as the basic building blocks of the text token sequence 122, corresponding to a text unit in the final output target text sequence 130. This unit can be a single character or a complete word, depending on the granularity of the model and the task requirements.

[0079] In some embodiments, the electronic device 120 may update the target text sequence 130 once each time a new text token is generated, or it may output the target text sequence 130 all at once after all text tokens have been generated. The specific method chosen to generate the target text sequence 130 can be determined according to actual needs, and the embodiments of this disclosure do not limit this.

[0080] During the training of the machine learning model 121, the electronic device 120 may employ various strategies as needed. In some embodiments, the electronic device 120 trains the machine learning model 121 based at least on a teacher-mandated mechanism.

[0081] Specifically, a teacher-forced mechanism is used as a training technique where, at each step of the sequence generation task, the machine learning model 121 does not use its previous output as the input for the next step, but instead uses the true, expected output (ground truth) as the input. This approach helps reduce error accumulation during training, enabling the model to learn the correct sequence generation patterns more quickly.

[0082] As can be clearly understood from the various embodiments described above, according to the embodiments of this disclosure, in the process of generating the target text token (e.g., the nth text token in text token sequence 122), by regenerating the feature representation of the first group of text tokens (e.g., the first n-1 text tokens in text token sequence 122, i.e., the prefix sequence of the target text token), the accumulated error present in the traditional autoregressive decoder is effectively reduced, and the correlation between text tokens within the prefix sequence is fully considered. This not only helps to improve the coherence and rationality of the final output target text sequence 130, but also significantly improves the quality of the target text sequence 130 and the user experience.

[0083] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 3 A schematic structural block diagram of an apparatus 300 for text sequence generation according to some embodiments of the present disclosure is shown. The apparatus 300 may be implemented as or included in an electronic device 120. Various modules / components in the apparatus 300 may be implemented by hardware, software, firmware, or any combination thereof.

[0084] Reference Figure 3The apparatus 300 includes: a feature representation acquisition module 310, a feature representation correction module 320, a text token generation module 330, and a target text sequence determination module 340. In some embodiments, the feature representation acquisition module 310 is configured to, during the generation of a target text token in the text token sequence, acquire the feature representation of each text token in the first group of text tokens in response to the completion of the generation of a first group of text tokens, to obtain a first set of feature representations, wherein the first set of feature representations is configured to indicate a first conditional probability distribution of the target text tokens. The feature representation correction module 320 is configured to regenerate the feature representation of each text token in the first group of text tokens based on the feature representations in the first set of feature representations, to obtain a second set of feature representations, wherein the second set of feature representations is configured to indicate a second conditional probability distribution of the target text tokens. The text token generation module 330 is configured to generate the target text token in the text token sequence based on the second conditional probability distribution indicated by the second set of feature representations. The target text sequence determination module 340 is configured to determine the target text sequence based on the generated text token sequence.

[0085] In some embodiments, the feature representation correction module 320 is further configured to: obtain feature representations related to at least one first text token in the first set of text tokens from the first set of feature representations to obtain at least one first feature representation; and regenerate feature representations of a third text token in the first set of text tokens based at least on each of the first feature representations in the at least one set of first feature representations; and wherein in the text token sequence, each of the first text tokens in the at least one set of first text tokens is located after the third text token.

[0086] In some embodiments, the first first text token in at least one first text token is adjacent to the third text token, and the last first text token in at least one first text token is the last text token in the currently generated text token sequence.

[0087] In some embodiments, the feature representation correction module 320 is further configured to: obtain feature representations related to at least one second text token in the first set of text tokens from the first set of feature representations to obtain at least one second feature representation; and regenerate the feature representation of the third text token based on each of the first feature representations in the at least one set of feature representations and each of the second feature representations in the at least one set of feature representations; and wherein in the text token sequence, each of the second text tokens in the at least one set of second text tokens is located before the third text token.

[0088] In some embodiments, in a text token sequence, the first second text token in at least one second text token is the first text token in the currently generated text token sequence, and the last second text token in at least one second text token is adjacent to a third text token.

[0089] In some embodiments, in the text token sequence, each of the first text tokens in at least one first text token is a plurality of text tokens arranged in succession, and / or, each of the second text tokens in at least one second text token is a plurality of text tokens arranged in succession.

[0090] In some embodiments, the feature representations of each text token in the first set of text tokens are regenerated using an encoder based on a self-attention mechanism in a machine learning model.

[0091] In some embodiments, the apparatus 300 further includes a training module. The training module is configured to train the machine learning model based at least on a teacher-mandated mechanism.

[0092] Figure 4 A block diagram is shown of an electronic device 400 in which one or more embodiments of the present disclosure may be implemented. The electronic device 400 may, for example, be used to implement... Figure 1 The electronic device 120 shown. It should be understood that, Figure 4 The electronic device 400 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0093] Reference Figure 4 Electronic device 400 is in the form of a general-purpose electronic device. Components of electronic device 400 may include, but are not limited to, one or more processors or processing units 410, memory 420, storage device 430, one or more communication units 440, one or more input devices 450, and one or more output devices 460. Processing unit 410 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 420. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 400.

[0094] Electronic device 400 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 400, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 420 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 430 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 400.

[0095] Electronic device 400 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 4 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 420 may include computer program product 425 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0096] Communication unit 440 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 400 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 400 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0097] Input device 450 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 460 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 400 can also communicate with one or more external devices (not shown) via communication unit 440 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 400, or with any device (e.g., network card, modem, etc.) that enables electronic device 400 to communicate with one or more other electronic devices 120. Such communication can be performed via input / output (I / O) interface (not shown).

[0098] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0099] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0100] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0101] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0103] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is determined to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for text sequence generation, comprising: in a generation process of a target text token in a sequence of text tokens, in response to completion of generation of a first group of text tokens, obtaining a feature representation of each text token in the first group of text tokens to obtain a first group of feature representations, wherein the first group of feature representations are configured to indicate a first conditional probability distribution of the target text token; based on each feature representation in the first group of feature representations, regenerating the feature representation of each text token in the first group of text tokens to obtain a second group of feature representations, the second group of feature representations being configured to indicate a second conditional probability distribution of the target text token; based on the second conditional probability distribution indicated by the second group of feature representations, generating the target text token in the sequence of text tokens; and based on the generated sequence of text tokens, determining a target text sequence.

2. The method of claim 1, wherein based on each feature representation in the first group of feature representations, regenerating the feature representation of each text token in the first group of text tokens to obtain a second group of feature representations, comprises: from the first group of feature representations, obtaining a feature representation related to at least one first text token in the first group of text tokens to obtain at least one first feature representation; and based on at least each first feature representation in the at least one first feature representation, regenerating a feature representation of a third text token in the first group of text tokens; and wherein in the sequence of text tokens, each first text token in the at least one first text token is located after the third text token.

3. The method of claim 2, wherein in the sequence of text tokens, a first first text token in the at least one first text token is adjacent to the third text token, and wherein a last first text token in the at least one first text token is a last text token in the currently generated text tokens in the sequence of text tokens.

4. The method of claim 2, wherein based on at least each first feature representation in the at least one first feature representation, regenerating a feature representation of a third text token in the first group of text tokens further comprises: from the first group of feature representations, obtaining a feature representation related to at least one second text token in the first group of text tokens to obtain at least one second feature representation; and based on each first feature representation in the at least one first feature representation and each second feature representation in the at least one second feature representation, regenerating the feature representation of the third text token; and wherein in the sequence of text tokens, each second text token in the at least one second text token is located before the third text token. ​ ​ ​ 5.The method of claim 4, wherein in the sequence of text tokens, a first one of the at least one second text token is a first one of the currently generated text tokens in the sequence of text tokens, and a last one of the at least one second text token is adjacent to the third text token. 6.The method of claim 4, wherein in the sequence of text tokens, each of the at least one first text token is a plurality of text tokens arranged consecutively, and / or each of the at least one second text token is a plurality of text tokens arranged consecutively. 7.The method of claim 1, wherein the feature representation of each of the first set of text tokens is regenerated using an encoder based on a self-attention mechanism in the machine learning model. 8.The method of claim 1, further comprising: training the machine learning model based at least on a teacher forcing mechanism. 9.An apparatus for text sequence generation, comprising: a feature representation obtaining module configured to, in a process of generating a target text token in a sequence of text tokens, in response to completion of generation of a first set of text tokens, obtain a feature representation of each of the first set of text tokens to obtain a first set of feature representations, wherein the first set of feature representations are configured to indicate a first conditional probability distribution of the target text token; a feature representation revising module configured to, based on each of the first set of feature representations, regenerate the feature representation of each of the first set of text tokens to obtain a second set of feature representations, wherein the second set of feature representations are configured to indicate a second conditional probability distribution of the target text token; a text token generating module configured to, based on the second conditional probability distribution indicated by the second set of feature representations, generate the target text token in the sequence of text tokens; and a text sequence determining module configured to determine a target text sequence based on the generated sequence of text tokens. 10.An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit cause the electronic device to perform the method according to any one of claims 1 to 8. 11.A computer readable storage medium having stored thereon a computer program, the computer program being executable by a processor to implement the method according to any one of claims 1 to 8. 12.A computer program product comprising computer executable instructions, wherein the computer executable instructions implement the method according to any one of claims 1 to 8 when executed by a processor. ​ ​

Images