Target text generation method and device based on large model, equipment and medium

Abstract

The invention relates to the technical field of large models, in particular to a target text generation method and device based on a large model, equipment and a medium, and the method comprises the following steps: obtaining a text sequence, and segmenting the text sequence to obtain sub-text sequences; for the first sub-text sequence at the first place, inputting the first sub-text sequence into a large model, and determining a first query key value pair corresponding to the first sub-text sequence in a Prefilling stage of the large model; for a non-first second sub-text sequence, inputting the second sub-text sequence and the first query key value pair into the large model, and determining a second query key value pair corresponding to the second sub-text sequence in a Prefilling stage of the large model; and in a Decoding stage of the large model, generating a target text based on the first query key value pair and each second query key value pair. The accurate target text is generated for the text sequence exceeding the maximum text length supported by the large model on the premise that high cost is not increased.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large model, and particularly relates to a target text generation method and device based on a large model, equipment and a medium. BACKGROUND

[0002] Based on the application of a large model, generally, the learning ability in the context of the large model (in-context learning) is relied on, a longer text is input, and the large model is required to give an answer, that is, a natural language text sequence is input to the large model, and the large model generates a target text based on the input text sequence and outputs the target text. However, the large model has certain limitations. Generally, the large model has a maximum text length supported. When the text length of the input text sequence is less than the maximum text length, the large model has the ability to generate an accurate target text. However, when the text length of the input text sequence exceeds the maximum text length, the large model cannot normally output the target text.

[0003] In the prior art, a sample text sequence longer than the maximum text length is generally used to further train the large model to expand the maximum text length supported by the large model. However, the complexity of further training the large model using a longer sample text sequence is very high, and the training cost is large. How to generate a target text for a text sequence longer than the maximum text length supported by the large model without increasing high costs is a problem to be solved at present. SUMMARY

[0004] The present application provides a target text generation method and device based on a large model, equipment and a medium to solve the problem of how to generate a target text for a text sequence longer than the maximum text length supported by the large model without increasing high costs in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a target text generation method based on a large model, the method comprising:

[0006] obtaining a text sequence, splitting the text sequence to obtain each subtext sequence; wherein the subtext length of each subtext sequence is less than the maximum text length supported by the large model;

[0007] for a first subtext sequence at the first position in the each subtext sequence, inputting the first subtext sequence into the large model to determine a first query key-value pair corresponding to the first subtext sequence in a pre-filling stage Prefilling of the large model; for a second subtext sequence at a position other than the first position in the each subtext sequence, inputting the second subtext sequence and the first query key-value pair into the large model to determine a second query key-value pair corresponding to the second subtext sequence in the pre-filling stage Prefilling of the large model;

[0008] In the decoding stage of the large model, the target text is generated based on the first query key-value pair and each of the second query key-value pairs.

[0009] Secondly, embodiments of this application provide a target text generation apparatus based on a large model, the apparatus comprising:

[0010] The text segmentation module is used to acquire a text sequence, segment the text sequence to obtain various sub-text sequences; wherein the length of each sub-text sequence is less than the maximum text length supported by the large model.

[0011] The determination module is used to input the first sub-text sequence, which is the first position of each sub-text sequence, into the large model, and determine the first query key-value pair corresponding to the first sub-text sequence during the prefilling stage of the large model; and to input the second sub-text sequence, which is not the first position of each sub-text sequence, and the first query key-value pair into the large model, and determine the second query key-value pair corresponding to the second sub-text sequence during the prefilling stage of the large model.

[0012] The text generation module is used in the decoding stage of the large model to generate target text based on the first query key-value pair and each of the second query key-value pairs.

[0013] Thirdly, embodiments of this application also provide an electronic device, which includes at least a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the target text generation method based on a large model as described above.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the target text generation method based on a large model as described above.

[0015] In this embodiment, after the electronic device acquires the text sequence, it first segments the text sequence to obtain various sub-text sequences; and the length of each sub-text sequence is less than the maximum text length supported by the large model. This ensures that the large model can process each sub-text sequence and output its corresponding query key-value pairs. Specifically, for the first sub-text sequence at the beginning of each sub-text sequence, the first sub-text sequence is directly input into the large model, and the first query key-value pair corresponding to the first sub-text sequence is determined during the prefilling stage of the large model. When processing each of the second sub-text sequences that are not at the beginning, the second sub-text sequence and the first query key-value pair are input into the large model, and the second query key-value pair corresponding to the second sub-text sequence is determined during the prefilling stage of the large model. Finally, during the decoding stage of the large model, the target text is generated based on the first query key-value pair and each of the second query key-value pairs. On the one hand, since the original text sequence is super-divided into various sub-text sequences, the first query key-value pair corresponding to the first sub-text sequence at the beginning is calculated directly and separately. On the other hand, the second query key-value pair corresponding to the second sub-text sequence (not at the beginning) is calculated by combining the first query key-value pairs corresponding to the second and first sub-text sequences. This improves the accuracy of determining the second query key-value pairs corresponding to the second sub-text sequence, and consequently improves the accuracy of generating target text based on the first and second query key-value pairs. Therefore, the target text generation method based on a large model provided in this application achieves accurate target text generation for text sequences exceeding the maximum text length supported by the large model without incurring high costs. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram illustrating the first target text generation process based on a large model provided in this application;

[0018] Figure 2 A schematic diagram of the second target text generation process based on a large model provided in this application;

[0019] Figure 3 A schematic diagram illustrating the first method of segmenting a text sequence provided in this application;

[0020] Figure 4 A schematic diagram illustrating the second method of segmenting a text sequence provided in this application;

[0021] Figure 5 A schematic diagram of the third target text generation process based on a large model provided in this application;

[0022] Figure 6 A schematic diagram of the candidate query key and the target query key provided in this application;

[0023] Figure 7 A schematic diagram illustrating the process of determining a group of key-value pairs for this application;

[0024] Figure 8 A schematic diagram illustrating the calculation of query key-value pairs for each subtext sequence provided in this application;

[0025] Figure 9 A schematic diagram of the target text generation device based on a large model provided in this application;

[0026] Figure 10 A schematic diagram of the electronic device structure provided in this application. Detailed Implementation

[0027] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0028] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0029] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0030] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0031] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0032] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0033] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

[0034] Figure 1 This application provides a schematic diagram of a first target text generation process based on a large model, which includes the following steps:

[0035] S101: Obtain a text sequence, segment the text sequence to obtain sub-text sequences; wherein the length of each sub-text sequence is less than the maximum text length supported by the large model;

[0036] S102: For the first sub-text sequence that is the first digit in each of the sub-text sequences, the first sub-text sequence is input into the large model, and the first query key-value pair corresponding to the first sub-text sequence is determined in the prefilling stage of the large model; For the second sub-text sequence that is not the first digit in each of the sub-text sequences, the second sub-text sequence and the first query key-value pair are input into the large model, and the second query key-value pair corresponding to the second sub-text sequence is determined in the prefilling stage of the large model.

[0037] S103: In the decoding stage of the large model, the target text is generated based on the first query key-value pair and each of the second query key-value pairs.

[0038] The target text generation method based on a large model provided in this application is applied to an electronic device, which can be a terminal device such as a PC or computer, or a server.

[0039] In this application, a user can input a text sequence into an electronic device. After acquiring the input text sequence, the electronic device segments the text sequence to obtain various sub-text sequences. The electronic device may include a display screen displaying a text sequence input window. The user can input a text sequence into the text sequence input window, and the electronic device can acquire the input text sequence and then segment it to obtain various sub-text sequences.

[0040] Users can also input voice information containing natural language into the electronic device. After collecting the voice information, the electronic device determines the corresponding text sequence based on a speech recognition algorithm. This text sequence is the text sequence acquired by the electronic device. Then, the text sequence is segmented to obtain individual sub-text sequences. The electronic device can be configured with a sound acquisition module, such as a microphone. Additionally, a speech recognition algorithm is deployed within the electronic device. Users can send voice information to the electronic device dictated. The sound acquisition module in the electronic device can collect the voice information, then use a speech recognition algorithm to recognize the information in the voice information, obtain the corresponding text sequence, and then segment the text sequence to obtain individual sub-text sequences.

[0041] When an electronic device segments a text sequence to obtain sub-text sequences, it considers the limitations of large models for text sequences exceeding their maximum supported text length. Specifically, when the length of the input text sequence exceeds the maximum, the large model may fail to output the target text correctly. This application ensures that the length of each sub-text sequence is less than the maximum supported text length of the large model during the segmentation process. The maximum supported text length refers to the maximum length of sample text sequences in the training set during model training. For example, if the maximum length of sample text sequences in the training set is 1024, then the maximum supported text length of the trained large model is also 1024. The electronic device first obtains the maximum supported text length of the large model, then configures a preset text length smaller than the maximum supported text length. It then sequentially segments the text sequence according to the preset text length. Segmentation continues as long as the length of the unsegmented portion is greater than the preset text length, until the length of the unsegmented portion is also less than the preset text length, thus obtaining the sub-text sequences.

[0042] The process of segmenting a text sequence follows the order of the text sequence, segmenting it sequentially from front to back. The first sub-text sequence obtained is designated as the first sub-text sequence. All sub-text sequences other than the first one are designated as second sub-text sequences (not the first). The second sub-text sequences that are not the first include the last one and the second sub-text sequences located between the first and last ones.

[0043] Large models consist of a prefilling phase and a decoding phase. The prefilling phase determines the query key-value pairs; the decoding phase generates the target text. In machine learning and deep learning, attention mechanisms are widely used in natural language processing, computer vision, and speech recognition tasks. Their goal is to simulate the focused attention and filtering processes humans use when processing information to improve the performance and results of large models.

[0044] In attention mechanisms, query, key, and value are three key elements. They help the model select and focus on relevant information according to the needs of the task. The query represents the problem or content of interest to the model; it acts like a label indicating which information the model should focus on. The role of the query is to help the model select and filter content relevant to the problem in order to better solve it.

[0045] The key is the part of the input information that is relevant to the query. It is usually a vector or matrix used to represent the features of the input information. The role of the key is to help the model select and focus on relevant information based on the content of the query. By calculating the similarity between the query and the key, the model can find the information most relevant to the query.

[0046] The value is the actual numerical value or vector corresponding to the key. It contains the specific numerical or vector representation of the input information. The role of the value is to provide the model with actual information so that the model can derive prediction results by calculating and processing the relationship between the key and the value.

[0047] In attention mechanisms, query, key, and value are typically matched and associated by calculating their similarity. This can be achieved in various ways, such as dot product attention, weighted averaging, etc. By calculating the similarity and relationship between query, key, and value, the model can select and focus on relevant information according to the needs of the task, in order to better solve the problem. In attention mechanisms, query, key, and value represent the problem the model focuses on, key information, and the actual numerical value or vector corresponding to the key information, respectively. By matching and associating these elements, the model can pay more attention to and filter relevant information when processing various tasks, thereby improving model performance and results.

[0048] In this application, for the first sub-text sequence at the beginning of each sub-text sequence, the first sub-text sequence is input into the large model. During the prefilling stage of the large model, the first query key-value pair corresponding to the first sub-text sequence is determined. It should be noted that each text token in the first sub-text sequence corresponds to a set of first query key-value pairs. For the second sub-text sequence that is not at the beginning of each sub-text sequence, the second sub-text sequence and the first query key-value pair are input into the large model. During the prefilling stage of the large model, the second query key-value pair corresponding to the second sub-text sequence is determined. It should be noted that each text token in the second sub-text sequence corresponds to a set of second query key-value pairs. Finally, during the decoding stage of the large model, the target text is generated based on the first query key-value pair corresponding to each text token in the first sub-text sequence and the second query key-value pair corresponding to each text token in each of the second sub-text sequences.

[0049] In this embodiment, after the electronic device acquires the text sequence, it first segments the text sequence to obtain various sub-text sequences; and the length of each sub-text sequence is less than the maximum text length supported by the large model. This ensures that the large model can process each sub-text sequence and output its corresponding query key-value pairs. Specifically, for the first sub-text sequence at the beginning of each sub-text sequence, the first sub-text sequence is directly input into the large model, and the first query key-value pair corresponding to the first sub-text sequence is determined during the prefilling stage of the large model. When processing each of the second sub-text sequences that are not at the beginning, the second sub-text sequence and the first query key-value pair are input into the large model, and the second query key-value pair corresponding to the second sub-text sequence is determined during the prefilling stage of the large model. Finally, during the decoding stage of the large model, the target text is generated based on the first query key-value pair and each of the second query key-value pairs. On the one hand, since the original text sequence is super-divided into various sub-text sequences, the first query key-value pair corresponding to the first sub-text sequence at the beginning is calculated directly and separately. On the other hand, the second query key-value pair corresponding to the second sub-text sequence (not at the beginning) is calculated by combining the first query key-value pairs corresponding to the second and first sub-text sequences. This improves the accuracy of determining the second query key-value pairs corresponding to the second sub-text sequence, and consequently improves the accuracy of generating target text based on the first and second query key-value pairs. Therefore, the target text generation method based on a large model provided in this application achieves accurate target text generation for text sequences exceeding the maximum text length supported by the large model without incurring high costs.

[0050] In this application, after obtaining the text sequence and before segmenting the text sequence, the method further includes:

[0051] Obtain the first text length of the text sequence and the maximum text length supported by the large model; if the first text length is greater than the maximum text length, perform the process of segmenting the text sequence;

[0052] If the length of the first text is not greater than the maximum text length, the method further includes:

[0053] The text sequence is input into a large model. During the prefilling stage of the large model, the third query key-value pair corresponding to the text sequence is determined. During the decoding stage of the large model, the target text is generated based on the third query key-value pair.

[0054] After acquiring a text sequence, the electronic device first obtains the initial text length and the maximum text length supported by the large model. If the initial text length is greater than the maximum text length, it indicates that the large model cannot directly process the text sequence to obtain accurate target text. Therefore, the text sequence is segmented at this point. If the initial text length is not greater than the maximum text length, it indicates that the large model has the ability to directly process the text sequence to obtain accurate target text. Therefore, the text sequence is directly input into the large model. In the prefilling stage of the large model, the third query key-value pairs corresponding to the text sequence are determined; in the decoding stage of the large model, the target text is generated based on the third query key-value pairs. It should be noted that in the prefilling stage of the large model, the third query key-value pairs corresponding to each text token in the text sequence are determined; in the decoding stage of the large model, the target text is generated based on the third query key-value pairs corresponding to each text token in the text sequence. This balances the accuracy and efficiency of generating the target text.

[0055] Figure 2 A second schematic diagram of a target text generation process based on a large model provided in this application embodiment, the process including the following steps:

[0056] S201: Obtain the first text length of the text sequence and the maximum text length supported by the large model;

[0057] S202: Determine whether the length of the first text is greater than the maximum text length. If yes, proceed to S203; otherwise, proceed to S205.

[0058] S203: Segment the text sequence to obtain sub-text sequences; wherein the length of each sub-text sequence is less than the maximum text length supported by the large model;

[0059] S204: For the first sub-text sequence at the beginning of each sub-text sequence, input the first sub-text sequence into the large model, and determine the first query key-value pair corresponding to the first sub-text sequence in the prefilling stage of the large model; for the second sub-text sequence that is not at the beginning of each sub-text sequence, input the second sub-text sequence and the first query key-value pair into the large model, and determine the second query key-value pair corresponding to the second sub-text sequence in the prefilling stage of the large model; in the decoding stage of the large model, generate the target text based on the first query key-value pair and each of the second query key-value pairs;

[0060] S205: Input the text sequence into the large model, determine the third query key-value pair corresponding to the text sequence in the prefilling stage of the large model; generate the target text based on the third query key-value pair in the decoding stage of the large model.

[0061] In this application, to make the segmentation of the text sequence into individual sub-text sequences more accurate, the text sequence is segmented to obtain individual sub-text sequences, including:

[0062] Obtain the first text length of the text sequence and the maximum text length supported by the large model;

[0063] Determine a first difference between the first text length and the first sub-text length of a preset first sub-text sequence, and a second difference between the maximum text length and the first sub-text length; determine a first ratio between the first difference and the second difference, and determine the second sub-text length of the middle position based on the first ratio;

[0064] The text sequence is segmented based on the length of the first sub-text and the length of the second sub-text in the middle position to obtain various sub-text sequences.

[0065] First, the electronic device obtains the first text length of the text sequence and the maximum text length supported by the large model. The first sub-text length of the first sub-text sequence is preset. It should be noted that the first sub-text length is less than the maximum text length supported by the large model. Then, a first difference is calculated between the first text length and the preset first sub-text length. Preferably, the first difference can be obtained by subtracting the first sub-text length from the first text length, and then subtracting a preset length parameter. The preset length parameter is, for example, 512. A second difference is calculated between the maximum text length and the first sub-text length. A first ratio between the first difference and the second difference is determined, and the second sub-text length of the middle position is determined based on the first ratio.

[0066] Optionally, a first ratio of the first difference and the second difference is determined, and this first ratio is rounded up to obtain a first intermediate parameter. The first difference is divided by the second difference, and the remainder is used as the second intermediate parameter. It is then determined whether the second intermediate parameter is less than a preset parameter threshold, such as 200 or 300. If the second intermediate parameter is less than the preset parameter threshold, the second difference between the maximum text length and the first sub-text length is used as the second sub-text length of the intermediate position. If the second intermediate parameter is not less than the preset parameter threshold, the first text length is subtracted from the first sub-text length, and then the preset length parameter is subtracted to obtain the first difference. The sum of the first intermediate parameter and 1 is calculated, and the ratio of the first difference to the sum is calculated. This ratio is rounded up to obtain the second sub-text length of the intermediate position.

[0067] Finally, the text sequence is segmented according to the length of the first sub-text and the length of the second sub-text at the middle position to obtain various sub-text sequences. The process of segmenting the text sequence is as follows: according to the order of the text sequence, it is segmented sequentially from front to back. First, the first sub-text sequence at the beginning is obtained based on the length of the first sub-text. Then, the second sub-text sequences at each middle position are segmented sequentially based on the length of the second sub-text at the middle position. The last second sub-text sequence whose length is not greater than the length of the second sub-text at the middle position is taken as the last second sub-text sequence.

[0068] Optionally, the length of the second subtext in the middle position can be determined using the following procedure:

[0069] Required: LLM parameters

[0070] Input: s (input token sequence)

[0071] Output: F, C

[0072] 1:set the values ​​of F and L respectively as constants

[0073] 2:set T according to the LLM parameters

[0074] 3: get I according to input sequence s

[0075] 4:

[0076] 5: M = (ILF) mod (TF)

[0077] 6:if M <M max then

[0078] 7:C = TF

[0079] 8:else

[0080] 9:

[0081] 10:end if

[0082] In the formula, I is the first text length, L is the preset length parameter, F is the preset first sub-text length, T is the maximum text length supported by the large model, N is the first intermediate parameter, M is the second intermediate parameter, and M max These are preset parameter thresholds.

[0083] Figure 3 The first schematic diagram of the process for segmenting a text sequence provided in this application includes the following steps:

[0084] S301: Obtain the first text length of the text sequence and the maximum text length supported by the large model;

[0085] S302: Determine a first difference between the first text length and the first sub-text length of a preset first sub-text sequence, and a second difference between the maximum text length and the first sub-text length; determine a first ratio between the first difference and the second difference, and determine the second sub-text length of the middle position based on the first ratio;

[0086] S303: The text sequence is segmented according to the length of the first sub-text and the length of the second sub-text in the middle position to obtain various sub-text sequences.

[0087] In this application, the text sequence is segmented according to the length of the first sub-text and the length of the second sub-text of the middle position to obtain various sub-text sequences, including:

[0088] The text sequence is segmented according to the length of the first sub-text to obtain the first sub-text sequence at the beginning;

[0089] Determine the second text length of the second text sequence other than the first sub-text sequence, and the second ratio of the second sub-text length of the middle position; determine the number of the second sub-text sequences of the middle position according to the second ratio; and sequentially segment the second text sequence according to the number to obtain the number of second sub-text sequences of the middle position.

[0090] The sub-text sequence other than the first sub-text sequence and the second sub-text sequences of each middle position in the text sequence is taken as the last second sub-text sequence.

[0091] When segmenting a text sequence based on the length of the first sub-text and the length of the second sub-text at the middle position, the text sequence is first segmented according to the length of the first sub-text to obtain the first sub-text sequence at the beginning. Then, the second text length of the second text sequence excluding the first sub-text sequence is obtained. The second ratio of the second text length to the length of the second sub-text at the middle position is then calculated and rounded down. The resulting value is the number of second sub-text sequences at the middle position. The second text length is then sequentially segmented into this number of second sub-text sequences at the middle position. The sub-text sequences excluding the first sub-text sequence and the second sub-text sequences at each middle position are considered as the last second sub-text sequence.

[0092] Figure 4 The second method for segmenting a text sequence provided in this application includes the following steps:

[0093] S401: Obtain the first text length of the text sequence and the maximum text length supported by the large model;

[0094] S402: Determine a first difference between the first text length and the first sub-text length of a preset first sub-text sequence, and a second difference between the maximum text length and the first sub-text length; determine a first ratio between the first difference and the second difference, and determine the second sub-text length of the middle position based on the first ratio;

[0095] S403: Divide the text sequence according to the length of the first sub-text to obtain the first sub-text sequence of the first digit;

[0096] S404: Determine the second text length of the second text sequence other than the first sub-text sequence, and the second ratio of the second sub-text length of the middle position; determine the number of second sub-text sequences of the middle position according to the second ratio; and sequentially segment the second text sequence according to the number to obtain the number of second sub-text sequences of the middle position.

[0097] S405: Take the sub-text sequence other than the first sub-text sequence and the second sub-text sequences of each middle position in the text sequence as the last second sub-text sequence.

[0098] In this application, the second sub-text sequence and the first query key-value pair are input into the large model. The prefilling stage of the large model determines the second query key-value pair corresponding to the second sub-text sequence, including:

[0099] If the second sub-text sequence is the second sub-text sequence in the middle position, the second sub-text sequence and the first query key-value pair are input into the large model, and the second query key-value pair corresponding to the second sub-text sequence is determined in the prefilling stage of the large model;

[0100] If the second sub-text sequence is the last sub-text sequence, the first query key value corresponding to the first sub-text sequence at the beginning and the second query key value pairs corresponding to the second sub-text sequences in each middle position are concatenated to obtain a query key value pair group. The second sub-text sequence and the query key value pair group are input into the large model. In the prefilling stage of the large model, the second query key value pair corresponding to the second sub-text sequence is determined.

[0101] In this application, different processing strategies are adopted for the second sub-text sequence in the middle and the second sub-text sequence at the end. For the second sub-text sequence in the middle, the second query key-value pair corresponding to the second sub-text sequence in the middle is determined based on the first query key-value pair corresponding to the first sub-text sequence at the beginning and the second sub-text sequence in the middle. That is, the first query key-value pair corresponding to the first sub-text sequence at the beginning and the second sub-text sequence in the middle are input into the large model, and the second query key-value pair corresponding to the second sub-text sequence is determined during the prefilling stage of the large model.

[0102] For the last second sub-text sequence, the second query key-value pair corresponding to the last second sub-text sequence is determined based on the first query key-value pair corresponding to the first sub-text sequence at the beginning, the second query key-value pairs corresponding to the second sub-text sequences in each middle position, and the last second sub-text sequence. Specifically, the first query key-value pair corresponding to the first sub-text sequence at the beginning and the second query key-value pairs corresponding to the second sub-text sequences in each middle position are first concatenated to obtain a query key-value pair group. Then, the last second sub-text sequence and the query key-value pair group are input into the large model. During the prefilling stage of the large model, the second query key-value pair corresponding to the last second sub-text sequence is determined.

[0103] In this application, since the text sequence is segmented into various sub-text sequences, to improve the accuracy of the generated target text, for the second sub-text sequence that is not the first position, the corresponding second query key-value pair is determined based on the second sub-text sequence and the first query key-value pair. Specifically, for the second sub-text sequence in the middle position, the corresponding second query key-value pair is determined based on the second sub-text sequence and the first query key-value pair. For the second sub-text sequence at the end position, the corresponding second query key-value pair is determined based on the second sub-text sequence, the preceding first query key-value pair, and each second query key-value pair. This ensures the efficiency and accuracy of determining the second query key-value pair, thereby ensuring the accuracy and efficiency of the generated target text.

[0104] Figure 5 A schematic diagram of a third target text generation process based on a large model provided in this application embodiment, the process including the following steps:

[0105] S501: Obtain a text sequence, segment the text sequence to obtain sub-text sequences; wherein the length of each sub-text sequence is less than the maximum text length supported by the large model;

[0106] S502: For the first sub-text sequence at the beginning of each sub-text sequence, input the first sub-text sequence into the large model, and determine the first query key-value pair corresponding to the first sub-text sequence in the prefilling stage of the large model; For the second sub-text sequence that is not at the beginning of each sub-text sequence, if the second sub-text sequence is the second sub-text sequence in the middle position, input the second sub-text sequence and the first query key-value pair into the large model, and determine the second query key-value pair corresponding to the second sub-text sequence in the prefilling stage of the large model;

[0107] S503: If the second sub-text sequence is the last second sub-text sequence, concatenate the first query key value corresponding to the first sub-text sequence at the beginning and the second query key value pairs corresponding to the second sub-text sequences at each middle position to obtain a query key value pair group. Input the second sub-text sequence and the query key value pair group into the large model. In the prefilling stage of the large model, determine the second query key value pair corresponding to the second sub-text sequence.

[0108] S504: In the decoding stage of the large model, the target text is generated based on the first query key-value pair and each of the second query key-value pairs.

[0109] In this application, the first query key value corresponding to the first sub-text sequence at the first position and the second query key value pairs corresponding to the second sub-text sequences at each middle position are concatenated to obtain a query key value pair group, including:

[0110] The first query key value corresponding to the first sub-text sequence of the first position is concatenated with the second query key value pair corresponding to the second sub-text sequence of each middle position to obtain a candidate query key value pair group.

[0111] Obtain the attention content marker Query of the candidate query key-value pair group; for each candidate query key Key in the candidate query key-value pair group, determine the relative position difference between the attention content marker Query and the candidate query key Key; determine the third difference between the relative position difference and a preset update query key; determine the third ratio between the third difference and a preset number of query key repetitions; and determine the target query key Key corresponding to the candidate query key Key based on the sum of the third ratio and the preset update query key.

[0112] For each candidate query key in the candidate query key-value pair group, the candidate query key is updated using the corresponding target query key to obtain the query key-value pair group.

[0113] In this application, the first query key value corresponding to the first sub-text sequence at the beginning and the second query key value pairs corresponding to the second sub-text sequences at each middle position are concatenated according to the order of the text sequence to obtain a candidate query key value pair group. Then, the candidate query key Key in the candidate query key value pair group is updated to obtain the query key value pair group.

[0114] The process of updating the candidate query key Key in the candidate query key-value pair group is as follows:

[0115] First, obtain the content tag "Query" for the candidate query key-value pair group. For each candidate query key "Key" in the candidate query key-value pair group, determine the relative position difference between the content tag "Query" and the candidate query key "Key". That is, calculate the value of "ti", where "t" represents the content tag "Query" and "i" represents the key sequence number corresponding to the candidate query key "Key". Then, determine the third difference between the relative position difference and a preset update query key. The preset update query key is, for example, 3, 4, etc. Determine the third ratio between the third difference and the preset number of repetitions of the query key. Round the third ratio down, and then sum the rounded value with the preset update query key as the target query key "Key" corresponding to the candidate query key "Key". The preset number of repetitions of the query key is, for example, 2, 3, etc. For each candidate query key "Key" in the candidate query key-value pair group, update the candidate query key "Key" using the corresponding target query key "Key" to obtain the query key-value pair group.

[0116] Specifically, the formula is used. Determine the target query key. I represents the target query key value, N represents the preset update query key, t represents the content to be monitored (Query), i represents the key index corresponding to the candidate query key, and E represents the preset number of duplicate query keys.

[0117] Figure 6 A schematic diagram of the candidate query key and target query key provided in this application is shown below. Figure 6 As shown, the candidate query key is obtained based on traditional PE encoding, while the target query key is obtained based on the Stair PE encoding proposed in this application. Stair PE encoding: when the relative position encoding exceeds w, the relative position is readjusted, and a step-like extrapolation is performed with a fixed window length. Figure 6 As can be seen, after a certain number of positions, the window length (the number of repetitions of the relative positional encoding) increases from w to w+1; it can be understood that after a certain number of positions, it can increase to w+2; the increase is 1.

[0118] like Figure 6As shown, in the candidate query keys, the content of interest is marked 'q9', and the key numbers corresponding to the candidate query keys from left to right are k0 to k9. The candidate query keys from left to right are 9 to 0. In the target query key, the content of interest is marked 'q9', and the key numbers corresponding to the candidate query keys from left to right are k0 to k9. The target query key from left to right is 6655443210. Figure 6 The target query key shown is based on a formula. The results are determined, and E = 2, N = 4. Taking sequence number k2 as an example, the corresponding candidate query key Key is 7. This application calculates...

[0119] Figure 7 The process diagram for determining the query key-value pair group provided in this application includes the following steps:

[0120] S701: Concatenate the first query key value corresponding to the first sub-text sequence of the first position and the second query key value pair corresponding to the second sub-text sequence of each middle position to obtain a candidate query key value pair group;

[0121] S702: Obtain the attention content marker Query of the candidate query key-value pair group; for each candidate query key Key in the candidate query key-value pair group, determine the relative position difference between the attention content marker Query and the candidate query key Key; determine the third difference between the relative position difference and a preset update query key; determine the third ratio between the third difference and a preset number of query key repetitions; and determine the target query key Key corresponding to the candidate query key Key based on the sum of the third ratio and the preset update query key.

[0122] S703: For each candidate query key Key in the candidate query key-value pair group, update the candidate query key Key using the corresponding target query key Key to obtain the query key-value pair group.

[0123] In this application, during the decoding stage of the large model, generating the target text based on the first query key-value pair and each of the second query key-value pairs includes:

[0124] During the decoding stage of the large model, the first query key-value pair and each of the second query key-value pairs are saved to the query key-value pair cache (KV cache); a first attention representation is determined based on the query key-value pair cache (KV cache); and the first text content token of the target text is predicted based on the first attention representation.

[0125] Based on the first text content token, the query key-value pair cache (KV cache) is updated, and a second attention representation is determined based on the updated query key-value pair cache (KV cache). The second text content token of the target text is predicted based on the second attention representation. The prediction of the text content token of the target text is completed.

[0126] During the decoding phase of the large model, the target text is generated one text content token at a time, based on the first query key-value pair and each of the second query key-value pairs. When generating the first text content token of the target text, the query key-value pair cache (KV cache) is initialized to empty. Then, the first query key-value pair and each of the second query key-value pairs are saved to the query key-value pair cache (KV cache). Based on the query key-value pair cache (KV cache), the first attention representation b1 is determined, and the first text content token of the target text is predicted based on the first attention representation b1. When generating the second text content token of the target text, the query key-value pair cache (KV cache) is updated based on the first text content token. Based on the updated query key-value pair cache (KV cache), the second attention representation b2 is determined, and the second text content token of the target text is predicted based on the second attention representation b2. When generating the third text content token of the target text, the query key-value pair cache (KV cache) is updated based on the second text content token. Based on the updated query key-value pair cache (KV cache), the third attention representation b3 is determined, and the third text content token of the target text is predicted based on the third attention representation b3. This process continues until the text content token prediction for the target text is complete.

[0127] This application addresses the limitation of large models in processing long texts, a problem known as the extrapolation problem. The length of the input text for a large model is finite; exceeding the maximum training text length will cause the model to fail to output correctly. However, due to the quadratic computational complexity, using longer texts to expand the input length window of a large model during training typically incurs high training costs. Therefore, this application provides a method to freely expand the input window of existing large models without increasing the length of the training text. Existing methods for increasing the input window of large models include PI techniques, FoT techniques, and Landmark techniques. These methods typically use long texts to further train the large model. However, further training of the large model usually increases training costs and leads to catastrophic forgetting, causing the large model to lose its previously acquired capabilities. The purpose of this application is to achieve free extrapolation through a free, plug-in approach, without requiring further training of the large model.

[0128] Figure 8 A schematic diagram illustrating the calculation of query key-value pairs for each subtext sequence provided in this application. Figure 8 The x-axis represents the key vectors, and the y-axis represents the query vectors. For example... Figure 8 As shown, the text sequence is obtained and segmented into individual sub-text sequences (chunks). When calculating query key-value pairs, the first chunk is calculated separately, the middle chunks are concatenated with the query key-value pairs of the first chunk, and the last chunk is concatenated with the query key-value pairs of all preceding chunks. This method effectively reduces memory usage and inference time.

[0129] In the specific implementation process, for large models using RoPE positional encoding, this application designs and implements a splitting chunk-based approach. By dynamically splitting chunks, memory usage is reduced and inference speed is improved. The specific algorithm is as follows:

[0130] Algorithm 1Mesa-Extrapolation Algorithm

[0131] Require:DynamicSplit,LLM,StairPE

[0132] Input:s[0:T-1](input tokens with length T)

[0133] Output: s[T,T+1,...]

[0134] #Prefll Stage

[0135] 1:first_length,chunk_width←DynamicSplit(s)

[0136] 2:K_cache,V_cache←□,□

[0137] 3:first_K,first_V←LLM(s[0:first_length])

[0138] 4.Append first_K to K_cache,first_V to V_cache

[0139] 5:i←first_length

[0140] 6:while i<T-1-chunk_width do

[0141] 7:K,V←LLM(s[i:i+chunk_width],first_K,first_V)

[0142] 8:K_cache append K,V_cache append V

[0143] 9:i←i+chunk_width

[0144] 10:end while

[0145] 11:apply StairPE to fix positions

[0146] 12:K,V←LLM(s[i:T-1],K_cache,V_cache)

[0147] #Decoding Stage

[0148] 13:apply StairPE to fix positions

[0149] 14:generate next-token one by one

[0150] In the above process, the method name is: mesa-extrapolation; LLM: large model; DynamicSplit: dynamic splitting function, used to split the sequence into several sub-text sequences according to the sequence length; Stair PE: relative position encoding scheme.

[0151] The process mainly consists of four steps: the first three steps correspond to the prefilling stage, used to compute all input tokens. The last step corresponds to the decoding stage, used to generate the next token one by one. First, the DynamicSplit function splits the input text sequence, and the first sub-text sequence after splitting is input into the LLM to generate the first attention matrix block (line 3). Second, subsequent sequences are processed iteratively, while the key-value pairs of the first block are input into the LLM to generate subsequent blocks (lines 6-10). Third, when computing the last sub-sequence, the key-value pairs of all the preceding chunks are concatenated together, and their relative position encoding is modified using Stair PE, and then input into the LLM to generate the last block (lines 11-12). Finally, Stair PE is applied to process the current token and the cached key-value pairs to generate the next token one by one (lines 13-14).

[0152] Compared to A-shaped attention matrices such as Streaming-LLM and LM-Infinite, this application significantly reduces information loss and achieves higher accuracy by adding bottom attention. Simultaneously, it achieves lower memory consumption and faster inference speed. It enables efficient extrapolation of existing large models without incurring any additional cost, while further reducing memory consumption and improving inference speed.

[0153] Figure 9 The schematic diagram of the target text generation device based on a large model provided in this application includes:

[0154] The text segmentation module 91 is used to acquire a text sequence, segment the text sequence to obtain various sub-text sequences; wherein the length of each sub-text sequence is less than the maximum text length supported by the large model.

[0155] The determination module 92 is used to input the first sub-text sequence, which is the first sub-text sequence at the beginning of each sub-text sequence, into the large model, and determine the first query key-value pair corresponding to the first sub-text sequence during the prefilling stage of the large model; and to input the second sub-text sequence and the first query key-value pair into the large model, which is the second sub-text sequence at the beginning of each sub-text sequence, and determine the second query key-value pair corresponding to the second sub-text sequence during the prefilling stage of the large model.

[0156] The text generation module 93 is used in the decoding stage of the large model to generate target text based on the first query key-value pair and each of the second query key-value pairs.

[0157] The text segmentation module 91 is also used to obtain the first text length of the text sequence and the maximum text length supported by the large model; if the first text length is greater than the maximum text length, the text sequence is segmented.

[0158] The determination module 92 is further configured to, if the length of the first text is not greater than the maximum text length, input the text sequence into the large model, and determine the third query key-value pair corresponding to the text sequence in the prefilling stage of the large model;

[0159] The text generation module 93 is also used in the decoding stage of the large model to generate target text based on the third query key-value pair.

[0160] The text segmentation module 91 is specifically used to obtain the first text length of the text sequence and the maximum text length supported by the large model; determine the first difference between the first text length and the first sub-text length of the preset first sub-text sequence, and the second difference between the maximum text length and the first sub-text length; determine the first ratio between the first difference and the second difference, and determine the second sub-text length of the middle position according to the first ratio; and segment the text sequence according to the first sub-text length and the second sub-text length of the middle position to obtain each sub-text sequence.

[0161] The text segmentation module 91 is specifically used to segment the text sequence according to the length of the first sub-text to obtain the first sub-text sequence at the beginning; determine the second text length of the second text sequence other than the first sub-text sequence and the second ratio of the length of the second sub-text at the middle position; determine the number of the second sub-text sequences at the middle position according to the second ratio; segment the second text sequence sequentially according to the number to obtain the number of second sub-text sequences at the middle position; and take the sub-text sequences in the text sequence other than the first sub-text sequence and the second sub-text sequences at each middle position as the last sub-text sequence.

[0162] The determination module 92 is specifically used to: if the second sub-text sequence is a middle-position second sub-text sequence, input the second sub-text sequence and the first query key-value pair into the large model, and determine the second query key-value pair corresponding to the second sub-text sequence in the prefilling stage of the large model; if the second sub-text sequence is a last-position second sub-text sequence, concatenate the first query key value corresponding to the first sub-text sequence at the first position and the second query key-value pair corresponding to each middle-position second sub-text sequence to obtain a query key-value pair group, input the second sub-text sequence and the query key-value pair group into the large model, and determine the second query key-value pair corresponding to the second sub-text sequence in the prefilling stage of the large model.

[0163] The determining module 92 is specifically used to concatenate the first query key value corresponding to the first sub-text sequence of the first position and the second query key value pair corresponding to the second sub-text sequence of each middle position to obtain a candidate query key value pair group; obtain the attention content marker Query of the candidate query key value pair group; for each candidate query key Key in the candidate query key value pair group, determine the relative position difference between the attention content marker Query and the candidate query key Key; determine the third difference between the relative position difference and a preset update query key; determine the third ratio between the third difference and a preset number of query key repetitions; determine the target query key Key corresponding to the candidate query key Key based on the sum of the third ratio and the preset update query key; and update the candidate query key Key with the corresponding target query key Key for each candidate query key Key in the candidate query key value pair group to obtain the query key value pair group.

[0164] The text generation module 93 is specifically used in the decoding stage of the large model to save the first query key-value pair and each of the second query key-value pairs to a query key-value pair cache (KV cache); determine a first attention representation based on the query key-value pair cache (KV cache); predict the first text content token of the target text based on the first attention representation; update the query key-value pair cache (KV cache) based on the first text content token; determine a second attention representation based on the updated query key-value pair cache (KV cache); predict the second text content token of the target text based on the second attention representation; and so on until the prediction of the text content token of the target text is completed.

[0165] Based on the above embodiments, Figure 10 This application provides a schematic diagram of an electronic device structure based on the above embodiments. The present application further provides an electronic device, such as... Figure 10 As shown, it includes: processor 801, communication interface 802, memory 803 and communication bus 804, wherein processor 801, communication interface 802 and memory 803 communicate with each other through communication bus 804.

[0166] The memory 803 stores a computer program, which, when executed by the processor 801, causes the processor 801 to perform the following steps:

[0167] Obtain a text sequence, segment the text sequence to obtain individual sub-text sequences; wherein the length of each sub-text sequence is less than the maximum text length supported by the large model;

[0168] For the first sub-text sequence at the beginning of each sub-text sequence, the first sub-text sequence is input into the large model, and the first query key-value pair corresponding to the first sub-text sequence is determined in the prefilling stage of the large model; for the second sub-text sequence that is not at the beginning of each sub-text sequence, the second sub-text sequence and the first query key-value pair are input into the large model, and the second query key-value pair corresponding to the second sub-text sequence is determined in the prefilling stage of the large model.

[0169] In the decoding stage of the large model, the target text is generated based on the first query key-value pair and each of the second query key-value pairs.

[0170] Furthermore, after obtaining the text sequence but before segmenting the text sequence, the method further includes:

[0171] Obtain the first text length of the text sequence and the maximum text length supported by the large model; if the first text length is greater than the maximum text length, perform the process of segmenting the text sequence;

[0172] If the length of the first text is not greater than the maximum text length, the method further includes:

[0173] The text sequence is input into a large model. During the prefilling stage of the large model, the third query key-value pair corresponding to the text sequence is determined. During the decoding stage of the large model, the target text is generated based on the third query key-value pair.

[0174] Furthermore, the text sequence is segmented to obtain sub-text sequences, including:

[0175] Obtain the first text length of the text sequence and the maximum text length supported by the large model;

[0176] Determine a first difference between the first text length and the first sub-text length of a preset first sub-text sequence, and a second difference between the maximum text length and the first sub-text length; determine a first ratio between the first difference and the second difference, and determine the second sub-text length of the middle position based on the first ratio;

[0177] The text sequence is segmented based on the length of the first sub-text and the length of the second sub-text in the middle position to obtain various sub-text sequences.

[0178] Further, the text sequence is segmented according to the length of the first sub-text and the length of the second sub-text at the middle position, resulting in various sub-text sequences including:

[0179] The text sequence is segmented according to the length of the first sub-text to obtain the first sub-text sequence at the beginning;

[0180] Determine the second text length of the second text sequence other than the first sub-text sequence, and the second ratio of the second sub-text length of the middle position; determine the number of the second sub-text sequences of the middle position according to the second ratio; and sequentially segment the second text sequence according to the number to obtain the number of second sub-text sequences of the middle position.

[0181] The sub-text sequence other than the first sub-text sequence and the second sub-text sequences of each middle position in the text sequence is taken as the last second sub-text sequence.

[0182] Further, the second sub-text sequence and the first query key-value pair are input into the large model. Determining the second query key-value pair corresponding to the second sub-text sequence during the prefilling stage of the large model includes:

[0183] If the second sub-text sequence is the second sub-text sequence in the middle position, the second sub-text sequence and the first query key-value pair are input into the large model, and the second query key-value pair corresponding to the second sub-text sequence is determined in the prefilling stage of the large model;

[0184] If the second sub-text sequence is the last sub-text sequence, the first query key value corresponding to the first sub-text sequence at the beginning and the second query key value pairs corresponding to the second sub-text sequences in each middle position are concatenated to obtain a query key value pair group. The second sub-text sequence and the query key value pair group are input into the large model. In the prefilling stage of the large model, the second query key value pair corresponding to the second sub-text sequence is determined.

[0185] Further, the first query key value corresponding to the first sub-text sequence at the first position and the second query key value pairs corresponding to the second sub-text sequences at each middle position are concatenated to obtain a query key value pair group including:

[0186] The first query key value corresponding to the first sub-text sequence of the first position is concatenated with the second query key value pair corresponding to the second sub-text sequence of each middle position to obtain a candidate query key value pair group.

[0187] Obtain the attention content marker Query of the candidate query key-value pair group; for each candidate query key Key in the candidate query key-value pair group, determine the relative position difference between the attention content marker Query and the candidate query key Key; determine the third difference between the relative position difference and a preset update query key; determine the third ratio between the third difference and a preset number of query key repetitions; and determine the target query key Key corresponding to the candidate query key Key based on the sum of the third ratio and the preset update query key.

[0188] For each candidate query key in the candidate query key-value pair group, the candidate query key is updated using the corresponding target query key to obtain the query key-value pair group.

[0189] Furthermore, in the decoding stage of the large model, generating the target text based on the first query key-value pair and each of the second query key-value pairs includes:

[0190] During the decoding stage of the large model, the first query key-value pair and each of the second query key-value pairs are saved to the query key-value pair cache (KV cache); a first attention representation is determined based on the query key-value pair cache (KV cache); and the first text content token of the target text is predicted based on the first attention representation.

[0191] Based on the first text content token, the query key-value pair cache (KV cache) is updated, and a second attention representation is determined based on the updated query key-value pair cache (KV cache). The second text content token of the target text is predicted based on the second attention representation. The prediction of the text content token of the target text is completed.

[0192] Since the principle behind the problem-solving by the aforementioned electronic device is similar to that of the target text generation method based on a large model, the implementation of the aforementioned electronic device can be found in the embodiments of the method, and repeated details will not be elaborated here.

[0193] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0194] Communication interface 802 is used for communication between the above-mentioned electronic device and other devices.

[0195] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0196] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0197] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program executable by a processor. When the program is run on the processor, the processor executes the following steps:

[0198] Obtain a text sequence, segment the text sequence to obtain individual sub-text sequences; wherein the length of each sub-text sequence is less than the maximum text length supported by the large model;

[0199] For the first sub-text sequence at the beginning of each sub-text sequence, the first sub-text sequence is input into the large model, and the first query key-value pair corresponding to the first sub-text sequence is determined in the prefilling stage of the large model; for the second sub-text sequence that is not at the beginning of each sub-text sequence, the second sub-text sequence and the first query key-value pair are input into the large model, and the second query key-value pair corresponding to the second sub-text sequence is determined in the prefilling stage of the large model.

[0200] In the decoding stage of the large model, the target text is generated based on the first query key-value pair and each of the second query key-value pairs.

[0201] Since the principle of the computer-readable storage medium in solving the problem is similar to that of the target text generation method based on a large model, the implementation of the computer-readable storage medium can be found in the embodiments of the method, and repeated details will not be repeated.

[0202] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0203] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxesFigure 1 A device that provides the functions specified in one or more boxes.

[0204] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0205] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0206] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A target text generation method based on a large model, characterized in that, The method includes: Obtain a text sequence, segment the text sequence to obtain individual sub-text sequences; wherein the length of each sub-text sequence is less than the maximum text length supported by the large model; For the first sub-text sequence at the beginning of each sub-text sequence, the first sub-text sequence is input into the large model, and the first query key-value pair corresponding to the first sub-text sequence is determined in the prefilling stage of the large model; for the second sub-text sequence that is not at the beginning of each sub-text sequence, the second sub-text sequence and the first query key-value pair are input into the large model, and the second query key-value pair corresponding to the second sub-text sequence is determined in the prefilling stage of the large model. In the decoding stage of the large model, the target text is generated based on the first query key-value pair and each of the second query key-value pairs.

2. The method according to claim 1, characterized in that, After obtaining the text sequence but before segmenting the text sequence, the method further includes: Obtain the first text length of the text sequence and the maximum text length supported by the large model; if the first text length is greater than the maximum text length, perform the process of segmenting the text sequence; If the length of the first text is not greater than the maximum text length, the method further includes: The text sequence is input into a large model. During the prefilling stage of the large model, the third query key-value pair corresponding to the text sequence is determined. During the decoding stage of the large model, the target text is generated based on the third query key-value pair.

3. The method according to claim 1, characterized in that, The text sequence is segmented to obtain sub-text sequences, including: Obtain the first text length of the text sequence and the maximum text length supported by the large model; Determine a first difference between the first text length and the first sub-text length of a preset first sub-text sequence, and a second difference between the maximum text length and the first sub-text length; determine a first ratio between the first difference and the second difference, and determine the second sub-text length of the middle position based on the first ratio; The text sequence is segmented based on the length of the first sub-text and the length of the second sub-text in the middle position to obtain various sub-text sequences.

4. The method according to claim 3, characterized in that, The text sequence is segmented based on the length of the first sub-text and the length of the second sub-text at the middle position, resulting in the following sub-text sequences: The text sequence is segmented according to the length of the first sub-text to obtain the first sub-text sequence at the beginning; Determine the second text length of the second text sequence other than the first sub-text sequence, and the second ratio of the second sub-text length of the middle position; determine the number of the second sub-text sequences of the middle position according to the second ratio; and sequentially segment the second text sequence according to the number to obtain the number of second sub-text sequences of the middle position. The sub-text sequence other than the first sub-text sequence and the second sub-text sequences of each middle position in the text sequence is taken as the last second sub-text sequence.

5. The method according to claim 1, characterized in that, The second sub-text sequence and the first query key-value pair are input into the large model. The prefilling stage of the large model determines the second query key-value pair corresponding to the second sub-text sequence, including: If the second sub-text sequence is the second sub-text sequence in the middle position, the second sub-text sequence and the first query key-value pair are input into the large model, and the second query key-value pair corresponding to the second sub-text sequence is determined in the prefilling stage of the large model; If the second sub-text sequence is the last sub-text sequence, the first query key value corresponding to the first sub-text sequence at the beginning and the second query key value pairs corresponding to the second sub-text sequences in each middle position are concatenated to obtain a query key value pair group. The second sub-text sequence and the query key value pair group are input into the large model. In the prefilling stage of the large model, the second query key value pair corresponding to the second sub-text sequence is determined.

6. The method according to claim 5, characterized in that, The query key value corresponding to the first sub-text sequence at the first position is concatenated with the second query key value pairs corresponding to the second sub-text sequences at each middle position to obtain a query key value pair group including: The first query key value corresponding to the first sub-text sequence of the first position is concatenated with the second query key value pair corresponding to the second sub-text sequence of each middle position to obtain a candidate query key value pair group. Obtain the attention content marker Query of the candidate query key-value pair group; for each candidate query key Key in the candidate query key-value pair group, determine the relative position difference between the attention content marker Query and the candidate query key Key; determine the third difference between the relative position difference and a preset update query key; determine the third ratio between the third difference and a preset number of query key repetitions; and determine the target query key Key corresponding to the candidate query key Key based on the sum of the third ratio and the preset update query key. For each candidate query key in the candidate query key-value pair group, the candidate query key is updated using the corresponding target query key to obtain the query key-value pair group.

7. The method according to claim 1, characterized in that, In the decoding stage of the large model, based on the first query key-value pair and each of the second query key-value pairs, the target text is generated, including: During the decoding stage of the large model, the first query key-value pair and each of the second query key-value pairs are saved to the query key-value pair cache (KV cache); a first attention representation is determined based on the query key-value pair cache (KV cache); and the first text content token of the target text is predicted based on the first attention representation. Based on the first text content token, the query key-value pair cache (KV cache) is updated, and a second attention representation is determined based on the updated query key-value pair cache (KV cache). The second text content token of the target text is predicted based on the second attention representation. The prediction of the text content token of the target text is completed.

8. A target text generation device based on a large model, characterized in that, The device includes: The text segmentation module is used to acquire a text sequence, segment the text sequence to obtain various sub-text sequences; wherein the length of each sub-text sequence is less than the maximum text length supported by the large model. The determination module is used to input the first sub-text sequence, which is the first position of each sub-text sequence, into the large model, and determine the first query key-value pair corresponding to the first sub-text sequence during the prefilling stage of the large model; and to input the second sub-text sequence, which is not the first position of each sub-text sequence, and the first query key-value pair into the large model, and determine the second query key-value pair corresponding to the second sub-text sequence during the prefilling stage of the large model. The text generation module is used in the decoding stage of the large model to generate target text based on the first query key-value pair and each of the second query key-value pairs.

9. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, wherein the processor is used to implement the target text generation method based on a large model as described in any one of claims 1-7 when executing a computer program stored in the memory.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the target text generation method based on a large model as described in any one of claims 1-7.

Images