Inference method and device of text data, electronic equipment and storage medium

Abstract

The application discloses a kind of inference methods, device, electronic equipment and storage medium of text data, comprising: obtaining the given text data corresponding to the foresight token sequence of user input;First part reply token sequence is obtained by inputting foresight token sequence into large language model, and intermediate token sequence is obtained by merging first part reply token sequence and foresight token sequence, and input multiple small language models, each prior reply token and each reply token prior probability of second part reply token sequence are obtained based on each small language model each inference;Each posterior probability of each validation token sequence and the posterior reply token corresponding to each posterior probability are obtained by inputting validation token sequence into large language model in parallel;According to the verification result of each prior probability and each posterior probability, update second part reply token sequence using posterior reply token, obtain updated second part reply token sequence;Output all reply token sequence, to improve inference accuracy and efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to a reasoning method and device for text data, an electronic device, and a storage medium. BACKGROUND

[0002] In recent years, with the rapid development and wide application of artificial intelligence (AI) technology, large language models have gradually integrated into daily life. Large language models usually have tens of billions or even more parameters, and these models perform well in various tasks and have powerful and diverse functions. For example, they can interact naturally and smoothly according to the dialogue context, achieve chat communication as humans, and can complete tasks such as writing emails, creating video scripts, editing scripts, translating texts, and generating code. The emergence of these large language models marks a major breakthrough in the field of deep learning and also brings important opportunities for a wide range of applications. They play a key role in natural language processing, artificial intelligence, and computer science, and provide effective tools for scientific research and commercial applications.

[0003] However, along with the potential of these models comes the problem of model inference performance. On the one hand, due to the large number of parameters of large language models, the number of parameters even reaches the scale of 100 billion or more, and the complex network structure, the model operation reaches hundreds of billions of FLOPs (Floating Point Operations), on the other hand, due to the serial way of large language model inference, that is, decoding one token at a time, the large language model inference time may be very long, which cannot perform efficient inference, and is not suitable for applying large models to practical scenarios. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art. To this end, the present application proposes a data reasoning method based on token weighting, device, electronic equipment and storage medium, which overcomes the problem of low inference efficiency of large language models.

[0005] In a first aspect, the present application provides a reasoning method for text data, comprising:

[0006] obtaining a given text data corresponding to a foreknowledge token sequence input by a user;

[0007] inputting the foreknowledge token sequence into a large language model, obtaining a first part of the reply token sequence based on the inference of the large language model, and merging the first part of the reply token sequence with the foreknowledge token sequence to obtain an intermediate token sequence;

[0008] input the intermediate token sequence into a plurality of small language models, and based on candidate tokens obtained by each small language model each time reasoning and a probability sequence corresponding to each candidate token, calculate each prior reply token of a second part reply token sequence and a prior probability of each prior reply token;

[0009] input the intermediate token sequence and each prior reply token sequence sequentially formed verification token sequence into the large language model in parallel, obtain posterior probability of each verification token sequence based on reasoning of the large language model, and obtain posterior reply token corresponding to each posterior probability;

[0010] verify each prior reply token according to each prior probability and each posterior probability, and update the second part reply token sequence using the posterior reply token according to the verification result, to obtain an updated second part reply token sequence;

[0011] output the whole reply token sequence obtained by the first part reply token sequence and the updated second part reply token sequence; wherein the first part token sequence and the second part reply token are part of the whole reply token sequence corresponding to the given text data, and the whole reply token sequence is used to generate the whole reply data of the given text data.

[0012] According to the text data reasoning method of the present application, after obtaining the priori token sequence corresponding to the given text data input by the user, the first part reply token sequence is reasoned by using the large language model, the accuracy of the initial reply token is improved, and the next step of reasoning by the small language model is prepared. Then, the first part reply token sequence and the priori token sequence are combined to obtain an intermediate token sequence, which is input into a plurality of small language models. The second part reply token sequence is obtained by reasoning of the plurality of small language models. Then, the verification token sequence composed of the intermediate token sequence and the priori reply token is input into the large language model in parallel. The large language model is used to verify each priori reply token output by the small language model, and the second part reply token sequence is updated according to the verification result. Finally, the whole reply token sequence used to generate the reply data is output. Not only the reasoning accuracy of the text data is ensured, but also the reasoning efficiency is improved, and the user experience is greatly improved.

[0013] According to an embodiment of the present application, the intermediate token sequence is input into a plurality of small language models, and based on candidate tokens obtained by each small language model each time reasoning and a probability sequence corresponding to each candidate token, calculate each prior reply token of a second part reply token sequence and a prior probability of each prior reply token, comprising:

[0014] inputting the intermediate token sequence into a plurality of small language models, obtaining a plurality of candidate tokens and probability sequences based on reasoning of each small language model, each small language model corresponding to one probability sequence, and each probability in the probability sequence corresponding to each candidate token;

[0015] performing weighted calculation on each candidate token according to a preset weighting factor of each small language model and the probability sequence, to obtain a candidate token probability of each candidate token;

[0016] determining a current prior reply token according to the candidate token probability, and taking the candidate token probability corresponding to the current prior reply token as the prior probability;

[0017] repeating the above two steps for a set number of times to obtain each prior reply token of the second part of the reply token sequence.

[0018] In the present application, when a small language model is added for reasoning, the candidate tokens output by multiple small language models are subjected to weighted calculation, that is, the reasoning results of each small language model are integrated, thereby overcoming the problem of low reasoning accuracy when using a single small language model, and further improving the reasoning accuracy.

[0019] According to an embodiment of the present application, the verification token sequence composed of the intermediate token sequence and each prior reply token sequence is input into the large language model in parallel, and the posterior probability of each verification token sequence and the posterior reply token corresponding to each posterior probability are obtained based on the reasoning of the large language model, including:

[0020] obtaining the prior reply token sequence obtained after each reasoning of the plurality of small language models, combining the intermediate token sequence and each prior reply token sequence in sequence to obtain a plurality of verification token sequences, and the prior reply token sequence being composed of a current prior reply token and a prior prior reply token reasoned by the plurality of small language models;

[0021] inputting the verification token sequence into the large language model in parallel, obtaining the posterior probability of each verification token sequence and the posterior reply token corresponding to each posterior probability based on the reasoning of the large language model.

[0022] According to an embodiment of the present application, each prior reply token is verified according to each prior probability and each posterior probability, and the posterior reply token is used to update the second part of the reply token sequence according to the verification result, to obtain an updated second part of the reply token sequence, including:

[0023] sequentially comparing each prior probability and each posterior probability corresponding thereto;

[0024] when the target posterior probability is greater than or equal to the corresponding posterior probability, replacing the prior reply token at the corresponding position in the second part of the reply token sequence with a target posterior reply token corresponding to the target posterior probability, and deleting the prior reply token after the target posterior reply token to obtain an updated second part of the reply token sequence;

[0025] when each of the prior probabilities is less than the corresponding posterior probability, adding a posterior reply token corresponding to the last posterior probability to the end of the second part of the reply token sequence to obtain an updated second part of the reply token sequence.

[0026] After obtaining the second part of the reply token sequence by the plurality of small language model inference outputs, the present application verifies each prior reply token of the second part of the reply token sequence using a large language model, and replaces the deviated prior reply token with the posterior token generated by the large language model, which can further improve the inference accuracy of the second part of the reply token sequence, i.e., improve the accuracy of the output reply data, and improve the user experience.

[0027] According to an embodiment of the present application, the step of outputting the entire reply token sequence obtained by the first part of the reply token sequence and the updated second part of the reply token sequence comprises:

[0028] determining whether the first part of the reply token sequence and the updated second part of the reply token sequence after merging are the entire reply token sequence, if yes, outputting the entire reply token sequence, if no, and returning to the step of inputting the intermediate token sequence into a plurality of small language models, calculating each prior reply token of the second part of the reply token sequence and the prior probability of each prior reply token based on the candidate token obtained by each small language model each time and the probability sequence corresponding to each candidate token, and iteratively inferring until it is determined that the first part of the reply token sequence and the updated second part of the reply token sequence after merging are the entire reply token sequence. In this way, the complete entire reply token sequence is obtained.

[0029] According to an embodiment of the present application, the determination that the first part of the reply token sequence and the updated second part of the reply token sequence after merging are the entire reply token sequence comprises at least one of the following manners:

[0030] when the first part of the reply token sequence and the updated second part of the reply token sequence after merging reach a preset length, it is determined that the first part of the reply token sequence and the updated second part of the reply token sequence after merging are the entire reply token sequence;

[0031] When it is determined that the last token of the updated second partial reply token sequence is an end character, it is determined that the first partial reply token sequence and the updated second partial reply token sequence are merged to be the complete reply token sequence.

[0032] By determining whether the first partial reply token sequence and the updated second partial reply token sequence are merged to reach a preset length and whether the last token is an end character, it is determined whether the output token sequence is the complete reply token sequence, ensuring that complete reply data is finally output.

[0033] According to an embodiment of the present application, the inputting of the priori token sequence into the large language model includes:

[0034] The inputting of the priori token sequence into the large language model includes iteratively reasoning the priori token sequence based on the large language model for a preset number of times to obtain the first partial reply token sequence.

[0035] The preset number of times is less than the total number of iterations, and the total number of iterations is the number of iterations of the large language model for reasoning the priori token sequence to obtain the complete reply token sequence.

[0036] Using the large language model to iteratively obtain the first partial reply token sequence for a preset number of times instead of using the large language model to obtain the complete reply token sequence at one time reduces the large amount of computation and long reasoning time caused by the large language model, and makes the first partial reply token sequence more accurate, thereby laying a foundation for improving the reasoning accuracy of the small language model.

[0037] In a second aspect, the present application provides an inference device for text data, which comprises:

[0038] An acquisition module is configured to acquire a priori token sequence corresponding to given text data input by a user;

[0039] A first inference module is configured to input the priori token sequence into a large language model, obtain a first partial reply token sequence based on reasoning of the large language model, and merge the first partial reply token sequence with the priori token sequence to obtain an intermediate token sequence.

[0040] A second inference module is configured to input the intermediate token sequence into a plurality of small language models, calculate each priori reply token of a second partial reply token sequence and a priori probability of each priori reply token based on a candidate token obtained by each reasoning of each small language model and a probability sequence corresponding to each candidate token.

[0041] a third inference module configured to input, in parallel, the intermediate token sequence and each of the validation token sequences composed of the intermediate token sequence and each of the prior reply token sequences into the large language model, and obtain, based on inference of the large language model, a posterior probability of each of the validation token sequences and a posterior reply token corresponding to each of the posterior probabilities;

[0042] a verification module configured to verify each of the prior reply tokens according to each of the prior probabilities and each of the posterior probabilities, and update the second part of the reply token sequence using the posterior reply token according to a verification result to obtain an updated second part of the reply token sequence, and output a merged reply token sequence of the first part of the reply token sequence and the updated second part of the reply token sequence;

[0043] an output module configured to output a complete reply token sequence obtained from the first part of the reply token sequence and the updated second part of the reply token sequence;

[0044] wherein the first part of the token sequence and the second part of the reply token are both part of a complete reply token sequence corresponding to the given text data, and the complete reply token sequence is used to generate complete reply data of the given text data.

[0045] According to the text data inference apparatus of the present application, after obtaining the priori token sequence corresponding to the given text data input by the user, the first part of the reply token sequence is inferred using the large language model, the accuracy of the initial reply token is improved, and the next step of inference of the small language model is prepared. Then, the first part of the reply token sequence and the priori token sequence are merged to obtain the intermediate token sequence, which is input into multiple small language models. The second part of the reply token sequence is obtained by inference of the multiple small language models. Then, the validation token sequence composed of the intermediate token sequence and the priori reply token is input into the large language model in parallel. The large language model is used to verify each of the priori reply tokens output by the small language model, and the second part of the reply token sequence is updated according to the verification result. Finally, the complete reply token sequence used to generate the reply data is output. Not only the inference accuracy of the text data is ensured, but also the inference efficiency is improved, and the user experience is greatly improved.

[0046] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the text data inference method of the first aspect.

[0047] In a fourth aspect, the present application provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the text data inference method of the first aspect.

[0048] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, the communication interface and the processor are coupled, the processor is used to run programs or instructions, and the inference method of the text data according to the first aspect is realized.

[0049] In a sixth aspect, the present application provides a computer program product, comprising a computer program, when the computer program is executed by a processor, the inference method of the text data according to the first aspect is realized.

[0050] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0051] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the appended drawings.

[0052] Figure 1 is one of the flow diagrams of the inference method of the text data provided by the embodiments of the present application;

[0053] Figure 2 is the second flow diagram of the inference method of the text data provided by the embodiments of the present application;

[0054] Figure 3 is the third flow diagram of the inference method of the text data provided by the embodiments of the present application;

[0055] Figure 4 is the fourth flow diagram of the inference method of the text data provided by the embodiments of the present application;

[0056] Figure 5 is the structural diagram of the inference device of the text data provided by the embodiments of the present application;

[0057] Figure 6 is the structural diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0059] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of a kind and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in a "or" relationship.

[0060] In recent years, with the rapid development and wide application of artificial intelligence (AI) technology, large language models have gradually integrated into daily life. Large language models usually have tens of billions or even more parameters, and these models perform well in a variety of tasks and are powerful and diverse. For example, they can interact naturally and smoothly according to the context of the conversation, achieve the same chat communication as humans, and can complete a variety of tasks such as writing emails, creating video scripts, editing scripts, translating texts, and generating code. The emergence of these large language models marks a major advance in the field of deep learning, and also brings important opportunities for a wide range of applications. They play a key role in natural language processing, artificial intelligence and computer science, and provide effective tools for scientific research and commercial applications.

[0061] But due to the large amount of parameters of the large language model, the operation amount is extremely large, and because the large language model adopts a serial mode for reasoning, and decodes one token at a time, the reasoning time of the large language model is very long, and cannot meet the efficient reply requirements in actual application scenarios. Patent 202311085639.X discloses a text data reasoning method, device, storage medium and electronic equipment, which adopts a method of splicing multiple tokens into a sequence, and then uses a large language model to simultaneously reason the token sequence and generate multiple pieces of reply data of text data in parallel, but this patent does not optimize the reasoning acceleration for single text data. The paper "Fast Inference from Transformers via Speculative Decoding" proposes a speculative sampling method for large language model reasoning. The paper uses a large language model and a small language model for reasoning acceleration, inputs a piece of text data, generates tokens by the small language model, and then the large language model reasons at certain intervals, but this method has two problems: first, the reasoning performance of the small language model itself is not high, which is easy to generate many wrong tokens; second, the accuracy of the initial reasoning token result generated by the small language model is too low. Based on this, the present application proposes a text data reasoning method, which can ensure the reasoning efficiency while ensuring the accuracy of the reasoning token.

[0062] Related name and fame explanation:

[0063] 1. Token: a kind of smallest semantic unit, also known as word segmentation, word unit, token;

[0064] 2. Word segmenter: a tool for converting natural language text information into tokens;

[0065] 3. Language model: a model for processing natural language, which can generate the next word or character from the given text information. Language model can be used for various natural language processing tasks, such as machine translation, text generation, etc.

[0066] 4. Large language model: a language model with large number of parameters and large amount of computation.

[0067] 5. Small language model: a language model with small number of parameters and small amount of computation.

[0068] The text data reasoning method, device, electronic equipment and storage medium provided by the embodiments of the present application will be described in detail in combination with the accompanying drawings and specific embodiments and their application scenarios.

[0069] Among them, the text data reasoning method can be applied to a terminal, and can be executed by hardware or software in the terminal.

[0070] The inference method for text data provided in the embodiments of the present application can be executed by an electronic device or a functional module or functional entity capable of implementing the inference method for text data in the electronic device. The electronic device mentioned in the embodiments of the present application includes but is not limited to a mobile phone, a tablet computer, a computer, a camera, a wearable device, and the like. The inference method for text data provided in the embodiments of the present application is described below by taking an electronic device as an execution subject.

[0071] As shown in the method for inferring text data, Figure 1 The method for inferring text data includes steps 110, 120, 130, 140, and 150.

[0072] In step 110, a foreknowledge token sequence corresponding to given text data input by a user is obtained.

[0073] In some embodiments, step S100 specifically includes obtaining given text data input by a user and converting the given text data into a foreknowledge token sequence by using a sentence switching algorithm. The sentence switching algorithm can be provided in a word segmentation tool, and the foreknowledge token sequence of the given text data can be obtained by using the word segmentation tool.

[0074] It can be understood that the foreknowledge token sequence is a sequence composed of foreknowledge tokens obtained by converting the given text data input by the user into the foreknowledge tokens. The foreknowledge token can be understood as a token that can be obtained from the given text data input by the user in advance, and is a token that can be known in advance without inputting a language model.

[0075] In step S120, the foreknowledge token sequence is input into a large language model, a first part of a reply token sequence is obtained based on inference of the large language model, and the first part of the reply token sequence is combined with the foreknowledge token sequence to obtain an intermediate token sequence.

[0076] Since the initial inference token result generated by the small language model has too low accuracy, the large language model is used to infer the initial reply token (i.e., the first part of the reply token sequence in the present application), so as to improve the accuracy of the initial reply token, and the inference accuracy of the subsequent small language model is also improved to a certain extent.

[0077] In some embodiments, step S120 specifically includes inputting the foreknowledge token sequence into the large language model, iteratively inferring the foreknowledge token sequence based on the large language model for a preset number of times to obtain the first part of the reply token sequence, wherein the preset number of times is less than the total number of iterations, and the total number of iterations is the number of iterations of the large language model for inferring the foreknowledge token sequence to obtain the total reply token sequence.

[0078] Since the result of one inference of the large language model is relatively accurate, but the large language model can only decode one token each time, in order to obtain relatively accurate reply data, the large language model is used to iteratively infer the first-knowledge token sequence for a preset number of times M to obtain M first reply tokens with high accuracy, and then the M first reply tokens are sequentially sorted according to the output order to obtain a first part of the reply token sequence [t1, t2, L, t i ,L,t M ], wherein t i represents the first reply token obtained by the large language model in the i-th inference.

[0079] It should be noted that the preset number of times M is less than the total number of iterations A, preferably, the preset number of times M is less than half, 1 / 3, 1 / 4, 1 / 5, 1 / 10 or less of the total number of iterations A, which not only effectively reduces the inference number of the large language model, but also reduces the long time problem caused by the inference of the large language model, and improves the inference accuracy of the initial reply token. It can be understood that if the large language model infers the entire reply token sequence of the first-knowledge token sequence, it needs 30000 times of inference, each time needs 0.001S, and it needs 30s, and if it is inferred 1 / 10 times, it only takes 3s, greatly reducing the time consumption of the large language model inference.

[0080] In step S130, the intermediate token sequence is input into a plurality of small language models, and based on the candidate token obtained by each small language model each time and the probability sequence corresponding to each candidate token, each prior reply token of the second part of the reply token sequence and the prior probability of each prior reply token are calculated.

[0081] After merging the first-knowledge token sequence and the first part of the reply token sequence inferred by the large language model to obtain the intermediate token sequence, the intermediate token sequence is input into a plurality of small language models. The reason why the intermediate token sequence is input into the small language model instead of the first-knowledge token sequence is that the inference accuracy of the small language model is low, and the inference accuracy of the small language model can be improved based on the part of the token sequence inferred by the large language model. Based on the candidate token obtained by each small language model each time and the probability sequence corresponding to each candidate token, each prior reply token of the second part of the reply token sequence and the prior probability of each prior reply token are calculated. It should be noted that the small language model can only infer one token each time, so it is necessary to set the number of iterations N to obtain N prior reply tokens, and then sequentially sort the N prior reply tokens according to the output order to obtain the second part of the reply token sequence [u1, u2, L, u j ,L,u N ], wherein u jrepresents the jth inference of a plurality of small language models to obtain a prior reply token. At the same time, the small language model also outputs the prior probability R(x) = [r1(x), r2(x), L, r j (x), L, r N (x)] corresponding to each prior reply token, wherein r j (x) is the prior probability corresponding to the prior reply token u j .

[0082] In some embodiments, as shown in FIG. 1, step S130 specifically comprises: S131, inputting the intermediate token sequence into a plurality of small language models, obtaining a plurality of candidate token sequences and probability sequences based on the inference of each small language model, each small language model corresponding to one probability sequence, and each probability in the probability sequence corresponding to each candidate token. Figure 2

[0083] S132, according to the preset weighting factor of each small language model and the probability sequence, performing weighted calculation on each candidate token to obtain the candidate token probability of each candidate token.

[0084] S133, determining the current prior reply token according to the candidate token probability, and taking the candidate token probability corresponding to the current prior reply token as the prior probability.

[0085] S134, repeating the above steps S132 and S133 for a plurality of times to obtain each prior reply token of the second part of the reply token sequence.

[0086] In step S133, the step can further comprise: comparing each candidate token probability to obtain the maximum candidate token probability, and taking the candidate token corresponding to the maximum candidate token probability as the current prior reply token.

[0087] The following will be described in detail taking two small language models as an example:

[0088] Suppose the preset weighting factor of the first small language model is w1, and the preset weighting factor of the second small language model is w2. In the jth inference, the candidate tokens inferred by the first small language model and the second small language model are a, b, c, d, and e in turn, the probability of each candidate token inferred by the first small language model forms a first probability sequence p(x), and suppose p(x) = [0.1, 0.2, 0.3, 0.4, 0]. The probability of each candidate token inferred by the second small language model forms a second probability sequence q(x), and suppose q(x) = [0, 0.1, 0.2, 0.3, 0.4]. Suppose w1 = 0.4 and w2 = 0.6, then according to the formula r(x) = w1 * p(x) + w2 * q(x), r j ​(x) = [0.4, 1.4, 2.4, 3.4, 2.4], that is, the candidate token probabilities of the candidate tokens a, b, c, d, and e are 0.4, 1.4, 2.4, 3.4, and 2.4 respectively. Comparing each candidate token probability, it can be seen that the probability of the candidate token d is the largest, so the candidate token d is taken as the prior reply token output by the first small language model and the second small language model at the jth reasoning, and 3.4 is taken as the prior probability of the prior reply token d.

[0089] It should be noted that the number of small language models is not limited to the above-mentioned two, and can be three, four, five, or the like. However, too many small language models will cause redundant calculation. Preferably, the number of small language models is two. The two small language models can be a LLaMA-68M model with a parameter quantity of 68 million and a LLaMA-160M model with a parameter quantity of 160 million. The large language model can be a LLaMA-30B model with a parameter quantity of 300 billion. Preferably, the parameter quantity (corresponding to the small weighting factor) of the first small language model in the above specific embodiment is smaller than the parameter quantity (corresponding to the large weighting factor) of the second small language model, so as to further improve the reasoning accuracy.

[0090] It should be noted that the large language model and the small language model in each of the above embodiments are trained by using a unified architecture and the same training data, so that the types of candidate tokens output by the large language model and the small language model are basically consistent.

[0091] When the small language model is used for reasoning in the present application, the candidate tokens output by the multiple small language models are weighted and calculated, that is, the reasoning results of the small language models are integrated, so as to overcome the problem of low reasoning accuracy when only one small language model is used, and further improve the reasoning accuracy.

[0092] In step S140, the intermediate token sequence and each of the prior reply token sequences are sequentially combined to obtain multiple verification token sequences, and the verification token sequences are input to the large language model in parallel. The posterior probability of each verification token sequence and the posterior reply token corresponding to each posterior probability are obtained based on the reasoning of the large language model.

[0093] In order to further determine the accuracy of the prior reply token output by each small language model, the large language model is also used for verification. In some embodiments, step S140 specifically comprises: obtaining the prior reply token sequence obtained after each reasoning of the multiple small language models, the prior reply token sequence being composed of the current prior reply token and the previous prior reply token reasoned by the multiple small language models; sequentially combining the intermediate token sequence and each of the prior reply token sequences to obtain multiple verification token sequences; inputting the verification token sequences to the large language model in parallel; and obtaining the posterior probability of each verification token sequence and the posterior reply token corresponding to each posterior probability based on the reasoning of the large language model.

[0094] For ease of understanding, the following is expanded to explain that when multiple small language models do not perform reasoning, there is no prior reply token, and the verification token sequence is the intermediate token sequence [V], the token output after the first reasoning is the prior reply token u1, so the corresponding prior reply token sequence is [u1], and after combining it with the intermediate token sequence V, the verification token sequence after the first reasoning is obtained [V, u1], similarly, the prior reply token sequence obtained after the jth reasoning is [u1, u2, u3,.... u j ], and the verification token sequence after the jth reasoning is obtained [V, u1, u2, u3,.... u j ], and so on, after N times of reasoning, multiple verification token sequences [V], [V, u1], L, [V, u1, u2, L, u i ], L, [V, u1, u2, L, u N ] are obtained, and the multiple verification token sequences are input into the large language model in parallel, and the large language model can obtain the posterior probability S j (x) of each verification token sequence after only one reasoning. j The posterior probability sequence composed of each posterior probability is S(x) = [s1(x), s2(x), L, s j (x), L, s N (x), s N+1 (x)], and the posterior reply token sequence composed of each posterior reply token is [u'1, u'2,.... u' j ,...., u' N+1 ].

[0095] It is worth noting that although the large language model is used to perform reasoning on multiple verification token sequences in this step, since it is parallel input (i.e., one-time reasoning), the amount of computation used by the large language model at this time is only one-time computation, which has little effect on the timeliness of reasoning.

[0096] In step S150, each prior reply token is verified according to each prior probability and each posterior probability, and the second part of the reply token sequence is updated using the posterior reply token according to the verification result, to obtain an updated second part of the reply token sequence.

[0097] In some embodiments, as Figure 3As shown, the step S150 specifically includes: S151, sequentially comparing each of the prior probabilities and each of the posterior probabilities. S152, when there is a target posterior probability that the prior probability is greater than or equal to the corresponding posterior probability, replacing the target posterior reply token corresponding to the target posterior probability with the prior reply token at the corresponding position in the second part of the reply token sequence, and deleting the prior reply token after the target posterior reply token, to obtain an updated second part of the reply token sequence. S153, when each of the prior probabilities is less than each of the corresponding posterior probabilities, adding the posterior reply token corresponding to the last posterior probability to the end of the second part of the reply token sequence to obtain an updated second part of the reply token sequence.

[0098] This step is to obtain a plurality of small language model inference output second part of the reply token sequence [u1, u2, L, u j , L, u N ] and its prior probability sequence R(x) = [r1(x), r2(x), L, r j (x), L, r N (x)], and the posterior probability sequence S(x) = [s1(x), s2(x), L, s j (x), L, s N (x), s N+1 (x)] and the posterior reply token sequence [u'1, u'2,..., u' j ,...., u' N+1 ] output by the large language model for the verification token sequence, and then determine how to use the posterior reply token to update the related prior reply token in the second part of the reply token sequence according to each prior probability and each posterior probability. There are two cases, the first is that the prior probability is greater than or equal to the posterior probability, at which time the target posterior reply token corresponding to the target posterior probability is replaced with the prior reply token at the corresponding position in the second part of the reply token sequence, and the prior reply token after the target posterior reply token is deleted, to obtain an updated second part of the reply token sequence. For example, if r j (x) > s j (x), the updated second part of the reply token sequence is [V, u1, u2, u3,.... u j-1 , u' jThe reason for this is that, in theory, the probability of the token inferred by the small language model should be smaller than the probability of the token inferred by the large language model. If the prior probability is greater than or equal to the posterior probability, it proves that the inference of multiple small language models produces new bias, and the token needs to be corrected. The second case is when the prior probability is less than the posterior probability. At this time, the last posterior probability corresponding to the posterior reply token is added to the end of the second part of the reply token sequence to obtain an updated second part of the reply token sequence, that is, the updated second part of the reply token sequence is [V, u1, u2, u3,.... u N , u' N+1 ]. Finally, the first part of the reply token sequence and the updated second part of the reply token sequence are output to obtain the complete reply token sequence. In this way, the inference accuracy of the second part of the reply token sequence can be further improved, that is, the accuracy of the output reply data is improved, and the user experience is improved.

[0099] Step S160: Output the complete reply token sequence obtained by the first part of the reply token sequence and the updated second part of the reply token sequence.

[0100] It should be noted that the first part of the token sequence and the second part of the reply token are both part of the complete reply token sequence corresponding to the given text data, and the complete reply token sequence is used to generate the complete reply data of the given text data.

[0101] In some embodiments, as shown in Figure 4 Step S160 specifically includes: S161, determining whether the first part of the reply token sequence and the updated second part of the reply token sequence are combined to be the complete reply token sequence. If yes, output the complete reply token sequence. If no, combine the updated second part of the reply token sequence with the intermediate token sequence to obtain a new intermediate token sequence, and return to steps S130-S150 until it is determined that the first part of the reply token sequence and the updated second part of the reply token sequence are combined to be the complete reply token sequence. In this way, complete reply data can be obtained, and user experience is further improved.

[0102] The inference method of text data provided by the embodiments of the present application can improve the accuracy of the initial reply token and lay the foundation for the next step of inference of the small language model after obtaining the given text data corresponding to the first token sequence. Then, the first part of the reply token sequence and the first token sequence are merged to obtain an intermediate token sequence, which is input into multiple small language models. The second part of the reply token sequence is obtained by using the inference of multiple small language models. Then, the verification token sequence composed of the intermediate token sequence and the first token sequence is input into the large language model in parallel. The large language model verifies each first token output by the small language model, and updates the second part of the reply token sequence according to the verification result. Finally, the complete reply token sequence for generating the reply data is output, which not only ensures the inference accuracy of the text data, but also improves the inference efficiency, greatly improving the user experience.

[0103] In some embodiments, step S161 includes at least one of the following: when the first part of the reply token sequence and the updated second part of the reply token sequence are merged to reach a preset length, it is determined that the first part of the reply token sequence and the updated second part of the reply token sequence are merged to be the complete reply token sequence. When it is determined that the last token of the updated second part of the reply token sequence is an end character, it is determined that the first part of the reply token sequence and the updated second part of the reply token sequence are merged to be the complete reply token sequence.

[0104] When the last token of the updated second part of the reply token sequence is an end character, it is proved that the first part of the reply token sequence and the updated second part of the reply token sequence are merged to be the complete reply token sequence, and the complete reply token sequence is output. In another embodiment, since the reply data can be predictable, i.e., the preset length of the final reply data is known in advance, it can also be determined whether the first part of the reply token sequence and the updated second part of the reply token sequence are merged to be the complete reply token sequence according to the preset length. Of course, it can also be a combination of the two, i.e., when it is determined that the first part of the reply token sequence and the updated second part of the reply token sequence are merged to reach a preset length and it is determined that the last token of the updated second part of the reply token sequence is an end character, it is determined that the first part of the reply token sequence and the updated second part of the reply token sequence are merged to be the complete reply token sequence, so as to fully guarantee the integrity of the reply data.

[0105] The inference method for text data provided in the embodiments of the present application can be executed by an inference device for text data. The inference method for text data is taken as an example in the embodiments of the present application to illustrate the inference device for text data provided in the embodiments of the present application.

[0106] The embodiments of the present application also provide an inference device for text data, as shown in Figure 5 The inference device for text data comprises an acquisition module 110 configured to acquire a foreknowledge token sequence corresponding to given text data input by a user;

[0107] A first inference module 120 is configured to input the foreknowledge token sequence into a large language model, obtain a first partial reply token sequence based on inference of the large language model, and combine the first partial reply token sequence with the foreknowledge token sequence to obtain an intermediate token sequence.

[0108] A second inference module 130 is configured to input the intermediate token sequence into a plurality of small language models, calculate each prior reply token of a second partial reply token sequence and a prior probability of each prior reply token based on a candidate token obtained by each inference of each small language model and a probability sequence corresponding to each candidate token.

[0109] A third inference module 140 is configured to input a verification token sequence composed of the intermediate token sequence and each prior reply token sequence in sequence into the large language model in parallel, obtain a posterior probability of each verification token sequence and a posterior reply token corresponding to each posterior probability based on inference of the large language model.

[0110] A verification module 150 is configured to verify each prior reply token according to each prior probability and each posterior probability, and update the second partial reply token sequence using the posterior reply token according to a verification result to obtain an updated second partial reply token sequence.

[0111] An output module 160 is configured to output a complete reply token sequence obtained by the first partial reply token sequence and the updated second partial reply token sequence.

[0112] The first partial token sequence and the second partial reply token are both part of a complete reply token sequence corresponding to the given text data, and the complete reply token sequence is used to generate complete reply data of the given text data.

[0113] According to the text data reasoning device provided in the embodiments of the present application, after the given text data corresponding to the priori token sequence input by the user is obtained, the first part of the reply token sequence is inferred by using the large language model, the accuracy of the initial reply token is improved, and the small language model inference in the next step is prepared, then the first part of the reply token sequence and the priori token sequence are combined to obtain an intermediate token sequence, the intermediate token sequence and the priori reply token are input into multiple small language models in parallel, the second part of the reply token sequence is obtained by using the inference of the multiple small language models, then the verification token sequence composed of the intermediate token sequence and the priori reply token is input into the large language model in parallel, each priori reply token output by the small language model is verified by using the large language model, and the second part of the reply token sequence is updated according to the verification result, and finally the whole reply token sequence used for generating the reply data is output, which not only ensures the reasoning accuracy of the text data, but also improves the reasoning efficiency, and greatly improves the user experience.

[0114] In some embodiments, the second reasoning module 130 is specifically configured to input the intermediate token sequence into multiple small language models, obtain multiple candidate tokens and probability sequences based on the inference of each small language model, each small language model corresponds to one probability sequence, and each probability in the probability sequence corresponds to each candidate token; each candidate token is weighted and calculated according to a preset weighting factor of each small language model and the probability sequence, to obtain a candidate token probability of each candidate token; a current priori reply token is determined according to the candidate token probability, and the candidate token probability corresponding to the current priori reply token is taken as the priori probability; the above steps are repeated for a set number of times to obtain each priori reply token of the second part of the reply token sequence.

[0115] In some embodiments, the third reasoning module 140 is specifically configured to obtain the priori reply token sequence obtained after each inference of the multiple small language models, combine the intermediate token sequence and each priori reply token sequence in sequence to obtain multiple verification token sequences, and the priori reply token sequence is composed of a current priori reply token and a previous priori reply token inferred by the multiple small language models; the verification token sequences are input into the large language model in parallel, and the posterior probability of each verification token sequence and the posteriori reply token corresponding to each posterior probability are obtained based on the inference of the large language model.

[0116] In some embodiments, the verification module 150 is specifically configured to compare each of the prior probabilities with the corresponding posterior probability in sequence; when there is a target posterior probability that is greater than or equal to the corresponding prior probability, replace the prior reply token at the corresponding position in the second part of the reply token sequence with a target posterior reply token corresponding to the target posterior probability, and delete the prior reply token after the target posterior reply token to obtain an updated second part of the reply token sequence; when each of the prior probabilities is less than the corresponding posterior probability, add a posterior reply token corresponding to the last posterior probability to the end of the second part of the reply token sequence to obtain an updated second part of the reply token sequence.

[0117] In some embodiments, the output module 160 is specifically configured to determine whether the first part of the reply token sequence and the updated second part of the reply token sequence after merging are the full reply token sequence, if yes, output the full reply token sequence, if not, return to the step of inputting the intermediate token sequence into a plurality of small language models, and based on the candidate token obtained by each small language model each time and the probability sequence corresponding to each candidate token, calculating each prior reply token of the second part of the reply token sequence and the prior probability of each prior reply token to perform iterative reasoning until it is determined that the first part of the reply token sequence and the updated second part of the reply token sequence after merging are the full reply token sequence.

[0118] In some embodiments, the determination that the first part of the reply token sequence and the updated second part of the reply token sequence after merging are the full reply token sequence is implemented in at least one of the following ways: when the first part of the reply token sequence and the updated second part of the reply token sequence after merging reach a preset length, it is determined that the first part of the reply token sequence and the updated second part of the reply token sequence after merging are the full reply token sequence. When it is determined that the token at the end of the updated second part of the reply token sequence is an end character, it is determined that the first part of the reply token sequence and the updated second part of the reply token sequence after merging are the full reply token sequence.

[0119] In some embodiments, the first reasoning module 120 is specifically configured to input the priori token sequence into the large language model, and based on the large language model iteratively reasoning the priori token sequence for a preset number of times to obtain the first part of the reply token sequence; wherein the preset number of times is less than the total number of iterations, and the total number of iterations is the number of iterations of the large language model for reasoning the priori token sequence to obtain the full reply token sequence.

[0120] The inference apparatus for text data in the embodiments of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and the like, and can also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like, and the embodiments of the present application are not limited in this regard.

[0121] The inference apparatus for text data in the embodiments of the present application can be a device with an operating system. The operating system can be a Windows operating system, an Android operating system, an IOS operating system, or other possible operating systems, and the embodiments of the present application are not limited in this regard.

[0122] The inference apparatus for text data provided in the embodiments of the present application can implement the method embodiments Figures 1 to 4 The processes implemented by the method embodiments are not repeated here to avoid repetition.

[0123] In some embodiments, as shown in Figure 6 The embodiments of the present application also provide an electronic device 800, which includes a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the program is executed by the processor 801, it implements the processes of the above-mentioned inference method embodiments for text data and achieves the same technical effects. The processes are not repeated here to avoid repetition.

[0124] It should be noted that the electronic device in the embodiments of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0125] The embodiment of the present application further provides a non-transitory computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement each process of the inference method for text data and achieve the same technical effects. To avoid repetition, details are not described herein.

[0126] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0127] The embodiment of the present application further provides a computer program product, which includes a computer program. The computer program is executed by a processor to implement the inference method for text data.

[0128] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0129] The embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface is coupled with the processor. The processor is used to run a program or an instruction to implement each process of the inference method for text data and achieve the same technical effects. To avoid repetition, details are not described herein.

[0130] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0131] It should be noted that, in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the process, method, article or apparatus including the element. In addition, it should be pointed out that the scope of the method and apparatus in the present application is not limited to the order of performing the functions as shown or discussed, but can also include performing the functions in a substantially simultaneous manner or in a reverse order, for example, the described method can be performed in an order different from that described, and various steps can be added, omitted or combined. In addition, the features described with reference to some examples can be combined in other examples.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned example methods can be realized by means of software and necessary general hardware platforms, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk), and includes a plurality of instructions for causing a terminal (which can be a mobile phone, a computer, a server, or a network device) to execute the methods described in the various embodiments of the present application.

[0133] The embodiments of the present application are described above in combination with the accompanying drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative, rather than limiting, and those skilled in the art can make many forms without departing from the purpose of the present application and the scope protected by the claims under the inspiration of the present application, which all belong to the protection of the present application.

[0134] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an illustrative embodiment", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0135] Although the embodiments of the present application have been shown and described, those skilled in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and purposes of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. An inference method of text data, characterized by, The method comprises the following steps: obtaining a given text data corresponding to a prophetic token sequence input by a user; inputting the prophetic token sequence into a large language model, obtaining a first part of the reply token sequence based on the inference of the large language model, and merging the first part of the reply token sequence with the prophetic token sequence to obtain an intermediate token sequence; inputting the intermediate token sequence into a plurality of small language models, and based on the candidate token and the probability sequence corresponding to each candidate token obtained by each small language model, calculating each prior reply token of the second part of the reply token sequence and the prior probability of each prior reply token; inputting the intermediate token sequence and each prior reply token sequence into the large language model in parallel to obtain the posterior probability of each validation token sequence and the posterior reply token corresponding to each posterior probability based on the inference of the large language model; verifying each prior reply token according to each prior probability and each posterior probability, and updating the second part of the reply token sequence using the posterior reply token according to the verification result to obtain an updated second part of the reply token sequence; outputting the entire reply token sequence obtained by the first part of the reply token sequence and the updated second part of the reply token sequence; wherein the first part of the reply token sequence and the second part of the reply token sequence are both part of the entire reply token sequence corresponding to the given text data, and the entire reply token sequence is used to generate the entire reply data of the given text data.

2. The method of claim 1, wherein, The method comprises the following steps: inputting the intermediate token sequence into a plurality of small language models, and based on the candidate token and the probability sequence corresponding to each candidate token obtained by each small language model, calculating each prior reply token of the second part of the reply token sequence and the prior probability of each prior reply token; inputting the intermediate token sequence into a plurality of small language models, and based on the inference of each small language model, obtaining a plurality of candidate tokens and probability sequences, each small language model corresponding to one probability sequence, and each probability in the probability sequence corresponding to each candidate token; weighting each candidate token according to the preset weighting factor of each small language model and the probability sequence to obtain the candidate token probability of each candidate token; determining the current prior reply token according to the candidate token probability, and taking the candidate token probability corresponding to the current prior reply token as the prior probability; returning to the step of inputting the intermediate token sequence into a plurality of small language models, and based on the inference of each small language model, obtaining a plurality of candidate tokens and probability sequences, each small language model corresponding to one probability sequence, and each probability in the probability sequence corresponding to each candidate token, and repeating the step for a set number of times to obtain each prior reply token of the second part of the reply token sequence.

3. The method of claim 1, wherein, The verification token sequence composed of the intermediate token sequence and each of the prior reply token sequences in sequence is input into the large language model in parallel, and posterior probabilities of each of the verification token sequences and posterior reply tokens corresponding to each of the posterior probabilities are obtained based on inference of the large language model, and the method comprises: obtaining the prior reply token sequences obtained after each inference of the plurality of small language models, and combining the intermediate token sequence and each of the prior reply token sequences in sequence to obtain the plurality of verification token sequences, wherein the prior reply token sequence is composed of a current prior reply token and a prior prior reply token inferred by the plurality of small language models; the verification token sequence is input into the large language model in parallel, and posterior probabilities of each of the verification token sequences and posterior reply tokens corresponding to each of the posterior probabilities are obtained based on inference of the large language model.

4. The method of claim 1, wherein, The first part of the reply token sequence and the second part of the reply token sequence are combined to obtain the full reply token sequence, and the full reply token sequence is output, and the method comprises: sequentially comparing each of the prior probabilities and each of the posterior probabilities; when there is a target posterior probability that the prior probability is greater than or equal to the corresponding posterior probability, replacing the prior reply token at the corresponding position in the second part of the reply token sequence with a target posterior reply token corresponding to the target posterior probability, and deleting the prior reply token after the target posterior reply token to obtain an updated second part of the reply token sequence; when each of the prior probabilities is less than each of the posterior probabilities, adding a posterior reply token corresponding to the last posterior probability to the end of the second part of the reply token sequence to obtain an updated second part of the reply token sequence.

5. The method according to any one of claims 1 to 4, characterized in that, The step of outputting the full reply token sequence obtained from the first part of the reply token sequence and the updated second part of the reply token sequence comprises: determining whether the first part of the reply token sequence and the updated second part of the reply token sequence are combined to be the full reply token sequence, if yes, outputting the full reply token sequence, if no, returning to the step of inputting the intermediate token sequence into the plurality of small language models, and calculating each prior reply token of the second part of the reply token sequence and the prior probability of each of the prior reply tokens based on each candidate token obtained after each inference of each of the small language models and the probability sequence corresponding to each candidate token to perform iterative inference until it is determined that the first part of the reply token sequence and the updated second part of the reply token sequence are combined to be the full reply token sequence.

6. The method of claim 5, wherein, The determination that the first part of the reply token sequence and the updated second part of the reply token sequence are combined to be the full reply token sequence comprises at least one of the following manners: when the first part of the reply token sequence and the updated second part of the reply token sequence are combined to reach a preset length, it is determined that the first part of the reply token sequence and the updated second part of the reply token sequence are combined to be the full reply token sequence; When it is determined that the last token of the updated second partial reply token sequence is an end character, it is determined that the first partial reply token sequence and the updated second partial reply token sequence are merged into the complete reply token sequence.

7. The method according to any one of claims 1 to 4, characterized in that, The inputting the prophet token sequence into the large language model comprises: The inputting the prophet token sequence into the large language model comprises: The inputting the prophet token sequence into the large language model comprises:

8. An inference device of text data, characterized by, The inputting the prophet token sequence into the large language model comprises: The device comprises: The acquisition module is configured to acquire a prophet token sequence corresponding to given text data input by a user. The first inference module is configured to input the prophet token sequence into a large language model, obtain a first partial reply token sequence based on inference of the large language model, and merge the first partial reply token sequence with the prophet token sequence to obtain an intermediate token sequence. The second inference module is configured to input the intermediate token sequence into a plurality of small language models, calculate each prior reply token of a second partial reply token sequence and a prior probability of each prior reply token based on a candidate token obtained by each inference of each small language model and a probability sequence corresponding to each candidate token. The third inference module is configured to input a verification token sequence composed of the intermediate token sequence and each prior reply token sequence in sequence into the large language model in parallel, and obtain a posterior probability of each verification token sequence and a posterior reply token corresponding to each posterior probability based on inference of the large language model. The verification module is configured to verify each prior reply token according to each prior probability and each posterior probability, update the second partial reply token sequence using the posterior reply token according to a verification result, obtain an updated second partial reply token sequence, and output a merged reply token sequence of the first partial reply token sequence and the updated second partial reply token sequence. The output module is configured to output a complete reply token sequence obtained from the first partial reply token sequence and the updated second partial reply token sequence.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The first partial reply token sequence and the second partial reply token sequence are both part of a complete reply token sequence corresponding to the given text data, and the complete reply token sequence is used to generate complete reply data of the given text data.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The processor implements the inference method of the text data according to any one of claims 1-7 when executing the program. The computer program implements the inference method of the text data according to any one of claims 1-7 when executed by the processor.

Images