Text data reasoning method and device, electronic equipment and storage medium

Through the joint inference between large language model and small language model, the prior and posterior probability verification are used to optimize token sequence generation, the problem of low inference efficiency of large language model is solved, and efficient inference and accuracy improvement of text data is achieved.

CN119047572AActive Publication Date: 2024-11-29GRG BANKING EQUIPMENT CO LTD

Patent Information

Application Number
CN202411004082.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-29
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

Due to the huge parameter volume and serial reasoning method, the large language model has low inference efficiency and cannot be suitable for practical application scenarios.

Method used

The data inference method based on token weighting is adopted, and the joint inference of large language model and small language model is used to optimize the generation of token sequences through prior and posterior probability verification, and improve the inference efficiency and accuracy.

Benefits of technology

It realizes efficient inference of text data, improves user experience, overcomes the problem of low inference accuracy using small language models alone, and reduces the amount and time of inference of large language models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119047572A_ABST
    Figure CN119047572A_ABST
Patent Text Reader

Abstract

The invention discloses a text data reasoning method and device, electronic equipment and a storage medium. The text data reasoning method comprises the following steps: acquiring a prophet token sequence corresponding to given text data input by a user; inputting the prophet token sequence into the large language model to obtain a first part of reply token sequence, combining the first part of reply token sequence with the prophet token sequence to obtain an intermediate token sequence, and inputting the intermediate token sequence into a plurality of small language models, reasoning each time based on each small language model to obtain each prior reply token of the second partial reply token sequence and the prior probability of each reply token; inputting the verification token sequences into a large language model in parallel to obtain posterior probabilities of the verification token sequences and posterior reply tokens corresponding to the posterior probabilities; according to verification results of each prior probability and each posterior probability, using a posterior reply token to update the second part reply token sequence to obtain an updated second part reply token sequence; and all reply token sequences are output, so that the reasoning precision and efficiency are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular, relates to a method, device, electronic device and storage medium for reasoning about text data. Background Art

[0002] In recent years, with the rapid development and widespread application of artificial intelligence (AI) technology, large language models have gradually become part of our daily lives. Large language models usually have billions or more parameters. These models perform well in a variety of tasks and are powerful and diverse. For example, they can interact naturally and fluently according to the context of the conversation, achieve human-like chat communication, and complete tasks such as writing emails, creating video scripts, editing copy, translating text, and generating code. The emergence of these large language models marks a huge advancement in the field of deep learning, and also brings important opportunities for a wide range of application areas. They play a key role in fields such as natural language processing, artificial intelligence, and computer science, and provide effective tools for scientific research and commercial applications.

[0003] However, along with the potential of these models comes the problem of model reasoning performance. On the one hand, due to the large number of parameters of large language models, which can even reach more than 1 trillion, and the complex network structure, the model calculation volume reaches hundreds of billions of FLOPs (Floating Point Operations). On the other hand, since large language models use serial reasoning, that is, decoding one token at a time, the reasoning time of large language models may be very long, and efficient reasoning cannot be performed, making it unsuitable to apply large models to actual scenarios. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a data reasoning method, device, electronic device and storage medium based on token weighting, which overcomes the problem of low reasoning efficiency of large language models.

[0005] In a first aspect, the present application provides a method for reasoning about text data, the method comprising:

[0006] Get the prophet token sequence corresponding to the given text data entered by the user;

[0007] Inputting the prophet token sequence into a large language model, obtaining a first partial reply token sequence based on the reasoning of the large language model, and merging the first partial reply token sequence with the prophet token sequence to obtain an intermediate token sequence;

[0008] Input the intermediate token sequence into multiple small language models, and calculate each prior reply token of the second part reply token sequence and the prior probability of each prior reply token based on the candidate tokens inferred by each small language model and the probability sequence corresponding to each candidate token;

[0009] Inputting the verification token sequence composed of the intermediate token sequence and each of the prior response token sequences in sequence into the large language model in parallel, and obtaining the posterior probability of each of the verification token sequences and the posterior response token corresponding to each posterior probability based on the reasoning of the large language model;

[0010] Verifying each of the priori reply tokens according to each of the priori probabilities and each of the posterior probabilities, and updating the second partial reply token sequence using the posterior reply token according to the verification result to obtain an updated second partial reply token sequence;

[0011] Output a total reply token sequence obtained from the first partial reply token sequence and the updated second partial reply token sequence; wherein the first partial token sequence and the second partial reply token sequence are both partial sequences of the total reply token sequence corresponding to the given text data, and the total reply token sequence is used to generate the total reply data for the given text data.

[0012] According to the inference method of text data of the present application, after obtaining the prophet token sequence corresponding to the given text data input by the user, the first part of the reply token sequence is first inferred using the large language model to improve the accuracy of the initial reply token and pave the way for the next step of small and medium language model inference. The first part of the reply token sequence is then merged with the prophet token sequence to obtain an intermediate token sequence which is input into multiple small language models. The second part of the reply token sequence is obtained by reasoning with multiple small language models. The verification token sequence composed of the intermediate token sequence and the prior reply token is then input into the large language model in parallel. The large language model is used to verify each prior reply token output by the small language model, and the second part of the reply token sequence is updated according to the verification result. Finally, the entire reply token sequence used to generate the reply data is output, which not only ensures the inference accuracy of the text data, but also improves the inference efficiency, thereby greatly improving the user experience.

[0013] According to an embodiment of the present application, the intermediate token sequence is input into a plurality of small language models, and based on the candidate tokens inferred by each of the small language models and the probability sequence corresponding to each of the candidate tokens, each priori reply token of the second part of the reply token sequence and the prior probability of each of the priori reply tokens are calculated, including:

[0014] Inputting the intermediate token sequence into a plurality of the small language models, obtaining a plurality of the candidate tokens and probability sequences based on the reasoning of the small language models, each of the small language models corresponds to a probability sequence, and each probability in the probability sequence corresponds to each of the candidate tokens;

[0015] Performing weighted calculation on each of the candidate tokens according to the preset weighting factors of each of the small language models and the probability sequence to obtain the candidate token probability of each of the candidate tokens;

[0016] Determine the current prior reply token according to the candidate token probability, and use the candidate token probability corresponding to the current prior reply token as the prior probability;

[0017] The above two steps are repeated a set number of times to obtain each priori reply token of the second part reply token sequence.

[0018] When adding a small language model for reasoning, the present application performs weighted calculations through the candidate tokens output by multiple small language models, that is, integrates the reasoning results of each small language model, thereby overcoming the problem of low reasoning accuracy when using a single small language model, and further improving the reasoning accuracy.

[0019] According to an embodiment of the present application, the verification token sequence composed of the intermediate token sequence and each of the priori reply token sequences is inputted in parallel into the large language model, and the posterior probability of each of the verification token sequences and the posterior reply token corresponding to each posterior probability are obtained based on the reasoning of the large language model, including:

[0020] Obtain a priori reply token sequences obtained after each inference of the plurality of small language models, and sequentially combine the intermediate token sequence and each of the a priori reply token sequences to obtain a plurality of the verification token sequences, wherein the a priori reply token sequences are composed of current a priori reply tokens inferred by the plurality of small language models and their previous a priori reply tokens;

[0021] The verification token sequences are input into the large language model in parallel, and the posterior probabilities of the verification token sequences and the posterior response tokens corresponding to the posterior probabilities are obtained based on the reasoning of the large language model.

[0022] According to one embodiment of the present application, the verifying each of the priori reply tokens according to each of the priori probabilities and each of the posterior probabilities, and updating the second part of the reply token sequence using the posterior reply token according to the verification result to obtain the updated second part of the reply token sequence, includes:

[0023] Comparing the corresponding prior probabilities and posterior probabilities in turn;

[0024] When there is a target posterior probability whose prior probability is greater than or equal to the corresponding posterior probability, replacing the prior reply token at the corresponding position in the second part reply token sequence with the target posterior reply token corresponding to the target posterior probability, and deleting the prior reply token after the target posterior reply token to obtain an updated second part reply token sequence;

[0025] When each of the prior probabilities is less than each of the corresponding posterior probabilities, the posterior reply token corresponding to the last posterior probability is added to the end of the second partial reply token sequence to obtain an updated second partial reply token sequence.

[0026] After obtaining the second part reply token sequence output by multiple small language model inferences, the present application uses a large language model to verify each prior reply token of the second part reply token sequence, and uses the posterior tokens generated by the large language model to replace the deviated prior reply tokens. This can further improve the reasoning accuracy of the second part reply token sequence, that is, improve the accuracy of the output reply data, and improve the user experience.

[0027] According to one embodiment of the present application, the step of outputting a complete reply token sequence obtained from the first partial reply token sequence and the updated second partial reply token sequence includes:

[0028] Determine whether the first partial reply token sequence and the updated second partial reply token sequence are the complete reply token sequence after being combined. If so, output the complete reply token sequence. If not, return to the step of inputting the intermediate token sequence into multiple small language models, and calculating each prior reply token of the second partial reply token sequence and the prior probability of each prior reply token based on the candidate tokens inferred by each small language model and the probability sequence corresponding to each candidate token, and perform iterative reasoning until it is determined that the first partial reply token sequence and the updated second partial reply token sequence are combined to form the complete reply token sequence. In this way, a complete complete reply token sequence is obtained.

[0029] According to an embodiment of the present application, the determining that the first partial reply token sequence and the updated second partial reply token sequence are combined into the entire reply token sequence includes at least one of the following methods:

[0030] When the first part of the reply token sequence and the updated second part of the reply token sequence are combined to reach a preset length, determining that the first part of the reply token sequence and the updated second part of the reply token sequence are combined to form the entire reply token sequence;

[0031] When it is determined that the last token of the updated second partial reply token sequence is an end character, it is determined that the first partial reply token sequence and the updated second partial reply token sequence are combined to form the entire reply token sequence.

[0032] By determining whether the first part reply token sequence and the updated second part reply token sequence reach a preset length after being combined and whether the last token is an end character, it is determined whether the output token sequence is a complete reply token sequence, ensuring that complete reply data is finally output.

[0033] According to one embodiment of the present application, the step of inputting the prophet token sequence into a large language model and obtaining a first part of the reply token sequence based on the reasoning of the large language model includes:

[0034] Inputting the prophet token sequence into the large language model, iteratively reasoning the prophet token sequence a preset number of times based on the large language model, and obtaining the first partial reply token sequence;

[0035] Among them, the preset number of times is less than the total number of iterations, and the total number of iterations is the number of iterations in which the large language model infers the prophet token sequence to obtain the total reply token sequence.

[0036] The large language model is iterated a preset number of times to obtain the first part of the reply token sequence instead of using the large language model to obtain the entire reply token sequence at one time. On the one hand, it reduces the large amount of computation and long reasoning time caused by the large language model. On the other hand, it makes the first part of the reply token sequence more accurate, paving the way for improving the reasoning accuracy of the small language model.

[0037] In a second aspect, the present application provides a text data reasoning device, the device comprising:

[0038] An acquisition module, used to acquire a prophet token sequence corresponding to a given text data input by a user;

[0039] a first reasoning module, configured to input the prophet token sequence into a large language model, obtain a first partial reply token sequence based on reasoning of the large language model, and merge the first partial reply token sequence with the prophet token sequence to obtain an intermediate token sequence;

[0040] A second reasoning module is used to input the intermediate token sequence into multiple small language models, and calculate each prior reply token of the second partial reply token sequence and the prior probability of each prior reply token based on the candidate tokens obtained by each reasoning of the small language models and the probability sequence corresponding to each candidate token;

[0041] A third reasoning module is used to input the verification token sequence composed of the intermediate token sequence and each of the prior reply token sequences in sequence into the large language model in parallel, and obtain the posterior probability of each of the verification token sequences and the posterior reply token corresponding to each posterior probability based on the reasoning of the large language model;

[0042] A verification module, used to verify each of the prior reply tokens according to each of the prior probabilities and each of the posterior probabilities, and to update the second partial reply token sequence using the posterior reply token according to the verification result to obtain an updated second partial reply token sequence, and to output a combined reply token sequence of the first partial reply token sequence and the updated second partial reply token sequence;

[0043] An output module, configured to output a complete reply token sequence obtained from the first partial reply token sequence and the updated second partial reply token sequence;

[0044] The first partial token sequence and the second partial reply token sequence are both partial sequences of the entire reply token sequence corresponding to the given text data, and the entire reply token sequence is used to generate the entire reply data for the given text data.

[0045] According to the inference device for text data of the present application, after obtaining the prophet token sequence corresponding to the given text data input by the user, the first part of the reply token sequence is first inferred by using the large language model to improve the accuracy of the initial reply token and pave the way for the next step of small and medium language model inference. Then, the first part of the reply token sequence is merged with the prophet token sequence to obtain an intermediate token sequence which is input into multiple small language models. The second part of the reply token sequence is obtained by reasoning with multiple small language models. Then, the verification token sequence composed of the intermediate token sequence and the prior reply token is input into the large language model in parallel. The large language model is used to verify each prior reply token output by the small language model, and the second part of the reply token sequence is updated according to the verification result. Finally, the entire reply token sequence used to generate the reply data is output, which can not only ensure the inference accuracy of the text data, but also improve the inference efficiency, thereby greatly improving the user experience.

[0046] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for reasoning about text data as described in the first aspect above is implemented.

[0047] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the text data reasoning method as described in the first aspect above.

[0048] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the text data reasoning method as described in the first aspect.

[0049] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the text data reasoning method as described in the first aspect above.

[0050] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0052] Figure 1 This is one of the flowcharts of the text data reasoning method provided in the embodiment of the present application;

[0053] Figure 2 This is the second flow chart of the text data reasoning method provided in the embodiment of the present application;

[0054] Figure 3 This is the third flow chart of the text data reasoning method provided in the embodiment of the present application;

[0055] Figure 4 This is a fourth flow chart of the text data reasoning method provided in the embodiment of the present application;

[0056] Figure 5 It is a schematic diagram of the structure of the text data reasoning device provided in the embodiment of the present application;

[0057] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0059] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0060] In recent years, with the rapid development and widespread application of artificial intelligence (AI) technology, large language models have gradually become part of our daily lives. Large language models usually have billions or more parameters. These models perform well in a variety of tasks and are powerful and diverse. For example, they can interact naturally and fluently according to the context of the conversation, achieve human-like chat communication, and complete tasks such as writing emails, creating video scripts, editing copy, translating text, and generating code. The emergence of these large language models marks a huge advancement in the field of deep learning, and also brings important opportunities for a wide range of application areas. They play a key role in fields such as natural language processing, artificial intelligence, and computer science, and provide effective tools for scientific research and commercial applications.

[0061] However, due to the large number of parameters of the large language model, the amount of calculation is very large, and because the large language model uses a serial method for reasoning and decodes a token each time it is inferred, the reasoning time of the large language model is very long, which cannot be applied to the efficient response requirements in actual application scenarios. Patent 202311085639.X discloses a method, device, storage medium and electronic device for reasoning text data. The patent adopts a method of splicing multiple tokens into a sequence, and then uses a large language model to reason about the token sequence at the same time, and generates response data for multiple text data in parallel, but the patent does not optimize reasoning acceleration for a single text data. The paper "Fast Inference from Transformers via Speculative Decoding" proposes a method of guessing sampling for large language model reasoning. The paper uses a large language model and a small language model for reasoning acceleration, inputs a text data, generates tokens by the small language model, and then the large language model performs reasoning at regular intervals, but this method has two problems: first, the reasoning performance of the small language model itself is not high, which easily leads to the generation of many wrong tokens; second, the accuracy of the initial reasoning token result generated by the small language model is too low. Based on this, the present application proposes a method for reasoning text data, which can ensure the efficiency of reasoning while ensuring the accuracy of reasoning tokens.

[0062] Related fame and fortune explanation:

[0063] 1. Token: It is the smallest semantic unit, also known as participle, word unit, token;

[0064] 2. Tokenizer: a tool that converts natural language text information into tokens;

[0065] 3. Language model: It is a model used to process natural language. It can generate the next word or character based on given text information. Language model can be used for a variety of natural language processing tasks, such as machine translation, text generation, etc.

[0066] 4. Large language model: refers to a language model with a large number of parameters and a large amount of calculation.

[0067] 5. Small language model: refers to a language model with small parameters and small amount of calculation.

[0068] In conjunction with the accompanying drawings, the text data reasoning method, device, electronic device and storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0069] The text data reasoning method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0070] The text data reasoning method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the text data reasoning method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The text data reasoning method provided in the embodiment of the present application is described below using an electronic device as an example of the reasoning subject.

[0071] like Figure 1 As shown, the reasoning method for text data includes: step 110, step 120, step 130, step 140, and step 150.

[0072] Step 110, obtaining a prophet token sequence corresponding to the given text data input by the user.

[0073] In some implementations, step S100 specifically includes: obtaining given text data input by a user, and converting the given text data into a prophet token sequence using a sentence switching algorithm. The sentence switching algorithm may be built into a word segmentation tool, and the prophet token sequence of the given text data may be obtained using the word segmentation tool.

[0074] It can be understood that the above-mentioned prophet token sequence is a sequence of prophet tokens after the given text data input by the user is converted into prophet tokens. Among them, the prophet token can be understood as a token that can be obtained in advance from the given text data input by the user, and is a token that can be known in advance without inputting a language model.

[0075] Step S120, input the prophet token sequence into the large language model, obtain a first partial reply token sequence based on the reasoning of the large language model, and merge the first partial reply token sequence with the prophet token sequence to obtain an intermediate token sequence.

[0076] In view of the fact that the accuracy of the initial inference token generated by the small language model is too low, the large language model is first used to infer the initial reply token (i.e., the first part of the reply token sequence in this application), thereby improving the accuracy of the initial reply token. For the subsequent reasoning of the small language model, it can also improve the reasoning accuracy to a certain extent.

[0077] In some embodiments, step S120 specifically includes: inputting the prophet token sequence into the large language model, iteratively reasoning the prophet token sequence a preset number of times based on the large language model, and obtaining the first partial reply token sequence, wherein the preset number of times is less than the total number of iterations, and the total number of iterations is the number of iterations in which the large language model reasoned the prophet token sequence to obtain the total reply token sequence.

[0078] Since the result of the large language model inference is more accurate, but the large language model can only decode one token each time, in order to obtain more accurate reply data, the large language model is first used to iteratively infer the prophet token sequence for a preset number of times M to obtain M first reply tokens with high precision, and then the M first reply tokens are sorted in sequence according to the output order to obtain the first part of the reply token sequence [t1, t2, L, t i ,L,t M ], where t i Represents the first reply token obtained by the large language model for the i-th inference.

[0079] It should be noted that the preset number M is less than the total number of iterations A. Preferably, the preset number M is less than half, 1 / 3, 1 / 4, 1 / 5, 1 / 10 or less of the total number of iterations A. This can not only effectively reduce the number of inferences of the large language model, reduce the problem of long time caused by the large language model inference, but also improve the inference accuracy of the initial reply token. It is understandable that if the large language model needs 30,000 times to infer the entire reply token sequence of the prophet token sequence and each inference takes 0.001S, then it takes 30s, then the inference 1 / 10 times only takes 3s, which greatly reduces the time consumption of the large language model inference.

[0080] Step S130, input the intermediate token sequence into multiple small language models, and calculate each prior response token of the second part response token sequence and the prior probability of each prior response token based on the candidate tokens obtained by each small language model inference and the probability sequence corresponding to each candidate token.

[0081] After merging the prophet token sequence with the first part of the reply token sequence inferred by the large language model, an intermediate token sequence is obtained, and the intermediate token sequence is input into multiple small language models. The reason why the intermediate token sequence is input into the small language model instead of the prophet token sequence is that the inference accuracy of the small language model is low. On the basis of the partial token sequence inferred by the large language model, the inference accuracy of the small language model can be improved. Based on the candidate tokens inferred by each small language model each time and the probability sequence corresponding to each candidate token, each prior reply token of the second part of the reply token sequence and the prior probability of each prior reply token are calculated. It should be noted here that the small language model can only infer one token each time, so it is necessary to iterate and set the number of times N to obtain N prior reply tokens, and then sort the N prior reply tokens in sequence according to the output order to obtain the second part of the reply token sequence [u1,u2,L,u j ,L,u N ], where u jIndicates that multiple small language models perform the jth inference to obtain a priori reply token. At the same time, the small language model also outputs the prior probability R(x) = [r1(x), r2(x), L, r j (x),L,r N (x)], where r j (x) is the prior reply token u j The corresponding prior probability.

[0082] In some embodiments, Figure 2 As shown, step S130 specifically includes: S131, inputting the intermediate token sequence into multiple small language models, and obtaining multiple candidate tokens and probability sequences based on the reasoning of each small language model, each small language model corresponds to one probability sequence, and each probability in the probability sequence corresponds to each candidate token.

[0083] S132, performing weighted calculation on each of the candidate tokens according to the preset weighting factors of each of the small language models and the probability sequence to obtain the candidate token probability of each of the candidate tokens.

[0084] S133, determining a current a priori reply token according to the candidate token probability, and taking the candidate token probability corresponding to the current a priori reply token as the a priori probability.

[0085] S134, repeat the above steps S132 and S133 a set number of times to obtain each priori reply token of the second part reply token sequence.

[0086] Among them, step S133 can further include: comparing the probabilities of the candidate tokens to obtain the maximum candidate token probability, and using the candidate token corresponding to the maximum candidate token probability as the current prior response token.

[0087] The following is a detailed explanation using two small language models as examples:

[0088] The preset weighting factor of the first small language model is set to w1, and the preset weighting factor of the second small language model is set to w2. In the jth inference, the candidate tokens inferred by the first small language model and the second small language model are a, b, c, d, and e respectively. The probabilities of the candidate tokens inferred by the first small language model form a first probability sequence p(x). Assume that p(x) = [0.1, 0.2, 0.3, 0.4, 0]. The probabilities of the candidate tokens inferred by the second small language model form a second probability sequence q(x). Assume that q(x) = [0, 0.1, 0.2, 0.3, 0.4]. Assume that w1 = 0.4 and w2 = 0.6. According to the formula r(x) = w1*p(x) + w2*q(x), we get r j(x) = [0.4, 1.4, 2.4, 3.4, 2.4], which means that the candidate token probabilities of candidate tokens a, b, c, d, e are 0.4, 1.4, 2.4, 3.4 and 2.4 respectively. By comparing the probabilities of each candidate token, it can be seen that the probability of candidate token d is the largest. Therefore, candidate token d is used as the prior response token output by the first small language model and the second small language model during the j-th inference, and 3.4 is used as the prior probability of the prior response token d.

[0089] It should be noted that the number of small language models is not limited to the above 2, and can be 3, 4, 5..., but the more the better, too many will lead to redundant calculations. Preferably, there are 2 small language models, which can be the LLaMA-68M model with 68 million parameters and the LLaMA-160M ​​model with 160 million parameters, and the large language model can be the LLaMA-30B model with 30 billion parameters. Preferably, the parameter amount of the first small language model in the above specific embodiment (the corresponding weighting factor is small) is smaller than the parameter amount of the second small language model (the corresponding weighting factor is large), thereby further improving the reasoning accuracy.

[0090] It is worth noting that the large language model and the small language model in the above embodiments are trained using a unified architecture and the same training data, so the types of candidate tokens outputted are basically consistent.

[0091] When adding a small language model for reasoning, the present application performs weighted calculations through the candidate tokens output by multiple small language models, that is, integrates the reasoning results of each small language model, thereby overcoming the problem of low reasoning accuracy when using a single small language model, and further improving the reasoning accuracy.

[0092] Step S140, inputting the verification token sequence composed of the intermediate token sequence and each of the prior response tokens in sequence into the large language model in parallel, and obtaining the posterior probability of each of the verification token sequences and the posterior response token corresponding to each posterior probability based on the reasoning of the large language model.

[0093] In order to further determine the accuracy of the prior response tokens output by each small language model, the large language model will also be used to verify it. In some embodiments, step S140 specifically includes: obtaining a prior response token sequence obtained after each reasoning of multiple small language models, the prior response token sequence is composed of the current prior response tokens and their previous prior response tokens inferred by multiple small language models, combining the intermediate token sequence and each prior response token sequence in sequence to obtain a plurality of verification token sequences, inputting the verification token sequences in parallel into the large language model, and obtaining the posterior probability of each verification token sequence and the posterior response token corresponding to each posterior probability based on the reasoning of the large language model.

[0094] For ease of understanding, the following is an explanation. When multiple small language models are not inferred, there is no prior reply token, then the verification token sequence is the intermediate token sequence [V]. The token output after the first inference is the prior reply token u1, so the corresponding prior reply token sequence is [u1]. After combining it with the intermediate token sequence V, the verification token sequence after the first inference is [V,u1]. Similarly, the prior reply token sequence obtained after the jth inference is [u1,u2,u3,....u j ], and then get the verification token sequence [V,u1,u2,u3,....u j ], and so on. After N inferences, we get multiple verification token sequences [V], [V,u1], L, [V,u1,u2, L,u i ],L,[V,u1,u2,L,u N ], the multiple verification token sequences are input into the large language model in parallel, and the large language model can obtain the posterior probability S of each verification token sequence by only one inference. j (x) and the a posteriori reply token u' j , the posterior probability sequence composed of each posterior probability is S(x) = [s1(x), s2(x), L, s j (x),L,s N (x),s N+1 (x)], the a posteriori reply token sequence composed of each a posteriori reply token is [u'1,u'2,...u' j ,....,u' N+1 ].

[0095] It is worth noting that although a large language model is used to infer multiple verification token sequences in this step, due to the parallel input (i.e., one-time inference), the amount of computation used by the large language model is only one-time, which has little impact on the timeliness of inference.

[0096] Step S150, verifying each of the priori reply tokens according to each of the priori probabilities and each of the posterior probabilities, and updating the second partial reply token sequence using the posterior reply token according to the verification result to obtain an updated second partial reply token sequence.

[0097] In some embodiments, Figure 3As shown, step S150 specifically includes: S151, comparing the corresponding prior probabilities and the corresponding posterior probabilities in turn. S152, when there is a target posterior probability whose prior probability is greater than or equal to the corresponding posterior probability, replacing the prior response token at the corresponding position in the second part response token sequence with the target posterior response token corresponding to the target posterior probability, and deleting the prior response token after the target posterior response token to obtain an updated second part response token sequence. S153, when each of the prior probabilities is less than the corresponding posterior probabilities, adding the posterior response token corresponding to the last posterior probability to the end of the second part response token sequence to obtain an updated second part response token sequence.

[0098] This step is to get the token sequence [u1,u2,L,u in the second part of the inference output of multiple small language models. j ,L,u N ] and its prior probability sequence R(x)=[r1(x),r2(x),L,r j (x),L,r N (x)], and the posterior probability sequence S(x) = [s1(x), s2(x), L, s j (x),L,s N (x),s N+1 (x)] and the posterior reply token sequence is [u'1,u'2,...u' j ,....,u' N+1 ]After that, determine how to use the posterior reply token to update the related prior reply tokens in the second part reply token sequence according to each prior probability and each posterior probability. There are two cases. The first is that the prior probability is greater than or equal to the posterior probability. In this case, the target posterior reply token corresponding to the target posterior probability needs to replace the prior reply token at the corresponding position in the second part reply token sequence, and delete the prior reply token after the target posterior reply token to obtain the updated second part reply token sequence. For example, if r j (x)>s j (x), then the updated second part reply token sequence is [V,u1,u2,u3,....u j-1 ,u' j]; The reason for doing this is: theoretically, the probability of the token inferred by the small language model should be smaller than the probability of the token inferred by the large language model. If the prior probability is greater than or equal to the posterior probability, it proves that the reasoning of multiple small language models has produced new deviations, and the token needs to be corrected. The second is the case where the prior probabilities are all less than the posterior probabilities. At this time, the posterior reply token corresponding to the last posterior probability is added to the end of the second part reply token sequence to obtain the updated second part reply token sequence, that is, the updated second part reply token sequence is [V,u1,u2,u3,....u N ,u' N+1 ], and finally the first part of the reply token sequence and the updated second part of the reply token sequence are used to output the entire reply token sequence, which can further improve the reasoning accuracy of the second part of the reply token sequence, that is, improve the accuracy of the output reply data and improve the user experience.

[0099] Step S160: outputting a complete reply token sequence obtained from the first partial reply token sequence and the updated second partial reply token sequence.

[0100] It should be noted that the first partial token sequence and the second partial reply token sequence are both partial sequences of the entire reply token sequence corresponding to the given text data, and the entire reply token sequence is used to generate the entire reply data of the given text data.

[0101] In some embodiments, Figure 4 As shown, step S160 specifically includes: S161, judging whether the first part reply token sequence and the updated second part reply token sequence are the whole reply token sequence after merging, if so, outputting the whole reply token sequence, if not, merging the updated second part reply token sequence with the intermediate token sequence to obtain a new intermediate token sequence, and returning to steps S130-S150 until it is determined that the first part reply token sequence and the updated second part reply token sequence are the whole reply token sequence after merging. In this way, complete reply data can be obtained, further improving user experience.

[0102] The text data reasoning method provided in the embodiment of the present application, after obtaining the prophet token sequence corresponding to the given text data input by the user, first uses the large language model to infer the first part of the reply token sequence, improves the accuracy of the initial reply token, and paves the way for the next step of small and medium language model reasoning, then merges the first part of the reply token sequence with the prophet token sequence to obtain an intermediate token sequence, which is input into multiple small language models, and uses the reasoning of multiple small language models to obtain the second part of the reply token sequence, and then inputs the verification token sequence composed of the intermediate token sequence and the prior reply token into the large language model in parallel, uses the large language model to verify each prior reply token output by the small language model, and updates the second part of the reply token sequence according to the verification result, and finally outputs the entire reply token sequence used to generate the reply data, which can not only ensure the reasoning accuracy of the text data, but also improve the reasoning efficiency, thereby greatly improving the user experience.

[0103] In some embodiments, step S161 includes at least one of the following methods: when the first partial reply token sequence and the updated second partial reply token sequence reach a preset length after being combined, determining that the first partial reply token sequence and the updated second partial reply token sequence are combined to form the entire reply token sequence. When it is determined that the last token of the updated second partial reply token sequence is an end character, determining that the first partial reply token sequence and the updated second partial reply token sequence are combined to form the entire reply token sequence.

[0104] When the last token of the updated second part reply token sequence is the end character, it is proved that the first part reply token sequence and the updated second part reply token sequence are combined to form the whole reply token sequence, and the whole reply token sequence is output, so that the complete reply data can be obtained. In another embodiment, since the reply data may be predictable, that is, the preset length of the final reply data is known in advance, it is also possible to determine whether the first part reply token sequence and the updated second part reply token sequence are combined to form the whole reply token sequence according to the preset length. Of course, it can also be a combination of the two. When it is determined that the first part reply token sequence and the updated second part reply token sequence reach the preset length after being combined and the last token of the updated second part reply token sequence is the end character, it is determined that the first part reply token sequence and the updated second part reply token sequence are combined to form the whole reply token sequence, so as to fully ensure the integrity of the reply data.

[0105] The text data reasoning method provided in the embodiment of the present application can be executed by a text data reasoning device. In the embodiment of the present application, the text data reasoning device provided in the embodiment of the present application is described by taking the text data reasoning method executed by the text data reasoning device as an example.

[0106] The present application also provides a text data reasoning device, such as Figure 5 As shown, the text data reasoning device includes: an acquisition module 110, which is used to acquire a prophet token sequence corresponding to given text data input by a user;

[0107] The first reasoning module 120 is used to input the prophet token sequence into the large language model, obtain a first partial reply token sequence based on the reasoning of the large language model, and merge the first partial reply token sequence with the prophet token sequence to obtain an intermediate token sequence.

[0108] The second reasoning module 130 is used to input the intermediate token sequence into multiple small language models, and calculate each prior response token of the second part response token sequence and the prior probability of each prior response token based on the candidate tokens obtained by each reasoning of the small language models and the probability sequence corresponding to each candidate token.

[0109] The third reasoning module 140 is used to input the verification token sequence composed of the intermediate token sequence and each of the prior response token sequences in parallel into the large language model, and obtain the posterior probability of each of the verification token sequences and the posterior response token corresponding to each posterior probability based on the reasoning of the large language model.

[0110] The verification module 150 is used to verify each of the priori reply tokens according to each of the priori probabilities and each of the posterior probabilities, and update the second partial reply token sequence using the posterior reply token according to the verification result to obtain an updated second partial reply token sequence.

[0111] The output module 160 is configured to output a complete reply token sequence obtained from the first partial reply token sequence and the updated second partial reply token sequence.

[0112] The first partial token sequence and the second partial reply token sequence are both partial sequences of the entire reply token sequence corresponding to the given text data, and the entire reply token sequence is used to generate the entire reply data for the given text data.

[0113] According to the inference device for text data provided by the embodiment of the present application, after obtaining the prophet token sequence corresponding to the given text data input by the user, the first part of the reply token sequence is first inferred by using the large language model to improve the accuracy of the initial reply token and pave the way for the next step of small and medium language model inference. Then, the first part of the reply token sequence is merged with the prophet token sequence to obtain an intermediate token sequence which is input into multiple small language models. The second part of the reply token sequence is obtained by reasoning with multiple small language models. Then, the verification token sequence composed of the intermediate token sequence and the prior reply token is input into the large language model in parallel. The large language model is used to verify each prior reply token output by the small language model, and the second part of the reply token sequence is updated according to the verification result. Finally, the entire reply token sequence used to generate the reply data is output, which can not only ensure the inference accuracy of the text data, but also improve the inference efficiency, thereby greatly improving the user experience.

[0114] In some embodiments, the second reasoning module 130 is specifically used to input the intermediate token sequence into multiple small language models, obtain multiple candidate tokens and probability sequences based on the reasoning of each small language model, each small language model corresponds to a probability sequence, and each probability in the probability sequence corresponds to each candidate token; weighted calculation is performed on each candidate token according to the preset weighting factor of each small language model and the probability sequence to obtain the candidate token probability of each candidate token; the current prior response token is determined according to the candidate token probability, and the candidate token probability corresponding to the current prior response token is used as the prior probability; the above steps are repeated a set number of times to obtain each prior response token of the second partial response token sequence.

[0115] In some embodiments, the third reasoning module 140 is specifically used to obtain the prior response token sequence obtained after each reasoning of the multiple small language models, and to obtain the multiple verification token sequences by combining the intermediate token sequence and each prior response token sequence in sequence, wherein the prior response token sequence is composed of the current prior response token and its previous prior response token inferred by the multiple small language models; the verification token sequence is input into the large language model in parallel, and the posterior probability of each verification token sequence and the posterior response token corresponding to each posterior probability are obtained based on the reasoning of the large language model.

[0116] In some embodiments, the verification module 150 is specifically used to compare the corresponding prior probabilities and the posterior probabilities in sequence; when there is a target posterior probability whose prior probability is greater than or equal to the corresponding posterior probability, the target posterior reply token corresponding to the target posterior probability replaces the prior reply token at the corresponding position in the second part reply token sequence, and deletes the prior reply token after the target posterior reply token to obtain an updated second part reply token sequence; when each of the prior probabilities is less than the corresponding posterior probabilities, the posterior reply token corresponding to the last posterior probability is added to the end of the second part reply token sequence to obtain an updated second part reply token sequence.

[0117] In some embodiments, the output module 160 is specifically used to determine whether the first partial reply token sequence and the updated second partial reply token sequence are a complete reply token sequence after being merged. If so, the complete reply token sequence is output; if not, the step of inputting the intermediate token sequence into multiple small language models, calculating each prior reply token of the second partial reply token sequence and the prior probability of each prior reply token based on the candidate tokens inferred each time by each small language model and the probability sequence corresponding to each candidate token, performs iterative reasoning until it is determined that the first partial reply token sequence and the updated second partial reply token sequence are merged to form the complete reply token sequence.

[0118] In some embodiments, the determination that the first partial reply token sequence and the updated second partial reply token sequence are combined to form the entire reply token sequence is implemented in at least one of the following ways: when the first partial reply token sequence and the updated second partial reply token sequence reach a preset length after being combined, the determination that the first partial reply token sequence and the updated second partial reply token sequence are combined to form the entire reply token sequence. When it is determined that the end token of the updated second partial reply token sequence is an end character, the determination that the first partial reply token sequence and the updated second partial reply token sequence are combined to form the entire reply token sequence.

[0119] In some embodiments, the first reasoning module 120 is specifically used to input the prophet token sequence into the large language model, iteratively reason the prophet token sequence a preset number of times based on the large language model, and obtain the first partial reply token sequence; wherein the preset number of times is less than the total number of iterations, and the total number of iterations is the number of iterations in which the large language model reasoned the prophet token sequence to obtain the total reply token sequence.

[0120] The text data reasoning device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0121] The text data reasoning device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0122] The text data reasoning device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0123] In some embodiments, Figure 6 As shown, an embodiment of the present application also provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the program is executed by the processor 801, each process of the above-mentioned text data reasoning method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0124] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0125] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned text data reasoning method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0126] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0127] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned text data reasoning method when executed by a processor.

[0128] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0129] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned text data reasoning method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0130] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0131] It should be noted that, in this article, the term "comprises", "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0133] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0134] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0135] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for reasoning about text data, characterized in that: include: Get the prophet token sequence corresponding to the given text data entered by the user; Inputting the prophet token sequence into a large language model, obtaining a first partial reply token sequence based on the reasoning of the large language model, and merging the first partial reply token sequence with the prophet token sequence to obtain an intermediate token sequence; Input the intermediate token sequence into multiple small language models, and calculate each prior reply token of the second part reply token sequence and the prior probability of each prior reply token based on the candidate tokens inferred by each small language model and the probability sequence corresponding to each candidate token; Inputting the verification token sequence composed of the intermediate token sequence and each of the prior response token sequences in sequence into the large language model in parallel, and obtaining the posterior probability of each of the verification token sequences and the posterior response token corresponding to each posterior probability based on the reasoning of the large language model; Verifying each of the priori reply tokens according to each of the priori probabilities and each of the posterior probabilities, and updating the second partial reply token sequence using the posterior reply token according to the verification result to obtain an updated second partial reply token sequence; Output a total reply token sequence obtained from the first partial reply token sequence and the updated second partial reply token sequence; wherein the first partial token sequence and the second partial reply token sequence are both partial sequences of the total reply token sequence corresponding to the given text data, and the total reply token sequence is used to generate the total reply data for the given text data.

2. The method according to claim 1, characterized in that The step of inputting the intermediate token sequence into a plurality of small language models, and calculating each priori reply token of the second part reply token sequence and the priori probability of each priori reply token based on the candidate tokens inferred by each of the small language models and the probability sequence corresponding to each candidate token, comprises: Inputting the intermediate token sequence into a plurality of the small language models, obtaining a plurality of the candidate tokens and probability sequences based on the reasoning of the small language models, each of the small language models corresponds to a probability sequence, and each probability in the probability sequence corresponds to each of the candidate tokens; Performing weighted calculation on each of the candidate tokens according to the preset weighting factors of each of the small language models and the probability sequence to obtain the candidate token probability of each of the candidate tokens; Determine the current prior reply token according to the candidate token probability, and use the candidate token probability corresponding to the current prior reply token as the prior probability; Return to the step of inputting the intermediate token sequence into multiple small language models, and obtaining multiple candidate tokens and probability sequences based on the reasoning of each small language model, each small language model corresponds to one probability sequence, and each probability in the probability sequence corresponds to each candidate token, and the step is repeated a set number of times to obtain each prior response token of the second partial response token sequence.

3. The method according to claim 1, characterized in that The step of inputting the verification token sequence composed of the intermediate token sequence and each of the priori reply token sequences in sequence into the large language model in parallel, and obtaining the posterior probability of each of the verification token sequences and the posterior reply token corresponding to each posterior probability based on the reasoning of the large language model, comprises: Obtain a priori reply token sequences obtained after each inference of the plurality of small language models, and sequentially combine the intermediate token sequence and each of the a priori reply token sequences to obtain a plurality of the verification token sequences, wherein the a priori reply token sequences are composed of current a priori reply tokens inferred by the plurality of small language models and their previous a priori reply tokens; The verification token sequences are input into the large language model in parallel, and the posterior probabilities of the verification token sequences and the posterior response tokens corresponding to the posterior probabilities are obtained based on the reasoning of the large language model.

4. The method according to claim 1, characterized in that: The method of verifying each of the priori reply tokens according to each of the priori probabilities and each of the posterior probabilities, and updating the second partial reply token sequence using the posterior reply token according to the verification result to obtain an updated second partial reply token sequence, comprises: Comparing the corresponding prior probabilities and posterior probabilities in turn; When there is a target posterior probability whose prior probability is greater than or equal to the corresponding posterior probability, replacing the prior reply token at the corresponding position in the second part reply token sequence with the target posterior reply token corresponding to the target posterior probability, and deleting the prior reply token after the target posterior reply token to obtain an updated second part reply token sequence; When each of the prior probabilities is less than each of the corresponding posterior probabilities, the posterior reply token corresponding to the last posterior probability is added to the end of the second partial reply token sequence to obtain an updated second partial reply token sequence.

5. The method according to any one of claims 1 to 4, characterized in that: The step of outputting a complete reply token sequence obtained from the first partial reply token sequence and the updated second partial reply token sequence comprises: Determine whether the first partial reply token sequence and the updated second partial reply token sequence are the complete reply token sequence after being combined. If so, output the complete reply token sequence. If not, return to the step of inputting the intermediate token sequence into multiple small language models, and calculating each prior reply token of the second partial reply token sequence and the prior probability of each prior reply token based on the candidate tokens inferred each time by each small language model and the probability sequence corresponding to each candidate token, and perform iterative reasoning until it is determined that the first partial reply token sequence and the updated second partial reply token sequence are combined to form the complete reply token sequence.

6. The method according to claim 5, characterized in that The determining that the first partial reply token sequence and the updated second partial reply token sequence are combined into the entire reply token sequence includes at least one of the following methods: When the first part of the reply token sequence and the updated second part of the reply token sequence are combined to reach a preset length, determining that the first part of the reply token sequence and the updated second part of the reply token sequence are combined to form the entire reply token sequence; When it is determined that the last token of the updated second partial reply token sequence is an end character, it is determined that the first partial reply token sequence and the updated second partial reply token sequence are combined to form the entire reply token sequence.

7. The method according to any one of claims 1 to 4, characterized in that: The step of inputting the prophet token sequence into a large language model and obtaining a first part of the reply token sequence based on the reasoning of the large language model comprises: Inputting the prophet token sequence into the large language model, iteratively reasoning the prophet token sequence a preset number of times based on the large language model, and obtaining the first partial reply token sequence; Among them, the preset number of times is less than the total number of iterations, and the total number of iterations is the number of iterations in which the large language model infers the prophet token sequence to obtain the total reply token sequence.

8. A text data reasoning device, characterized in that: The device comprises: An acquisition module, used to acquire a prophet token sequence corresponding to a given text data input by a user; a first reasoning module, configured to input the prophet token sequence into a large language model, obtain a first partial reply token sequence based on reasoning of the large language model, and merge the first partial reply token sequence with the prophet token sequence to obtain an intermediate token sequence; A second reasoning module is used to input the intermediate token sequence into multiple small language models, and calculate each prior reply token of the second partial reply token sequence and the prior probability of each prior reply token based on the candidate tokens obtained by each reasoning of the small language models and the probability sequence corresponding to each candidate token; A third reasoning module is used to input the verification token sequence composed of the intermediate token sequence and each of the prior reply token sequences in sequence into the large language model in parallel, and obtain the posterior probability of each of the verification token sequences and the posterior reply token corresponding to each posterior probability based on the reasoning of the large language model; A verification module, used to verify each of the prior reply tokens according to each of the prior probabilities and each of the posterior probabilities, and to update the second partial reply token sequence using the posterior reply token according to the verification result to obtain an updated second partial reply token sequence, and to output a combined reply token sequence of the first partial reply token sequence and the updated second partial reply token sequence; An output module, configured to output a complete reply token sequence obtained from the first partial reply token sequence and the updated second partial reply token sequence; The first partial token sequence and the second partial reply token sequence are both partial sequences of the entire reply token sequence corresponding to the given text data, and the entire reply token sequence is used to generate the entire reply data for the given text data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for inferring text data as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for inferring text data as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Data processing method and related equipment

    CN116541492A

  • Text data reasoning method and device, storage medium and electronic equipment

    CN116822632A

  • Large language model reasoning acceleration method and related device

    CN118333172A

  • Methods and systems for determining characteristics of a dialog between a computer and a user

    US20210375277A1

Cited By

  • Scientific and technological achievement intelligent pushing and public service question and answer method, device and equipment

    CN121882243A