Text generation method and electronic device

Abstract

This application provides a text generation method and an electronic device, relating to the field of terminal technology. The method includes: inputting first feature information, a first instruction, a first lexical unit, and first text into a first model, causing the first model to infer from the first lexical unit to obtain a plurality of second lexical units output by the first model. For any one of the plurality of second lexical units, a first acceptance probability is determined based on a first prediction probability corresponding to the second lexical unit. The first acceptance probability corresponding to the second lexical unit is compared with a first threshold to determine whether to perform further inference for the second lexical unit. Thus, by dynamically adjusting the lexical unit inference process based on the prediction probability corresponding to each predicted lexical unit, the method determines whether to perform further inference for that predicted lexical unit, thereby improving the efficiency of text generation.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a text generation method and an electronic device. Background Technology

[0002] In the information society, various forms of text data are constantly emerging, and how to extract valuable core content from them has become an urgent problem to be solved.

[0003] Currently, some algorithms can extract text information by building models. These models learn from large amounts of training data and can gain a deep understanding of the context, thereby accurately generating key information based on user input. However, this process is computationally intensive and relatively inefficient. Summary of the Invention

[0004] This application provides a text generation method and an electronic device, applicable to the field of terminal technology, to improve the efficiency of text generation.

[0005] Firstly, embodiments of this application propose a text generation method. The method includes:

[0006] The first feature information, the first instruction, the first word element, and the first text are input into the first model so that the first model can infer the first word element and obtain multiple second word elements output by the first model. The multiple second word elements correspond to their respective first prediction probabilities. The first prediction probabilities are used to indicate the probability that the second word element is the next word of the first word element. The first feature information is used to reflect the feature information of the first text, the feature information of the first instruction, and the feature information of at least one historical word element obtained by the first model inference. The first instruction is used to instruct the extraction of the first type of information from the first text.

[0007] For any one of the multiple second word elements, a first acceptance probability is determined based on the first prediction probability corresponding to the second word element. The first acceptance probability is used to indicate the probability that the second word element is accepted by the second model. The second model is used to verify the word elements output by the first model.

[0008] The first acceptance probability corresponding to the second word is compared with the first threshold to determine whether to perform inference for the second word.

[0009] In this embodiment, compared to the word-by-word (lexical) reasoning process of the second model, the first model first performs multiple inferences on a lexical to obtain multiple possible subsequent lexicals. These lexicals are then input into the second model for verification. This approach allows the second model to verify multiple lexicals at once, thereby enabling it to determine multiple subsequent lexicals simultaneously and improving the overall efficiency of the reasoning. Based on this, this application calculates a first acceptance probability for each lexical and compares it with a set first threshold to determine whether further reasoning is needed for that lexical. This method effectively selects appropriate lexicals for reasoning, thus improving the flexibility of lexical reasoning.

[0010] Meanwhile, by promptly filtering and discarding inferred words, the efficiency of the first model during the inference process can be improved. Specifically, when it is no longer suitable to continue inference, timely stopping the execution of subsequent inference can avoid unnecessary inference processes, thereby saving time and improving efficiency. When inference can still continue, the subsequent inference process can continue, thus outputting as much longer inference text as possible during the inference process, avoiding the time delay caused by having to start inference again from a word where the previous inference stopped.

[0011] In one possible implementation, determining the first acceptance probability of the second word element based on the first predicted probability includes:

[0012] Based on the first predicted probability corresponding to the second word, the first confidence level corresponding to the second word is determined, and the first confidence level is proportional to the first predicted probability.

[0013] Based on the first confidence level corresponding to the second term and the preset mapping relationship, the first acceptance probability corresponding to the second term is determined. The preset mapping relationship is used to indicate the correspondence between the confidence level and the acceptance probability.

[0014] In this implementation, the concept of a first confidence level allows for a more refined evaluation of the quality of the second lexical unit. Furthermore, a mapping relationship is established between the first confidence level of a lexical unit and its acceptance by the second model. Using this mapping, the probability of a lexical unit being accepted by the second model can be quantitatively determined based on its corresponding first confidence level. This method effectively filters out high-quality lexical units for further inference. This not only improves lexical unit utilization but also ensures the accuracy and efficiency of the inference process.

[0015] In one possible implementation, a first acceptance probability corresponding to the second word element is compared with a first threshold to determine whether to perform inference for the second word element, including:

[0016] Based on the first delay and the second delay, a first threshold is determined, where the first delay is the time required for one inference by the first model, and the second delay is the time required for one inference by the second model.

[0017] The first acceptance probability corresponding to the second word is compared with the first threshold to determine whether to perform inference for the second word.

[0018] In this implementation, the first threshold is dynamically determined based on the inference latency of the first and second models, allowing the system to adjust flexibly under different computational resource and time constraints. This dynamic adjustment helps optimize system performance, ensuring the best inference results with limited resources. Furthermore, by comparing the first acceptance probability with the first threshold, further inference is only performed on those terms with higher probability, reducing unnecessary computation and improving overall inference efficiency. This, in turn, reduces overall system latency and increases response speed.

[0019] In one possible implementation, determining the first threshold based on the first delay and the second delay includes:

[0020] Determine the difference between the first delay and the second delay, and determine the first threshold based on the difference and the first delay;

[0021] The first threshold is directly proportional to the first time delay and inversely proportional to the difference.

[0022] In this implementation, the cost duration and benefit duration of lexical reasoning are considered based on the first delay and the second delay, respectively. Furthermore, by comparing the relationship between cost duration and benefit duration, a reasonable first threshold can be effectively determined.

[0023] In one possible implementation, the method further includes:

[0024] If the first acceptance probability is greater than the first threshold, the second feature information, the first instruction and the second word are input into the first model so that the first model can reason about the second word and obtain multiple third words output by the first model. The multiple third words correspond to their respective second prediction probabilities. The second prediction probabilities are used to indicate the probability that the third word is the next word of the second word.

[0025] The second feature information is used to reflect the feature information of the first text, the feature information of the first instruction, and the feature information of at least one historical word element obtained by reasoning from the first model and the second model. The first instruction is used to instruct the extraction of a first type of information from the first text.

[0026] In this implementation, subsequent processing is only performed if the initial acceptance probability is greater than a first threshold. This selective inference mechanism reduces unnecessary computational overhead. The model only continues inferring the next word when it has sufficient confidence in the second word, thereby improving overall processing efficiency.

[0027] In one possible implementation, the first model infers about the first lexical unit, including:

[0028] The first model infers the first word element based on the target vocabulary;

[0029] The method also includes:

[0030] The first decoding speed is determined based on the first delay, the second delay, the average acceptance length, and the first length. The first length is the number of inferences performed by the first model, and the average acceptance length is the average length of the word tokens output by the first model that are accepted by the second model. The first delay, the second delay, and the first length are all inversely proportional to the first decoding speed, and the average acceptance length is directly proportional to the first decoding speed.

[0031] The first vocabulary is truncated based on the first decoding speed and the first dataset to obtain the target vocabulary, which contains multiple texts in the first dataset.

[0032] In this implementation, the first decoding speed is determined by comprehensively considering the first latency, the second latency, the average reception length, and the first length. This method can accurately evaluate the efficiency of the current decoding process. Based on this evaluation, the first vocabulary is pruned to remove words that do not contribute much to decoding, thereby improving the overall decoding efficiency. During decoding, the vocabulary is pruned based on the first dataset (containing multiple texts). This means that the target vocabulary can be optimized for specific datasets or tasks, which helps improve the model's adaptability when processing specific types of text. The pruned target vocabulary is usually smaller than the original first vocabulary, which means that the number of words that need to be loaded and processed during decoding is reduced. This helps reduce memory usage, improve system resource utilization, and enhance the model's inference efficiency.

[0033] In one possible implementation, determining the first decoding speed based on the first delay, the second delay, the average reception length, and the first length includes:

[0034] The first product is calculated based on the first length and the first time delay;

[0035] The first value is determined by summing the first product and the second time delay;

[0036] The first decoding speed is determined based on the ratio of the average received length to the first value.

[0037] In this implementation, the decoding speed is determined by comprehensively considering the first delay, the second delay, the average reception length, and the first length, which allows for a relatively effective evaluation of the efficiency of the current decoding process. This evaluation data also lays a solid foundation for subsequent processing steps.

[0038] In one possible implementation, the first vocabulary is pruned according to a first decoding speed and a first dataset to obtain a target vocabulary, including:

[0039] The first vocabulary is trimmed based on the first dataset to obtain the second vocabulary;

[0040] Update the first latency and average reception length based on the first vocabulary and the second vocabulary;

[0041] The second decoding speed is determined based on the second delay, the updated first delay, the first length, and the updated average receiving length.

[0042] If the second decoding speed is greater than the first decoding speed, the second vocabulary is pruned according to the first dataset to obtain the third vocabulary, and the next round of pruning is determined based on the decoding speed corresponding to the third vocabulary.

[0043] If the second decoding speed is less than or equal to the first decoding speed, the vocabulary obtained after the previous round of pruning is determined as the target vocabulary.

[0044] In this implementation, by progressively pruning the vocabulary, words that contribute little to decoding can be removed, resulting in a more concise and efficient target vocabulary. This helps improve decoding speed and reduce computational resource consumption. By continuously updating the first latency and average reception length, and recalculating the decoding speed, it is ensured that decoding performance is not excessively sacrificed while pruning the vocabulary. Comparing the decoding speed after different pruning rounds provides guidance for subsequent vocabulary adjustments. Pruning stops when the second decoding speed is less than or equal to the first decoding speed, ensuring the stability of the pruned vocabulary in terms of decoding performance.

[0045] In one possible implementation, the first vocabulary is pruned based on the first dataset to obtain the second vocabulary, including:

[0046] For any fourth word element in the first vocabulary, determine the first word frequency of the fourth word element in the first dataset;

[0047] The first word frequencies corresponding to each of the multiple lexical units in the first vocabulary are sorted to obtain the first sorting result;

[0048] In the first sorting result, the word whose first frequency is located in the first position interval is retained to obtain the second vocabulary. The first position interval is the interval before the nth sorting position. The nth sorting position is located at position p% in the first sorting result. n and p are both integers greater than or equal to 1.

[0049] In this implementation, the degree of pruning can be flexibly controlled by adjusting the value of p%, achieving a balance between decoding speed and accuracy. By setting a first position interval, high-frequency words are retained. Since these words appear frequently in the dataset, their retention helps maintain decoding accuracy. The pruned vocabulary typically contains the most frequently occurring words in the dataset, which have a high probability of appearing during decoding. Therefore, using a pruned vocabulary for decoding reduces the search space and improves decoding speed. Furthermore, it can be used with different datasets and vocabularies.

[0050] In one possible implementation, the average accepted length is updated based on the first vocabulary and the second vocabulary, including:

[0051] The total frequency of the first word in the first vocabulary is determined by summing the frequencies of each word element in the first vocabulary.

[0052] The total frequency of the second word corresponding to each word element in the second word list is determined based on the sum of the frequencies of each word element in the second word list.

[0053] The first adjustment parameter is determined based on the ratio of the total frequency of the second term to the total frequency of the first term;

[0054] The average acceptance length is updated based on the first adjustment parameter.

[0055] In this implementation, a first adjustment parameter reflecting the degree of word pruning can be obtained by calculating the ratio of the total second word frequency to the total first word frequency. By updating the average accepted length based on the first adjustment parameter, the change in the average accepted length after word pruning can be effectively reflected, providing a data foundation for subsequent processing.

[0056] In one possible implementation, the first time delay is updated based on the first vocabulary and the second vocabulary, including:

[0057] The second adjustment parameter is determined based on the ratio of the number of lexical units contained in the second vocabulary to the number of lexical units contained in the first vocabulary.

[0058] The first delay is updated based on the second adjustment parameter.

[0059] In this implementation, a second adjustment parameter reflecting the degree of word pruning can be obtained by calculating the ratio of the number of lexical units in the second vocabulary to the number of lexical units in the first vocabulary. By updating the first delay based on the second adjustment parameter, the degree of change in the first delay after word pruning can be effectively reflected, providing a data foundation for subsequent processing.

[0060] In one possible implementation, the method further includes:

[0061] The first text and the first instruction are input into the second model to obtain the first word and the third feature information output by the second model. The third feature information is used to reflect the feature information of the first text and the feature information of the first instruction.

[0062] In this implementation, the first text and the first instruction are input into the second model, allowing the second model to effectively process the first text and the first instruction to obtain the first word unit of the expected text and the feature information of the first text. The feature information of the first text can effectively guide the first model to make preliminary predictions about the subsequent word units of the expected text based on the first word unit, thereby reducing the number of direct inferences by the second model and accelerating the entire decoding process.

[0063] In one possible implementation, the method further includes:

[0064] At least one word to be verified is input into the second model so that the second model can reason about the at least one word to be verified and obtain the second reasoning word corresponding to each of the at least one word to be verified. The word to be verified is the word output by the reasoning of the first model.

[0065] For any one of the at least one to-be-verified lexical units, if the second inference lexical unit output by the second model is the same as the first inference lexical unit inferred by the first model for the to-be-verified lexical unit, then the verification result of the to-be-verified lexical unit is determined to be successful; or,

[0066] If the second inference term output by the second model is different from the first inference term obtained by the first model for the term to be verified, the verification result of the term to be verified is determined to be verification failure.

[0067] Based on the verified words that passed the verification, the target information extracted from the first text is determined.

[0068] In this implementation, by using a second model to verify the output of the first model, inaccurate or erroneous reasoning results can be effectively filtered out, which helps to improve the accuracy and reliability of the final output.

[0069] Secondly, embodiments of this application provide a text generation device, which may be an electronic device, or a chip or chip system within an electronic device. The text generation device may include a display unit and a processing unit.

[0070] When the text generation device is an electronic device, the display unit therein can be a display screen. The display unit is used to perform the display step so that the electronic device implements a text generation method described in the first aspect or any possible implementation of the first aspect.

[0071] When the text generation apparatus is an electronic device, the processing unit may be a processor. The text generation apparatus may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement a text generation method described in the first aspect or any possible implementation thereof.

[0072] When the text generation device is a chip or chip system within an electronic device, the processing unit can be a processor. The processing unit executes instructions stored in the storage unit to cause the electronic device to implement a text generation method described in the first aspect or any possible implementation of the first aspect. The storage unit can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0073] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory for storing code instructions, and the processor for running the code instructions to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0074] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0075] Fifthly, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0076] Sixthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0077] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0078] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0079] Figure 1 This is a schematic diagram illustrating an application scenario of the text generation method provided in the embodiments of this application;

[0080] Figure 2 A schematic diagram of the autoregressive decoding process of a large language model provided in an embodiment of this application;

[0081] Figure 3 A flowchart illustrating the speculative sampling algorithm provided in an embodiment of this application;

[0082] Figure 4 This is a schematic diagram of the speculative model reasoning process provided in the embodiments of this application;

[0083] Figure 5 This is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of this application;

[0084] Figure 6 This is a schematic diagram of the software structure of a terminal device provided in an embodiment of this application;

[0085] Figure 7 This is a schematic diagram illustrating the implementation of text generation in an embodiment of this application;

[0086] Figure 8 A schematic diagram illustrating the calculation of the first confidence level provided in an embodiment of this application;

[0087] Figure 9 A flowchart illustrating the lexical reasoning process provided in this application embodiment;

[0088] Figure 10A schematic diagram illustrating the calculation of the first acceptance probability provided in an embodiment of this application;

[0089] Figure 11 A schematic diagram illustrating the process of obtaining the target vocabulary provided in an embodiment of this application;

[0090] Figure 12 A schematic diagram illustrating the calculation of average receiving length provided for an embodiment of this application;

[0091] Figure 13 A schematic diagram of a single vocabulary trimming process provided in an embodiment of this application;

[0092] Figure 14 A schematic diagram illustrating word frequency calculation provided in an embodiment of this application;

[0093] Figure 15 A schematic diagram of vocabulary trimming provided in an embodiment of this application;

[0094] Figure 16 A schematic diagram of the first adjustment parameter provided in an embodiment of this application;

[0095] Figure 17 A flowchart illustrating the text generation method provided in this application embodiment;

[0096] Figure 18 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0097] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0098] 1. Large Language Model

[0099] Large Language Models (LLMs) are deep learning-based natural language processing models designed to understand and generate natural language text. They are artificial intelligence models trained on massive amounts of data, leveraging deep learning techniques to capture the complexity and diversity of language.

[0100] 2. Other terms

[0101] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0102] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0103] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0104] 3. Electronic equipment

[0105] The electronic devices in this application embodiment may include handheld devices, vehicle-mounted devices, etc., with data processing functions. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving vehicles, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices, wearable devices, terminal devices in 5G networks, or future evolution of public land mobile communication networks. Terminal devices in a network (PLMN), etc., are not limited to this in the embodiments of this application.

[0106] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0107] Furthermore, in this embodiment of the application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0108] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0109] In this embodiment, the electronic device or various network devices include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0110] To better understand the technical solution of this application, the relevant technologies involved in this application will be further described in detail below.

[0111] In the information society, various forms of text data are emerging in an endless stream. How to extract valuable core content from them has become an urgent problem to be solved. With the advancement of technology, text generation, a text processing method, has emerged. It can effectively extract key information from text, enabling users to more easily obtain useful information from text, thereby helping users improve work efficiency.

[0112] The following section introduces application scenarios for text generation methods. In one application scenario, text generation methods can be used to automatically generate call summaries.

[0113] The following is combined Figure 1 The application scenarios of the automatic call summary generation function are introduced. Figure 1This is a schematic diagram illustrating an application scenario of the text generation method provided in this application embodiment.

[0114] like Figure 1 As shown in (a) above, this can be understood as the call interface of an electronic device. The call interface can display multiple functional controls, namely, as shown in the figure, recording, mute, SMS, alerts, keyboard, speakerphone, and call summary.

[0115] Users can click on the call summary control to instruct the electronic device to extract a summary from the text information of the call. (See reference...) Figure 1 In (a), the user can click the call summary control. In response to the user's click on the call summary control, the electronic device can jump to display as shown in (a). Figure 1 The summary details interface shown in (b) is shown in the image.

[0116] Figure 1 (b) illustrates that the summary details screen can display a summary of the call content, which may include the topic of the call, the expected task to be performed, and plans or arrangements for related times or events. (See also...) Figure 1 In (b) of the text, the content of the call summary could be, for example, "We have agreed to meet at xxx time in xxx to discuss the xxx solution. We will focus on xxx to determine xxx. xxx."

[0117] However, it should be understood that the text generation method provided in this application is not limited to the call summary scenario mentioned above. The technical solution of this application can be applied to any scenario that involves extracting key information from text. For example, it can also extract key information from train ticket reminder SMS messages, flight ticket reminder SMS messages, etc.

[0118] Based on the application scenarios described above, the following section will further explain the specific implementation of the call summary generation function.

[0119] One approach involves using a large language model to implement call summarization. In this process, the large language model first performs preliminary processing on the call content to accurately capture the core information. Then, the model progressively generates summaries corresponding to the call content. This summarization stage can be understood as the decoding stage. During decoding, the model can progressively predict and generate the next word based on the input context (or parts of the already generated summary). This process relies on the characteristics of autoregressive models, where the model considers all previously generated words when generating each word. Therefore, after generating a word, the model adds it to the already generated summary and generates the next word based on this updated context.

[0120] The following is combined Figure 2 This paper introduces the reasoning (decoding) process of large language models. Figure 2 This is a schematic diagram of the autoregressive decoding process of a large language model provided in an embodiment of this application.

[0121] like Figure 2 As shown, assuming there is a word A, when word A is input into a large language model, the model can output the next word B after one inference process. This output can be generated based on the language knowledge that the model has already learned and mastered, as well as the context information provided above. In this process, the model's output is not only affected by the current input word A, but may also be influenced by the language patterns within the model and the inference results based on context information.

[0122] In autoregressive generation, the model combines previously generated content (such as lexical units A and B) and uses it as new input to feed back into the large language model. (Reference) Figure 2 First, words A and B are input together into the large language model. The model can then output the next word, C, as shown in the diagram. Similarly, words A, B, and C can be input together into the large language model, and the model can output the next word, D. This process demonstrates that in the autoregressive decoding process of the large language model, each inference result is generated based on the previous inference output; that is, each newly generated word is derived from the accumulated contextual information.

[0123] Significant memory overhead challenges arise when reasoning with large language models. Specifically, large language models consist of multi-layered Transformer structures, each containing numerous parameters such as weight matrices and bias terms. During inference, these parameters need to be fully loaded into computational units (e.g., GPUs). Simultaneously, the model performs complex computations on each input lexical unit, including numerous matrix multiplications and non-linear activation operations. The intermediate results of these computations also consume substantial memory, including GPU memory.

[0124] On the other hand, due to the extremely large number of parameters in large language models, weights and intermediate calculation results must be frequently transferred between GPU memory and main memory (CPU memory). GPU computing units need to access this data rapidly for efficient parallel computation. However, during the inference process of large language models, the massive amount of data necessitates frequent readings of large amounts of weights and intermediate results from GPU memory for each calculation, often causing memory bandwidth to become a performance bottleneck. Therefore, GPU computing units may not be able to fully utilize their computing power, limiting the amount of data processed and resulting in reduced GPU utilization.

[0125] To address this issue, one approach employs speculative sampling algorithms to accelerate the autoregressive decoding process of large language models. In this approach, a speculative model is constructed to perform multiple inferences, predicting several possible subsequent lexical units. These predicted lexical units are then fed into the large language model. Since the speculative model aims to accelerate the decoding process, the candidate lexical units it generates may not all be entirely accurate. The large language model, after receiving these candidate lexical units, performs further verification and inference to ensure the quality of the generated text.

[0126] During the validation phase, the large language model infers for each input lexical unit and generates its corresponding next lexical unit. In practical applications, the large language model may not directly generate the next lexical unit sequence that is completely consistent with the speculative model. Instead, it may generate the next lexical unit that best fits the current context based on its own language model capabilities and contextual information.

[0127] If most of the lexical units predicted by the speculative model are acceptable to the large language model (i.e., the large language model considers these lexical units to be reasonable or correct in the given context), then it can be assumed that multiple valid subsequent lexical units have been obtained after one inference verification by the large language model. In this way, the number of direct inferences by the large language model can be effectively reduced during text generation, thereby accelerating the entire decoding process.

[0128] During the inference process of the speculative model, due to its relatively small size, the number of weights and parameters required is significantly reduced compared to large language models. This means that the inference latency of the speculative model is relatively low. In the decoding process of the speculative model, an autoregressive approach is also used, i.e., inference is performed word by word.

[0129] The following is combined Figure 3 This section introduces the speculative sampling algorithm. Figure 3 This is a flowchart illustrating the speculative sampling algorithm provided in an embodiment of this application.

[0130] like Figure 3 As shown, assuming the current input word A is given to the speculative model, the speculative model will generate the next few candidate words, namely B, C, and E as shown in the figure. The reasoning process of the speculative model is similar to... Figure 2 The described autoregressive decoding process is similar; after three autoregressive inferences, the three word units B, C, and E are obtained sequentially. Subsequently, the speculative model can pass its input word unit A and the generated candidate word units B, C, and E as a sequence [A,B,C,E] to the large language model. The large language model receives the word unit sequence [A,B,C,E] generated by the speculative model and performs inference for each word unit in the sequence to generate its corresponding next word unit.

[0131] exist Figure 3 In the illustration, the large language model infers the word sequence [B,C,D,G] from the word sequence [A,B,C,E]. Specifically, after one inference by the large language model, the model predicts that the next word after input word A is word B, the next word after input word B is word C, the next word after input word C is word D, and the next word after input word E is word G.

[0132] When validating the predicted words of the speculative model, if the next word corresponding to a word predicted by the large language model is the same as the prediction of the speculative model for that word, then the prediction result of the speculative model for that word is considered correct, and the large language model can accept the prediction result of the speculative model for that word.

[0133] Reference Figure 3 Based on the word sequence [B,C,D,G] output by the large language model, the word sequence [A,B,C,E] inferred by the speculative model is validated. It can be understood that the word sequence [A,B,C,E] means that the speculative model infers that the next word after word A is word B, the next word after word B is word C, and the next word after word C is word E.

[0134] Next, the inference results [A,B,C,E] of the speculative model are verified in sequence. First, the prediction of the next word of word A is verified: based on the word sequence [B,C,D,G], it can be determined that the large language model predicts that the next word of word A is word B. Based on the word sequence [A,B,C,E], it can be determined that the speculative model predicts that the next word of word A is word B. Therefore, it can be determined that the speculative model's prediction for word A is correct, and thus, the large language model can accept word B.

[0135] Since the logic that the next word after word A in the input sequence [A,B,C,E] is word B is correct, the word inferred by the large language model for the input word B can be used as a reference to verify the predicted word output by the speculative model. Therefore, we next verify the prediction of the next word after word B. Based on the word sequence [B,C,D,G], we can determine that the large language model infers that the next word after word B is word C. Based on the word sequence [A,B,C,E], we can determine that the speculative model predicts that the next word after word B is word C. Therefore, we can confirm that the speculative model's prediction for word B is correct, and thus, the large language model can accept word C.

[0136] Similarly, since the logic that the next word after word B in the input sequence [A,B,C,E] is word C is correct, we can then verify the prediction of the next word after word C: Based on the word sequence [B,C,D,G], we can determine that the large language model infers that the next word after word C is word D. Based on the word sequence [A,B,C,E], we can determine that the speculative model predicts that the next word after word C is word E. Therefore, we can determine that the speculative model's prediction for word C is incorrect. Thus, the large language model does not accept word E and can discard the predicted word G inferred by the large language model for the incorrect word E.

[0137] This process illustrates that after the predicted terms [A, B, C, E] inferred by the speculative model are input into the large language model, the large language model, after inference and verification, can accept terms B and C as subsequent terms to the input term A. Simultaneously, the large language model itself infers that the next term to C is term D. Therefore, the next three terms to A can be determined as terms B, C, and D.

[0138] In practice, the speculative model, after one inference process, can output multiple predicted terms. The model considers these predicted terms to be likely the next term in the input sequence. Simultaneously, each predicted term has a corresponding prediction probability, representing the model's likelihood that it is the next term. During this process, the speculative model can select multiple terms with higher probabilities from a predefined vocabulary based on the given context. In this way, the speculative model can consider multiple possibilities when generating text and make choices based on prediction probabilities, thereby generating richer and more natural text content.

[0139] In other words, after one inference iteration, the speculative model can output multiple predicted terms for the input term and assign a prediction probability to each predicted term. These predicted terms and their corresponding prediction probabilities together constitute the model's prediction for the next term. During the next inference iteration, one can select a predicted term from the multiple predicted terms output by the speculative model as the input, or select multiple predicted terms as input for the speculative model to perform parallel inference and explore different generation paths. Alternatively, all predicted terms output by the speculative model can be used as input for inference, but this typically leads to a significant increase in computational cost. Alternatively, one can choose not to perform further inference.

[0140] In one approach, the number of predicted terms selected for each inference iteration of the speculative model is pre-set. That is, the number of predicted terms corresponding to the input terms is fixed when the speculative model outputs them. (Combined with...) Figure 4The reasoning process for this type of speculative model will be introduced. Figure 4 This is a schematic diagram of the speculative model reasoning process provided in the embodiments of this application.

[0141] like Figure 4 As shown, root node 401 represents the user input. We currently assume the user input is 'a', and the generation depth of the speculative model is preset to 5. This means that after inputting the word 'a' into the speculative model, 5 inferences will be performed. In each inference process, the number of predicted words output is preset.

[0142] For example, in the first inference process, suppose four predicted terms are selected for output. (See reference...) Figure 4 After the first inference, the four words with the highest predicted probabilities are selected from the vocabulary as predicted words, for example, b, c, d, and e. Therefore, the root node 'a' has four child nodes, each representing one of the four predicted words.

[0143] Next, these four predicted terms (b, c, d, e) can be input into the speculative model for a second inference. In this second inference, the number of outputs for each predicted term may differ. For example, for predicted term b, three predicted terms might be output; for predicted term c, two predicted terms; for predicted term d, two predicted terms; and for predicted term e, one predicted term. (See reference...) Figure 4 After the second inference, each predicted word will have multiple child nodes, representing its corresponding predicted word.

[0144] In this process, reasoning is performed on predicted word 'b', selecting the three words with the highest predicted probabilities from the vocabulary as predicted words, such as 'f', 'g', and 'h'. Therefore, child node 'b' has three child nodes, each representing one of the three predicted words. Similarly, reasoning is performed on predicted word 'c', selecting the two words with the highest predicted probabilities from the vocabulary, such as 'i' and 'j'. Therefore, child node 'c' has two child nodes, each representing one of the two predicted words. The understanding of the child nodes corresponding to other predicted words in the second reasoning process is similar and will not be elaborated upon here.

[0145] Similarly, a third inference can be performed. In the third inference, the number of outputs is again set for each selected predictive term. For example, for predictive term f, 3 predictive terms are selected for output; for predictive term g, 2 are selected for output; and so on. However, it is worth noting that in the third inference, it is possible to choose not to perform further inference for some predictive terms.

[0146] Similarly, fourth and fifth inferences can be performed. (See reference) Figure 4After five inference iterations, the speculative model can infer 25 predicted words for word 'a'. These 25 predicted words have a specific order, for example: [a,b,f,n,v,y], [a,b,f,n,v,z], [a,b,f,n,w], [a,b,f,n,x], [a,b,f,o], [a,b,f,p], [a,b,g,q], [a,b,g,r], [a,b,h,s], [a,b,h,t], [a,c,i,u], [a,c,j], [a,d,k], [a,d,l], [a,e,m]. The predicted words can then be input into a large language model for validation.

[0147] It's important to understand that when a large language model validates multiple predicted terms output by a speculative model, these predicted terms are arranged in a specific order. If a predicted term is found to be incorrect during validation, all subsequent predicted terms can be considered incorrect and discarded by the large language model without further validation. Conversely, if the large language model finds all predicted terms to be incorrect after validation, it can use the next term generated during validation as new input and feed it back into the speculative model for further reasoning.

[0148] However, reasoning based on a fixed generation depth can lead to inflexible reasoning processes and potentially reduced efficiency. Specifically, in the actual reasoning process of a speculative model, when a word has appeared in previous inferences, the model may smoothly infer multiple subsequent words with high accuracy due to prior knowledge. In this case, performing a moderate number of inferences on that word is reasonable. However, if a fixed generation depth is used, the reasoning process will have to stop after reaching the preset number of inferences, even if further inference is still necessary. This may require inputting the words from which the inference stopped into the speculative model for continued reasoning, and using a large language model to verify the inferred words, potentially reducing reasoning efficiency.

[0149] Conversely, when a word has not appeared in previous inference processes, the model may not be able to accurately infer subsequent words due to a lack of relevant knowledge. In this case, if multiple inferences are performed on this word, the results are likely to be incorrect. If a fixed generation depth is used for inference in this situation, the model will still need to continue inference until the fixed depth is met when it is no longer appropriate to continue (because the inference results are likely to be incorrect). This wastes inference time and reduces inference efficiency.

[0150] For example, assuming a preset generation depth of 10, the model will perform 10 inferences for each word. When a speculative model can directly infer the subsequent 20 words for a given word based on prior knowledge, performing only 10 inferences for that word at the preset generation depth would miss the opportunity to utilize the model's capabilities. Understandably, if all 20 words inferred by the speculative model are accepted by the large language model, the large language model's processing time will be reduced, equivalent to saving the latency of 20 large language model inferences. Conversely, if the speculative model only infers 10 words that are accepted by the large language model, the large language model's processing time will only be reduced by the latency equivalent to 10 inferences.

[0151] It is understandable that if we need to reason about the lexical unit that stopped at this point, then we need to input the lexical unit into the speculative model for reasoning, and we need to verify the reasoned lexical unit by the large language model, which may increase the latency of the reasoning process.

[0152] When a speculative model infers about a word it has never encountered before, it may struggle to infer the correct word after inferring the last two words. If, at this point, the model continues to infer about the word 10 times according to the preset generation depth, the last 8 words inferred are likely to be incorrect, thus wasting inference time.

[0153] It can also be understood that during a single inference process, for a given word, the model can infer multiple predicted words. Each of these predicted words corresponds to a prediction probability output by the speculative model. Typically, among the prediction probabilities corresponding to these multiple predicted words, there will be higher and lower prediction probabilities. A higher prediction probability indicates that the model believes the predicted word is more likely to become the next word. A lower prediction probability indicates that the model believes it is less likely to become the next word.

[0154] When a large language model validates predicted tokens passed from a speculative model, a window is typically set up to limit the maximum number of tokens that can be input in each validation cycle. For example, a maximum of 32 tokens can be input into the large language model in a single validation process.

[0155] Therefore, it's understandable that if, during a single validation process, the 32 tokens input to the large language model contain many predicted tokens with low probability, the number of predicted tokens that the large language model can accept will decrease accordingly. This leads to reduced utilization of validation nodes because each validation includes some tokens that are unlikely to be correct, thus reducing the overall efficiency of text generation.

[0156] On the other hand, in the speculative process, the speculative model infers from input words based on a vocabulary. Understandably, directly using a large original vocabulary for inference would increase the computational burden due to the large amount of data required, potentially leading to significant latency. Here, the original vocabulary refers to the vocabulary obtained from training a large language model.

[0157] To address the problems described above, this application proposes the following technical concept: based on the predicted probability corresponding to each predicted word, it can be determined whether further reasoning should be performed for that predicted word. This allows for flexible adjustment of the word reasoning process, thereby improving the flexibility of reasoning. Simultaneously, this flexibility helps to improve the efficiency of text generation. Furthermore, to further improve efficiency, the original vocabulary can be pruned, and the pruned vocabulary can be applied to the speculative model. This reduces the number of words the model needs to process, thereby increasing the speed of a single inference by the speculative model, ultimately achieving the goal of improving text generation efficiency.

[0158] The text generation method of this application can be executed by an electronic device with data processing capabilities, or by a chip, chip system, or processor that supports the implementation of the text generation method in the electronic device, or by a logic module or software that can implement all or part of the functions of the electronic device. This application does not impose specific limitations in this regard. The text generation method of this application embodiment will be described in detail below using an electronic device as the execution subject.

[0159] Electronic devices can be, for example, terminal devices. The following section will first combine... Figure 5 and Figure 6 A brief introduction to the terminal equipment.

[0160] For example, Figure 5 This is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of this application.

[0161] Figure 5 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. The terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0162] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device. In other embodiments of this application, the terminal device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0163] The processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors. In this embodiment, the text generation method proposed in this application can be implemented based on the processor.

[0164] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel.

[0165] The software system of a terminal device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc. This application uses the layered architecture Android system as an example to illustrate the software structure of the terminal device.

[0166] For example, Figure 6 This is a schematic diagram of the software structure of a terminal device provided in an embodiment of this application.

[0167] like Figure 6 As shown, the layered architecture divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the system may include an application layer, an application framework layer, an Android runtime and system libraries, a hardware abstraction layer (HAL), and a kernel layer. It should be noted that this application uses the Android system as an example; however, the solution can also be implemented in other operating systems (such as HarmonyOS, iOS, etc.) as long as the functions implemented by each module are similar to those in the embodiments of this application.

[0168] The application layer can include a series of application packages.

[0169] like Figure 6 As shown, the application package may include applications such as camera, calendar, phone, map, music, settings, email, video, and social media. Of course, the application layer may also include other application packages, such as third-party applications like payment apps, shopping apps, banking apps, and social media apps; this application is not limited to these. In this embodiment, a first model and a second model may be deployed in the application layer, for example. The second model processes input instructions and input text to obtain the first word unit of the expected text. Subsequently, the first model can perform multiple inferences on the word units output by the second model to generate multiple predicted word units. Then, the second model can verify the multiple predicted word units inferred by the first model to obtain the expected text that meets the instruction requirements.

[0170] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0171] like Figure 6 As shown, the application framework layer may include a window manager, content provider, resource manager, view system, notification manager, etc.

[0172] The Android runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0173] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0174] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0175] The system library can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0176] The HAL layer is a wrapper around Linux kernel drivers, providing interfaces to the upper layers and shielding them from the implementation details of the lower-level hardware.

[0177] The HAL layer can include Wi-Fi HAL, audio HAL, camera service (Camera HALServer) unit, and software code library, etc.

[0178] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0179] The technical solutions of the embodiments of this application and how the technical solutions of the embodiments of this application solve the above-mentioned technical problems will be described in detail below with reference to the accompanying drawings and specific examples. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0180] The technical solution of this application involves a text generation method, which will be described below. The flow of the text generation method will be explained with reference to specific embodiments. The following section will combine... Figures 7 to 10 A detailed introduction will be given, including, Figure 7 This is a schematic diagram illustrating the implementation of text generation in an embodiment of this application. Figure 8 This is a schematic diagram illustrating the calculation of the first confidence level provided in an embodiment of this application. Figure 9 This is a flowchart illustrating the lexical reasoning process provided in the embodiments of this application. Figure 10 This is a schematic diagram illustrating the calculation of the first acceptance probability provided in an embodiment of this application. The text generation method may include, for example, the following steps:

[0181] S701. Input the first text and the first instruction into the second model to obtain the first word and the third feature information output by the second model.

[0182] In traditional solutions, the second model can directly process the first text according to the first instruction to generate text content that conforms to the requirements of the first instruction. In this embodiment, the first model can be used to implement lexical reasoning to improve the reasoning speed. Furthermore, the second model is used for lexical verification to ensure the accuracy of the generated lexical units. Both the first and second models are pre-trained and possess the ability to extract information from the text. In one implementation, the second model can be, for example, a large language model (also known as a target model), and the first model can be, for example, the speculative model (also known as a draft model) described above.

[0183] In practical implementation, the first text and the first instruction can be input into the second model. The second model can process the first text and the first instruction to generate a first word element and third feature information. The first instruction is used to instruct the extraction of a first type of information from the first text. For example, the first instruction could be "Please extract the summary based on the above text content." The first type can be in the form of text or numbers, etc. The first word element is the first word element of the text content that meets the requirements of the first instruction. The third feature information can be used to reflect all the feature information obtained during the reasoning process of the second model, such as the feature information of the first text and the first instruction. Specifically, it can include and reflect the contextual information in the first text and the contextual information in the first instruction, etc.

[0184] After generating the first lexical unit, the first text, the first lexical unit, the first instruction, and the third feature information can be input into the first model. The first model can use this information to perform multiple inferences on the first lexical unit to generate subsequent text content. In this process, the third feature information provides the first model with contextual background about the first text, helping the first model to infer subsequent lexical units more accurately.

[0185] The following is combined Figure 7 The processing flow of the second model will be introduced.

[0186] like Figure 7 As shown, assuming there is a first text "how can xxx", the first text is input into the second model to generate the first word unit. Figure 7 The illustration shows the processing of the word "how can" after it is input into the second model. The area enclosed in box 701 illustrates the process by which the second model processes the word "how can".

[0187] It's important to understand that in the actual implementation, the entire content of the first text and the first instruction need to be concatenated together and input into the second model. Due to space limitations, this will not be discussed further here. Figure 7 The details are not shown further. However, it is understandable that what is being conveyed here is that the first text and the first instruction are input into the second model for processing.

[0188] Before inputting the text into the second model, it can typically be preprocessed. These preprocessing operations may include, for example, word segmentation and stop word removal. The preprocessed text can then be input into the second model.

[0189] Reference Figure 7After the preprocessed text is input into the second model, it undergoes encoding. This step converts each word in the preprocessed text into a corresponding vector, thus transforming the text into a fixed format for easier subsequent processing by the model. After obtaining the vector corresponding to each word, these vectors are input into the feature extraction layer. The feature extraction layer analyzes and understands the vectors corresponding to each word to output the feature information corresponding to each word. This feature information reflects the contextual relationship of the word in the text.

[0190] Subsequently, the feature information corresponding to each of the multiple word units is input into the language model head. The language model head stores a vocabulary, which is generated during the training of the second model. When the feature information corresponding to a word unit is input into the language model head, the language model head analyzes the relationship between the input word unit and each word unit in the vocabulary, and generates a probability distribution based on these relationships. This probability distribution represents the probability that each word unit in the vocabulary is the next word unit.

[0191] After the sampling layer, the word with the highest probability in the vocabulary can be obtained through a single sampling operation and used as the output. For example... Figure 7 As shown, after the input text "how can" is fed into the second model, the word units "how" and "can" are analyzed and processed, and the word units "can" and "I" are output sequentially. Specifically, the output word unit "can" is the second model's response to the input word unit "how," and the output word unit "I" is the second model's response to the input word unit "can."

[0192] Based on the above introduction, it can be determined that after text is input into the second model, the second model can output for each word in the text. In this process, the processing of the second model can be regarded as two stages.

[0193] In the first stage, the second model analyzes and understands the content of the first text and the first instruction to obtain a more accurate representation of the text content. This stage mainly involves text encoding, feature extraction, and preliminary language model analysis.

[0194] In the second stage, based on the text content representation obtained from the first stage analysis, the second model can generate user-specified information. This stage mainly involves reasoning out the text word by word (or phrase by phrase) based on the extracted key information and using it as the model's final output.

[0195] Therefore, it can be understood that when the first instruction and the first text are input into the second model, the second model will extract key information from the first text based on the first instruction. Then, the second model can output the first word element of the expected text (such as a summary), i.e., the word element "I" shown in the figure. Afterwards, the first word element, the first text, the first instruction, and the third feature information can be input into the first model.

[0196] Theoretically, when information is first input into the first model (i.e., after the second model generates the first lexical unit, it is expected that the first lexical unit will be input into the first model for lexical inference), since the behavior of the first model needs to be consistent with the behavior of the second model (i.e., the information input into the second model also needs to be input into the first model), the first text and the first instruction need to be transmitted to the first model.

[0197] In one implementation, the first word, the first text, and the first instruction can be concatenated and used as the input to the first model. Meanwhile, to ensure dimensionality consistency, the first word in the concatenated text can be removed before being used as the input to the first model.

[0198] We can continue to combine Figure 7 To understand this, after obtaining the first word element "I", we can concatenate it with the first text "how can" to obtain the text "how can I". Additionally, refer to... Figure 7 It can be determined that the text input to the second model consists of two terms. To ensure dimensionality consistency, the text input to the first model also consists of two terms. For example... Figure 7 As shown, "can I" can be transmitted to the first model without needing to input the word "how". Simultaneously, the third feature information, i.e., the feature "f" shown in the diagram, needs to be transmitted to the first model. how "and "f can ".

[0199] It's important to understand that while the second model outputs information from the first text, it responds to each word in the first text (i.e., infers a word, which may not necessarily match the content of the first text). However, when using the words output by the second model, it uses the predicted word corresponding to the last word in the first text for subsequent operations. In essence, inputting all words from the first text except the last one into the second model allows the model to analyze and understand the overall content of the first text.

[0200] S702. Input the first feature information, the first instruction, the first word element, and the first text into the first model, so that the first model infers from the first word element and obtains multiple second word elements output by the first model. The multiple second word elements correspond to their respective first prediction probabilities. The first prediction probability is used to indicate the probability that the second word element is the next word after the first word element.

[0201] Based on the above description, it can be determined that after the second model generates the first word element, all feature information, the first instruction, the first text, and the first word element obtained during the reasoning process of the second model are input into the first model. The first model can then perform multiple inferences on the first word element. In this embodiment, the first model can output at least one word element after each inference process.

[0202] In one implementation, after inputting all feature information obtained from the second model's inference, the first instruction, the first text, and the first word element into the first model, the first model infers from the first word element to obtain multiple second word elements output by the first model. Each of the multiple second word elements corresponds to its own first predicted probability. The first predicted probability indicates the probability that the second word element is the next word after the first word element.

[0203] In this embodiment, the first model can be used for reasoning on word units. The second model can be used to generate the first word unit of the expected text, or to verify multiple word units output by the first model.

[0204] It is important to understand that, when the second model is used to achieve different functions, the first feature information input to the first model will also be different in different situations because the content of its output feature information is different.

[0205] Specifically, when the second model outputs the first word of the expected text based on the first instruction and the first text, the feature information output by the second model can be the feature information of the first text and the first instruction. Then, the first feature information input into the first model can be used to reflect the feature information of the first text and the first instruction, that is, the third feature information mentioned in step S701.

[0206] When the second model verifies multiple lexical units output by the first model, the feature information output by the second model can be the feature information of the first text, the feature information of the first instruction, and the feature information of at least one historical lexical unit accepted by the second model for verification. Then, the first feature information input into the first model at this time can be used to reflect the feature information of the first text, the feature information of the first instruction, and the feature information of at least one historical lexical unit accepted by the second model for verification.

[0207] Furthermore, during the multiple inference processes of the first model, the feature information input into the first model can be used to reflect the feature information of the first text, the feature information of the first instruction, and the feature information of at least one historical word obtained by the inference of the first and second models.

[0208] We can continue to combine Figure 7 To understand the reasoning process of the first model.

[0209] like Figure 7 As shown, the area enclosed in 702 illustrates the reasoning process of the first model. Based on the above introduction, it can be determined that the word unit "can", the first word unit "I", and the text feature information "f" in the first text need to be considered. how "and "f can "The input is fed into the first model. Then, the first model can perform inference on the first word 'I'."

[0210] refer to Figure 7 After the word "can" is input into the first model, the model first encodes the word "can" and converts it into the vector "e". can This facilitates subsequent operations by the model. Afterwards, the vector "e" corresponding to the first word "can" can be... can "and the feature information "f" corresponding to the word "how" how "The input is fed into the autoregressive head. In this way, the autoregressive head of the first model can obtain relevant feature information. The main function of the autoregressive head is to predict the next word or symbol step by step based on the previously generated words as context. Through continuous word prediction steps, it ensures that the generated sequence is contextually related, thereby improving the coherence of the text."

[0211] Furthermore, after the first word "I" is input into the first model, the model first encodes the first word "I" and converts it into the vector "e". I This facilitates subsequent operations by the model. Afterwards, the vector "e" corresponding to the first word "I" can be... I "and the feature information "f" corresponding to the word "can" can "Input to the autoregressive head to obtain the feature information "f" corresponding to the first word "I". I ".

[0212] In the first word element "I", the feature information "f" is... I "After being input into the language model head, it can calculate the relationship between the first word element "I" and every word element in the vocabulary, and obtain the first prediction probability of the first word element "I" corresponding to every word element in the vocabulary. The first prediction probability can represent the probability that any word element in the vocabulary is the next word element of the first word element "I".

[0213] After multiple samplings, the model can output multiple second word units for the first word unit "I". For example, such as... Figure 7 As shown, by sampling twice, two second word units with higher probabilities in the vocabulary can be obtained, and these two second word units and their corresponding first predicted probabilities are used as the output of the model, namely the two second word units "make" and "help" shown in the figure.

[0214] S703. For any one of the multiple second word elements, determine the first confidence level corresponding to the second word element based on the first prediction probability corresponding to the second word element.

[0215] Based on the above introduction, it can be determined that each second word element inferred by the first model has a corresponding first predicted probability. The higher the first predicted probability of a second word element, the greater the likelihood that the first model considers that second word element to be the next word element, which also means that the reliability of that second word element as the next word element is higher. Conversely, the lower the first predicted probability of a second word element, the less likely the first model considers that second word element to be the next word element, which also means that the reliability of that second word element as the next word element is lower.

[0216] In this embodiment, when the model performs multiple inferences, since deeper predicted terms are derived based on shallower predicted terms (and possibly other information), the reliability of deeper predicted terms depends on the reliability of shallower predicted terms. Therefore, the reliability of deeper predicted terms can be determined based on the reliability of shallower predicted terms.

[0217] In one implementation, the first confidence level corresponding to a deeper predicted term can be defined as the product of all predicted probabilities on that branch. In this way, the first confidence level corresponding to each predicted term contains information from the previous layer, thus providing a more comprehensive reflection of the reliability of the entire inference path.

[0218] The following is combined Figure 8 This section provides a brief introduction to the calculation of the first confidence level corresponding to the predicted term.

[0219] like Figure 8 As shown, it is currently assumed that the first model can perform three inferences. In the first inference, after inputting the word "I", the first model can infer two predicted words, namely "want" and "go" as shown in the figure. The predicted probability of the word "want" is 0.6, and the predicted probability of the word "go" is 0.2. Therefore, the first confidence level corresponding to the predicted word "want" is 0.6 × 1.0, i.e., 0.6. And the first confidence level corresponding to the predicted word "go" is 0.2 × 1.0, i.e., 0.2.

[0220] After that, it is assumed that the predicted tokens "want" and "go" are input into the first model for the second inference. Reference Figure 8 , after inputting the predicted token "want" into the first model, the first model can output 2 predicted tokens, namely "eat" and "go" as shown in the figure. The prediction probability of the predicted token "eat" output by the first model is 0.8, and the prediction probability of the predicted token "go" is 0.1.

[0221] Based on the above description, it can be determined that the first confidence corresponding to the predicted token "eat" may be the product of all prediction probabilities on this branch, that is, the product of the prediction probabilities corresponding to the predicted tokens "I", "want" and "eat", which is 1.0×0.6×0.8, that is, 0.48 as shown in the figure. Similarly, the first confidence corresponding to the predicted token "go" may be the product of the prediction probabilities corresponding to the predicted tokens "I", "want" and "go", which is 1.0×0.6×0.1, that is, 0.06 as shown in the figure. The calculation of the first confidence for other predicted tokens is similar, and will not be repeated here.

[0222] S704. For any one of the plurality of second tokens, a first acceptance probability corresponding to the second token is determined according to the first confidence corresponding to the second token and a preset mapping relationship, where the preset mapping relationship is used to indicate the correspondence between the confidence and the acceptance probability.

[0223] After obtaining a plurality of second tokens, it is impossible to determine which of these second tokens are to be further inferred, which can improve the overall inference efficiency. For example, if the first model performs multiple inferences on a certain token or certain tokens, but the verification results of these tokens are wrong, the resources and time spent on inferring these tokens will be wasted, which may lead to a reduction in the overall inference efficiency of the system.

[0224] Therefore, after obtaining a plurality of predicted tokens (the second tokens) each time, a specific rule can be used to filter these predicted tokens. The purpose of this is to select from the plurality of predicted tokens those tokens that are more likely to be accepted by the second model, and then use these filtered tokens as the input of the first model to continue the inference.

[0225] It can be understood that the first confidence can represent the degree of confidence of the first model in the inferred second token, however, it cannot directly reflect whether the predicted token (i.e., the second token) will be accepted by the second model.

[0226] To achieve more accurate filtering, one implementation can determine the first acceptance probability of a predicted word based on its first confidence level. The first acceptance probability indicates the likelihood that the predicted word will be accepted by the second model. Thus, in subsequent processing, the first acceptance probability can be determined based on the first confidence level of the predicted word. Furthermore, the first acceptance probability can be used to determine whether further inference should be performed on the predicted word.

[0227] In one implementation, a mapping between a first confidence level and a first acceptance probability can be established through experimentation, thereby directly determining the first acceptance probability corresponding to a word based on its first confidence level. Specifically, multiple text datasets can be used for the experiment. During the experiment, for each text, it can be input into a first model for multiple inferences, resulting in multiple words output by the first model. These words can then be input into a second model for verification to determine their probability of acceptance by the second model. The number of inferences in the first model can be preset according to actual conditions; for example, it can be preset to 16.

[0228] In one implementation, for example, it can be done according to Figure 9 The single-chain tree structure shown is used for lexical reasoning. During each reasoning process, the first model can perform one sampling to output a predicted lexical. (Refer to...) Figure 9 Suppose we input word 'a' into the first model. After one inference process, the first model can output word 'b'. Then, we can input word 'b' back into the first model, and the model can perform further inference based on word 'b'. For example... Figure 9 As shown, after inputting word b into the first model, the first model can output word c. The subsequent reasoning process is similar and will not be elaborated here. This continues until 16 word inferences are completed.

[0229] For each predicted term, its corresponding first confidence level can be calculated. Furthermore, after verification, these predicted terms will produce a definite verification result: accepted or unaccepted. Therefore, based on the first confidence levels of these predicted terms and the verification results, a mapping relationship can be established to determine the first acceptance probability corresponding to different first confidence levels.

[0230] In one implementation, for example, statistics can be performed on multiple predicted words under the same first confidence level, the number of accepted predicted words can be calculated, and the ratio of the accepted number to the total number of predicted words under the first confidence level can be determined as the first acceptance probability corresponding to the first confidence level.

[0231] In another implementation, for example, the first confidence range [0,1] can be divided into multiple intervals, and the predicted words in each interval can be statistically analyzed to calculate the number of accepted predicted words. Then, the ratio of this number to the total number of predicted words in that interval can be determined as the first acceptance probability corresponding to the first confidence interval.

[0232] The following section will use any one of multiple first confidence intervals as an example to illustrate the calculation method for its corresponding first acceptance probability. The calculation method for the first acceptance probability of other first confidence intervals is similar. (Combined with...) Figure 10 The calculation process for the first acceptance probability is introduced.

[0233] like Figure 10 As shown, assume there exists a first confidence interval [0.7, 0.8], with 10 predicted terms within this interval. The first confidence level for term 1 is 0.78, for term 2 it is 0.72, for term 3 it is 0.76, for term 4 it is 0.74, for term 5 it is 0.72, for term 6 it is 0.77, for term 7 it is 0.73, for term 8 it is 0.79, for term 9 it is 0.71, and for term 10 it is 0.75.

[0234] After validation using the second model, 4 predicted terms were accepted, while 6 were not. Based on the above, the first acceptance probability corresponding to the first confidence interval [0.7, 0.8] is 4 / 10, or 0.4.

[0235] In this embodiment, 128 text data entries can be randomly selected from the first dataset for the experiment, according to... Figure 9 The single-chain tree structure shown is used for lexical inference. Subsequently, the resulting 128×16 predicted lexical units are input into the second model for validation. The first confidence level range is divided into the following four intervals, and the first acceptance probability corresponding to different first confidence level intervals is calculated, as shown in Table 1. In this embodiment, the first dataset may include scenario task data and non-scenario task data. Scenario task data refers to text data in the application scenario of this solution; for example, in a call summary scenario, call-related text data belongs to scenario task data. Non-scenario task data refers to text data from other scenarios.

[0236] Table 1:

[0237] First confidence interval First acceptance probability 0-0.7 0.2 0.7-0.8 0.4 0.8-0.9 0.6 0.9-1.0 0.7

[0238] As shown in Table 1, the first acceptance probability corresponding to the first confidence interval [0-0.7] is 0.2, the first acceptance probability corresponding to the first confidence interval [0.7-0.8] is 0.4, the first acceptance probability corresponding to the first confidence interval [0.8-0.9] is 0.6, and the first acceptance probability corresponding to the first confidence interval [0.9-1.0] is 0.7.

[0239] S705. Determine the first threshold based on the first delay and the second delay.

[0240] In this embodiment, the first delay is the time required for one inference by the first model, and the second delay is the time required for one inference by the second model.

[0241] Based on the above, it can be determined that the first acceptance probability of a word element reflects the likelihood that the word element will be accepted by the second model. Specifically, a higher first acceptance probability means that the word element is more likely to be accepted by the second model. Conversely, a lower first acceptance probability indicates that the word element is less likely to be accepted by the second model.

[0242] When the second model has a low probability of accepting a term, it means that the term is more likely to be an error term, and further reasoning on that term may become meaningless. Therefore, in this case, it is advisable to stop further reasoning on that term.

[0243] To achieve this goal, a first threshold can be set. By comparing the first acceptance probability corresponding to a word with this first threshold, it can be determined whether further reasoning is needed for that word. For example, words with a first acceptance probability lower than the first threshold can be discarded and no further reasoning can be performed on them. Conversely, words with a first acceptance probability higher than the first threshold can be input into the first model for further word reasoning.

[0244] Based on the above introduction, it can be determined that the purpose of using the first model for word inference is to accelerate decoding speed. That is, the goal is to save time when inferring each word. Therefore, if the first model fails to save time in inferring a word, it can be considered worthless and discarded without further processing. Conversely, if the first model achieves time savings in inferring a word, it can be considered valuable and accepted for further processing.

[0245] In one implementation, the time-saving objective of inferring a word can be calculated quantitatively. This can be considered from two aspects: firstly, the initial latency (cost time) required for the first model to infer the word; and secondly, the second latency (benefit time) that can be saved if the second model accepts the word. However, after the first model infers a word, it cannot be immediately determined whether the word will be accepted by the second model.

[0246] Therefore, for example, the payout duration corresponding to a word can be estimated based on the first acceptance probability (reflecting the likelihood that the word will be accepted by the second model) and the time required for the second model to make one inference.

[0247] As an example, in one implementation, the formula for calculating the revenue duration corresponding to a word element can be referenced as follows: Formula 1:

[0248] T b1 =T llm *R accept Formula 1

[0249] In Formula 1, T llm R represents the second time delay, indicating the time required for the second model to perform one inference operation. accept Let be the first acceptance probability corresponding to the word, representing the likelihood that the word will be accepted by the second model. It is understandable that after the second model completes training, its reasoning ability tends to stabilize; therefore, the second delay T... llm It is a relatively fixed value. Based on Formula 1, it can be determined that the first acceptance probability R... accept The larger the value, the longer the payout duration T for the corresponding word element. b1 The larger the probability, the greater the time gain. This means that if the tokens inferred by the first model have a high probability of being accepted by the second model, then the time gain from these tokens will be greater.

[0250] You can combine examples to target the revenue duration T b1 To understand this, suppose there are 10 words, each with a first acceptance probability of 0.4. This means that, theoretically, if we consider the probability of each word being accepted by the second model independently, then out of these 10 words, approximately 4 (i.e., 10 × 0.4) words might be accepted by the second model.

[0251] However, it's important to note that "approximately 4" here is an expected or predicted value; the actual number of accepted tokens may vary. Based on this prediction, we can calculate that if all 10 tokens are inferred and assuming 4 are accepted, the total revenue duration would be 4 × T. llm T llmIt is the time required for the second model to perform one inference.

[0252] Therefore, when calculating the revenue duration for a single word, the revenue duration can be understood as the potential time saved for that single word.

[0253] After determining the revenue duration corresponding to a word, the value of the inferred word can be determined based on the cost duration and revenue duration, and a decision can be made on whether to proceed with subsequent operations accordingly.

[0254] Specifically, when the revenue duration corresponding to a word element is greater than the cost duration, it means that the first model has achieved the goal of saving time in the process of inferring that word element. Therefore, the word element can be considered valuable, and subsequent operations can proceed. Conversely, when the revenue duration corresponding to a word element is less than or equal to the cost duration, it means that the first model has not achieved the goal of saving time in the process of inferring that word element. Therefore, the word element can be considered worthless and discarded to avoid unnecessary resource consumption. Based on this, the first threshold can be determined.

[0255] For example, in one implementation, in T llm *R accept >T draft In the case of R accept >T draft / T llm The first model can receive the word and perform further reasoning on it. In T llm *R accept ≤T draft In the case of R accept ≤T draft / T llm The first model can discard the word and not perform further reasoning on it. Where T draft The first delay represents the time required for the first model to perform one inference operation.

[0256] Therefore, the formula for calculating the first threshold can be referenced as follows: Formula 2

[0257]

[0258] It is understandable that, given that a word is accepted, the time saved by inferring that word can be understood as the second delay minus the first delay, i.e., T. llm -T draft .

[0259] Therefore, in another implementation, the revenue duration corresponding to a word can be estimated based on the first acceptance probability of the word and the time saved by the word.

[0260] For example, in one implementation, the formula for calculating the revenue duration corresponding to a word element can refer to the following formula three:

[0261] T b2 =(T llm -T draft )*R accept Formula 3

[0262] In Formula 3, T llm For the second delay, T draft For the first delay, R accept Let T be the first acceptance probability corresponding to the word. It is understandable that after the first model completes training, its reasoning ability tends to stabilize; therefore, the first delay T... draft It is also a relatively fixed value. This can be understood as the initial acceptance probability R... accept The larger the value, the longer the payout duration T for the corresponding word element. b2 It will get bigger and bigger.

[0263] In this case, it can be determined that in (T) llm -T draft )*R accept >T draft In the case of R accept >T draft / (T llm -T draft The first model can receive the word and perform further reasoning on it. In (T) llm -T draft )*R accept ≤T draft In the case of R accept ≤T draft / (T llm -T draft The first model can discard the word and not perform further reasoning on it.

[0264] Therefore, the formula for calculating the first threshold can be referenced as follows: Formula 4:

[0265]

[0266] It is understandable that the first threshold shown in Formula 4 has a stricter requirement for the first acceptance probability of a word compared to the first threshold shown in Formula 2. In this case, by raising the first threshold standard, higher-quality words can be selected, thereby improving the overall system performance or accuracy.

[0267] S706. If the first acceptance probability is greater than the first threshold, the second feature information, the first instruction, and the second word are input into the first model so that the first model can reason about the second word and obtain multiple third words output by the first model. The multiple third words correspond to their respective second prediction probabilities. The second prediction probabilities are used to indicate the probability that the third word is the next word of the second word.

[0268] In this embodiment, the second feature information is used to reflect the feature information of the first text, the feature information of the first instruction, and the feature information of at least one historical word obtained by reasoning from the first model and the second model.

[0269] We can continue to combine Figure 7 Based on the understanding, the process described in S702 can lead to two second word units, namely the word units "make" and "help" shown in the diagram. The first acceptance probability corresponding to the second word unit "make" can be compared with a first threshold to determine whether further reasoning should be performed on "make". Similarly, the first acceptance probability corresponding to the second word unit "help" can be compared with a first threshold to determine whether further reasoning should be performed on "help".

[0270] If the first acceptance probability corresponding to the second word element is less than or equal to the first threshold, then the second word element can be discarded and no further processing is performed on it. If the first acceptance probability corresponding to the second word element is greater than the first threshold, then further reasoning can be performed on the second word element.

[0271] Assuming that the first acceptance probability corresponding to the second word "make" and the second word "help" is greater than the first threshold, then parallel reasoning can be performed on the second word "make" and the second word "help".

[0272] Reference Figure 7 In the second reasoning process, the second feature information, the first instruction, and the second word units "make" and "help" can be input into the first model so that the first model can perform parallel reasoning on the second word units "make" and "help" to obtain multiple third word units output by the first model. The multiple third word units correspond to their respective second prediction probabilities.

[0273] exist Figure 7In the illustration, the second feature information is used to reflect the feature information of the first text "how can" and the feature information of the first word "I" obtained by the second model inference. It should be understood that in the third inference and subsequent inference processes, the feature information input to the first model is used to reflect the feature information of the first text "how can", the feature information of the first word "I" obtained by the second model inference, and the feature information of at least one historical word obtained by the first model inference.

[0274] For example, such as Figure 7 As shown, assuming we are reasoning about the second word "make", we can obtain two third word elements "a" and "our". Similarly, reasoning about the second word "help" can obtain two third word elements "with" and "you".

[0275] Meanwhile, assuming that the first acceptance probabilities for the third word units "a" and "our" are both less than the first threshold, while the first acceptance probabilities for the third word units "with" and "you" are both greater than the first threshold, then parallel inference can be performed for the third word units "with" and "you".

[0276] refer to Figure 7 In the third inference process, the feature information of the first model is used to reflect the feature information of the first text "howcan", the feature information of the first word "I" obtained by the second model inference, and the feature information of the second word "help" obtained by the first model inference.

[0277] S707. After obtaining multiple third-term elements, any one of the third-term elements can be compared with a first threshold to determine whether to perform inference for that third-term element. This process continues until the first model is unable to perform inference for any third-term element, at which point the word element output by the first model can be verified.

[0278] S708. Input at least one word to be verified into the second model, so that the second model infers on the at least one word to be verified to obtain the second inference word corresponding to each of the at least one word to be verified. The word to be verified is the word output by the inference of the first model.

[0279] S709. For any one of the at least one tokens to be verified, determine the verification result of the token to be verified.

[0280] In this embodiment, for any one of the at least one to-be-verified lexical units, if the second inference lexical unit output by the second model is the same as the first inference lexical unit inferred by the first model for the to-be-verified lexical unit, the verification result of the to-be-verified lexical unit is determined to be verified as passed; or,

[0281] If the second inference term output by the second model is different from the first inference term obtained by the first model for the term to be verified, the verification result of the term to be verified is determined to be verification failure.

[0282] S710. Based on the verification results of the verified words to be verified, determine the target information extracted from the first text.

[0283] The verification process described in steps S708, S709, and S710 is similar to the lexical verification described above, and will not be repeated here.

[0284] In this embodiment, by introducing the concept of a first confidence level, the quality of the second word can be evaluated more precisely. Furthermore, a mapping relationship is established between the first confidence level and the first acceptance probability of a word. Using this mapping relationship, the first acceptance probability corresponding to a word can be quantitatively determined based on its first confidence level. Further, the first acceptance probability of each word is compared with a set first threshold to determine whether further reasoning is needed for that word. This allows for the effective selection of appropriate words for reasoning, thereby improving the flexibility of word reasoning. Simultaneously, by timely filtering and discarding of reasoned words, unnecessary computation can be reduced, improving the efficiency of the first model during the reasoning process, thus reducing the overall system latency and increasing response speed.

[0285] Based on the above embodiments, it can be determined that during the reasoning process targeting lexical units, the lexical units inferred by the first model come from its vocabulary. Therefore, the more lexical units the vocabulary contains, the greater the computational load of the first model in a single reasoning process, which may slow down the speed of the first model's reasoning in a single operation. In the text generation method provided in this application, the first model can perform reasoning on the first lexical units based on the target vocabulary. The following is a combination of... Figure 11 The process of obtaining the target vocabulary is described. Figure 11 This is a schematic diagram of the process for obtaining the target vocabulary provided in an embodiment of this application.

[0286] You can refer to this. Figure 11 To understand this, the implementation of obtaining the target vocabulary in this application is a cyclical process. In the first loop, the decoding speed corresponding to the original vocabulary needs to be calculated. Afterwards, the original vocabulary can be pruned, and the decoding speed can be updated according to specific rules. In subsequent loops (excluding the first loop), the pruned vocabulary from the previous loop can be pruned again, and the decoding speed can be updated according to specific rules.

[0287] After each decoding speed update, the decoding speed before the update needs to be compared with the decoding speed after the update to determine whether to terminate the algorithm.

[0288] If the updated decoding speed is less than or equal to the original decoding speed, the vocabulary obtained after the previous round of pruning can be determined as the target vocabulary, and the iteration process can be stopped.

[0289] If the updated decoding speed is greater than the original decoding speed, the vocabulary pruning and decoding speed update operations according to specific rules can be repeated until the algorithm termination condition is met.

[0290] The calculation of decoding speed will be explained below.

[0291] It's understandable that after multiple inferences on a single word, the first model will output multiple words. These words are then input into the second model for verification. After verification, the second model may accept some of these words, considering them as correctly decoded words. Based on this, the decoding speed can be calculated.

[0292] In one implementation, the decoding speed can be determined based on a first delay, a second delay, an average acceptance length, and a first length. In this embodiment, the average acceptance length is the average length of the tokens output by the first model that are accepted by the second model, and the first length is the number of inference attempts made by the first model.

[0293] It is understandable that the second model will validate the multiple word units output by the first model and may accept some of them. If the multiple word units output by the first model to the second model are considered as a word unit sequence, then it can be determined that some word units in this word unit sequence will be accepted by the second model.

[0294] Furthermore, after multiple such word sequences are validated by the second model, it can be understood that some words in each word sequence will be accepted by the second model. However, the length of words accepted by the second model in each word sequence may differ. To achieve a quantitative evaluation, in one implementation, for example, the average length of words accepted by the second model for each of the multiple word sequences can be calculated to determine the average length of words output by the first model that are accepted by the second model. This average accepted length can be understood as the average length of words correctly decoded by the first model.

[0295] The following is combined Figure 12 A brief explanation of the calculation of average received length is provided. Figure 12 This is a schematic diagram illustrating the calculation of the average receiving length provided in an embodiment of this application.

[0296] like Figure 12As shown, we currently assume there are 5 lexical sequences: 1201, 1202, 1203, 1204, and 1205. It's important to understand that these 5 lexical sequences are independent of each other; they are presented here for the sake of illustrating with multiple examples. In lexical sequence 1201, lexical a1 is the input lexical, and lexicals a2, a3, a4, a5, and a6 are lexicals inferred by the first model, corresponding to a first length of 5. In lexical sequence 1202, lexical b1 is the input lexical, and lexicals b2, b3, b4, and b5 are lexicals inferred by the first model, corresponding to a first length of 4. In lexical sequence 1203, lexical c1 is the input lexical, and lexicals c2, c3, c4, c5, c6, c7, c8, and c9 are lexicals inferred by the first model, corresponding to a first length of 8. The understanding of the other lexical sequences is similar and will not be elaborated upon here.

[0297] Based on the above description, it can be confirmed that after inputting these 5 word sequences into the second model for verification, the second model can accept the correctly verified word sequences and use them as subsequent word sequences in the input sequence. (Reference) Figure 12 For word sequence 1201, assuming word units a2, a3, and a4 are accepted by the second model, then its corresponding first acceptance length is determined to be 3. For word sequence 1202, assuming word units b2, b3, b4, and b5 are accepted by the second model, then its corresponding first acceptance length is determined to be 4. For word sequence 1203, assuming word units c2, c3, c4, c5, c6, c7, and c8 are accepted by the second model, then its corresponding first acceptance length is determined to be 7. The understanding of other word sequences is similar and will not be elaborated here.

[0298] After determining the first acceptance length for each of the multiple lexical sequences, the average length is calculated as the average acceptance length. For example... Figure 12 As shown, the formula for calculating the average received length is 20 / 5, which is 4.

[0299] After determining the average received length and the first length, the decoding speed can be determined based on the first delay, the second delay, the average received length, and the first length.

[0300] As an example, in one implementation, the decoding speed can be referenced from Formula 5 below:

[0301]

[0302] In Formula 5, N is the average receiving length of the first model, and T llm For the second delay, T draft For the first delay, N max Given the first length, () -1 A function can be understood as a function that takes the reciprocal.

[0303] Based on the above introduction, it can be determined that during the validation process of the second model, the second model needs to perform one inference on the input word sequence and verify multiple predicted words. In actual implementation, the time required for the validation process of the second model is relatively small and can be ignored. Therefore, the time consumed by the second model to validate the word sequence can be regarded as the latency of the second model performing one inference on the input words. Referring to Formula 1, T llm This can be understood as the time delay required for the second model verification process.

[0304] N max *T draft This can be understood as the first model inferring N through autoregression. max The latency required for each word. Then, T llm +N max *T draft This can be understood as the system decoding N max The time delay required for each word. (T) llm +N max *T draft Dividing by the average acceptance length N, we can understand it as the average time required for the system to obtain a correctly validated token. Therefore, we can understand that the average acceptance length N and T... llm +N max *T draft The ratio can be understood as the number of correctly deduced words that can be deduced within a 1-second time period. In other words, the decoding speed can be understood as the number of correctly deduced words that can be deduced within a 1-second time period.

[0305] In this implementation, the decoding speed is determined by comprehensively considering the first delay, the second delay, the average reception length, and the first length, which allows for a relatively effective evaluation of the efficiency of the current decoding process. By comparing the changes in decoding speed after different pruning rounds, it can be determined whether subsequent word pruning should be performed to improve the accuracy of word pruning.

[0306] Next, we will explain the implementation of vocabulary pruning in conjunction with another processing step. The following section focuses on... Figures 13 to 16 This section describes the workflow of a single vocabulary pruning process. Figure 13 This is a schematic diagram of a single vocabulary trimming process provided in an embodiment of this application. Figure 14 This is a schematic diagram of word frequency calculation provided in an embodiment of this application. Figure 15 This is a schematic diagram of vocabulary trimming provided in an embodiment of this application. Figure 16 This is a schematic diagram of the first adjustment parameter provided in an embodiment of this application. The process of a single vocabulary pruning operation may include, for example, the following steps:

[0307] S1301. The first vocabulary is cropped based on the first dataset to obtain the second vocabulary.

[0308] It is understandable that the lexical units in the vocabulary are the source for the second and first models to infer the next lexical unit. In order to minimize the impact on the inference of the next lexical unit while improving the inference speed of the first model, for example, infrequently used lexical units in the vocabulary can be discarded to reduce the computational burden on the first model.

[0309] Therefore, for example, in call summarization applications, text closely related to everyday call scenarios can be acquired to determine which lexical terms are common in call scenarios. To more comprehensively cover call scenarios, some text that, while not directly limited to calls, may be collected but potentially related to the call content can also be included. In this embodiment, the first dataset may contain such text.

[0310] In one implementation, for example, for any fourth word element in the first vocabulary, the frequency of its occurrence in the first dataset can be counted to determine the first word frequency corresponding to that fourth word element. In the first loop, the first vocabulary can be the original vocabulary. In subsequent loops, the first word can be the vocabulary obtained after the previous round of pruning.

[0311] Then, the first word frequencies corresponding to the fourth word elements in the first word list can be sorted in descending order to obtain the first sorting result. Based on the first sorting result, the fourth word elements whose first word frequencies are located in the first position interval in the first sorting result can be retained, and the fourth word elements whose first word frequencies are located in the second position interval in the second sorting result can be discarded to obtain the second word list.

[0312] In one implementation, a threshold p% can be preset, and the sorting result can be divided into two position intervals based on the preset threshold p%. These two position intervals are the first position interval and the second position interval. For example, the first position interval can be the interval from the first sorting position to the nth sorting position in the sorting result, and the second position interval can be the interval from the (n+1)th sorting position to the last sorting position in the sorting result. The nth sorting position is located at position p% in the first sorting result. Both n and p are integers greater than or equal to 1. If n or p does not meet the integer condition, the value of n or p can be modified by rounding up or down.

[0313] For example, assuming the sorted sequence includes 100 elements, and assuming the preset threshold p% is 50%, then the sorting position corresponding to the preset threshold 50% can be determined as the position of the 50th element.

[0314] The following example uses a preset threshold of 50%. Figure 14 and Figure 15 A brief explanation of the vocabulary pruning process is provided.

[0315] like Figure 14 As shown, suppose there exists a vocabulary 1401 containing 10 lexical elements: a, b, c, d, e, f, g, h, i, and j. Based on the above description, it can be determined that we need to count the frequency of each lexical element in the vocabulary in the first dataset.

[0316] The current assumption is that word a has a frequency of 120 in the first dataset, word b has a frequency of 24, word c has a frequency of 88, word d has a frequency of 98, word e has a frequency of 10, word f has a frequency of 65, word g has a frequency of 52, word h has a frequency of 80, word i has a frequency of 0, and word j has a frequency of 9.

[0317] After sorting the word frequencies corresponding to each word element in the vocabulary, we can obtain sorting result 1402. In sorting result 1402, the word frequency of word element a is located in the first sorting position, and the word frequency of word element j is located in the last sorting position.

[0318] After obtaining the first sorting result, the sorting position corresponding to the preset threshold can be determined, such as the 5th sorting position shown in the figure. Then, the fourth word elements from the first to the 5th sorting position can be retained, and the fourth word elements from the 6th to the last sorting position can be discarded to obtain the second vocabulary. (Refer to...) Figure 15 The trimmed second vocabulary 1501 can contain word unit a, word unit d, word unit c, word unit h and word unit f.

[0319] S1302. Update the first delay and average reception length based on the first vocabulary and the second vocabulary.

[0320] Based on the above introduction, it can be understood that after word list pruning, the computational load of the first model is reduced due to the reduction in the number of lexical units in the word list, which in turn leads to a reduction in the first latency.

[0321] In one implementation, the first delay can be updated based on the number of lexical units contained in the second vocabulary and the number of lexical units contained in the first vocabulary.

[0322] As an example, in one implementation, the update formula for the first delay can refer to Formula Six below:

[0323] T draft =σ*T draft Formula Six

[0324] In Formula 6, σ is the second adjustment parameter. In one implementation, the second adjustment parameter can be determined based on the number of lexical units contained in the second vocabulary and the number of lexical units contained in the first vocabulary. For example, assuming the second vocabulary contains 10,000 lexical units and the first vocabulary contains 130,000 lexical units, then the second adjustment parameter could be 10,000 / 130,000. It can be understood that the fewer lexical units contained in the second vocabulary, the smaller the second adjustment parameter, and therefore the smaller the first delay.

[0325] It's also understandable that, because some words in the vocabulary are pruned, these pruned words cannot be inferred as potential next words in the first model's inference process. In this case, if the pruned words originally had a high probability of becoming the next word, then the average acceptance length of the first model would also decrease.

[0326] For example, suppose that without lexicon pruning, a word A is input into the first model, and the first model can infer the next word as word B. After verification, word B is accepted by the second model. However, after pruning the vocabulary, if the vocabulary no longer contains word B, the first model will be unable to infer word B. In this case, the first model might infer other words besides word B, such as word C. But since the second model still infers word B as the next word in subsequent verification, the second model will consider word C to be incorrect and reject it. In this case, the average accepted length of the first model may decrease.

[0327] In one implementation, the average acceptance length can be updated based on the total frequency of the first term corresponding to the second term and the total frequency of the second term corresponding to the second term.

[0328] As an example, in one implementation, the update formula for the average accepted length N can be referenced as Formula Seven below:

[0329] N = θ * N (Formula 7)

[0330] In Formula 7, θ is the first adjustment parameter. In one implementation, the first adjustment parameter can be determined, for example, based on the total second word frequency corresponding to the second vocabulary and the total first word frequency corresponding to the first vocabulary. The total second word frequency is the sum of the word frequencies corresponding to each word element in the second vocabulary, and the total first word frequency is the sum of the word frequencies corresponding to each word element in the first vocabulary.

[0331] The following is combined Figure 16 The calculation of the first adjustment parameter θ is introduced.

[0332] like Figure 16 As shown, there are currently two word lists: a first word list 1401 and a second word list 1601. The first word list 1401 can contain word elements a, b, c, d, e, f, g, h, i, and j, and a word frequency can be calculated for each word element. The second word list 1601 can contain word elements a, c, d, f, and h, and a word frequency can be calculated for each word element.

[0333] Based on the above introduction, we can determine that the formula for calculating the first total word frequency is 120+98+88+80+65+52+24+10+9+0, which is 546 as shown in the figure. The formula for calculating the second total word frequency is 120+98+88+80+65, which is 451 as shown in the figure. Therefore, we can determine that the first adjustment parameter θ is 451 / 546.

[0334] It is understandable that a decrease in the sum of the frequencies of each word element in the second vocabulary will lead to a decrease in the first adjustment parameter, which in turn will lead to a decrease in the average acceptance length.

[0335] S1303. After determining the updated first delay and the updated average receiving length, the updated decoding speed can be determined according to Formula 5, the second delay, the updated first delay, the first length, and the updated average receiving length.

[0336] Thus, by pruning the first vocabulary based on decoding speed, words that contribute little to decoding can be removed, thereby improving overall decoding efficiency. During vocabulary pruning, frequently occurring words are retained, which helps maintain decoding accuracy. The vocabulary pruning is performed based on the first dataset (containing multiple texts), meaning the target vocabulary can be optimized for specific datasets or tasks, thereby improving the model's adaptability when handling specific types of text.

[0337] The pruned target vocabulary is usually smaller than the original first vocabulary, which means that the number of words that need to be loaded and processed during decoding is reduced, thereby helping to reduce memory usage, improve system resource utilization, and enhance the model's inference efficiency.

[0338] In summary, this method balances inference quality and efficiency by combining decoding speed optimization and vocabulary pruning. By improving decoding speed and optimizing the vocabulary, the system can reduce latency during inference and improve real-time performance.

[0339] Based on the above embodiments, the following is combined with... Figure 17 The text generation method provided in this application will be further explained. Figure 17 This is a flowchart illustrating the text generation method provided in an embodiment of this application. (Refer to...) Figure 17 The process includes:

[0340] S1701. Input the first feature information, the first instruction, the first word, and the first text into the first model, so that the first model infers the first word and obtains multiple second words output by the first model. The multiple second words correspond to their respective first prediction probabilities. The first prediction probabilities are used to indicate the probability that the second word is the next word of the first word. The first feature information is used to reflect the feature information of the first text, the feature information of the first instruction, and the feature information of at least one historical word obtained by the first model inference. The first instruction is used to instruct the extraction of first type of information from the first text.

[0341] For example, the specific implementation of the current step can be referred to the above content, and will not be repeated here.

[0342] S1702. For any one of the multiple second word elements, determine the first acceptance probability corresponding to the second word element based on the first prediction probability corresponding to the second word element. The first acceptance probability is used to indicate the probability that the second word element is accepted by the second model. The second model is used to verify the word elements output by the first model.

[0343] First, based on the first predicted probability corresponding to the second word, the first confidence level corresponding to the second word is determined, and the first confidence level is proportional to the first predicted probability.

[0344] Then, based on the first confidence level corresponding to the second word and the preset mapping relationship, the first acceptance probability corresponding to the second word is determined. The preset mapping relationship is used to indicate the correspondence between the confidence level and the acceptance probability.

[0345] For example, the specific implementation of the current step can be referred to the above content, and will not be repeated here.

[0346] S1703. Compare the first acceptance probability corresponding to the second word with the first threshold to determine whether to perform inference for the second word.

[0347] First, based on the first delay and the second delay, a first threshold is determined. The first delay is the time required for one inference by the first model, and the second delay is the time required for one inference by the second model.

[0348] In one implementation, determining a first threshold based on a first delay and a second delay includes: determining the difference between the first delay and the second delay, and determining the first threshold based on the difference and the first delay; wherein the first threshold is directly proportional to the first delay and inversely proportional to the difference.

[0349] Then, the first predicted probability corresponding to the second word is compared with the first threshold to determine whether to perform inference for the second word.

[0350] If the first acceptance probability is greater than the first threshold, the second feature information, the first instruction and the second word are input into the first model so that the first model can reason about the second word and obtain multiple third words output by the first model. The multiple third words correspond to their respective second prediction probabilities. The second prediction probabilities are used to indicate the probability that the third word is the next word of the second word.

[0351] The second feature information is used to reflect the feature information of the first text, the feature information of the first instruction, and the feature information of at least one historical word element obtained by reasoning from the first model and the second model. The first instruction is used to instruct the extraction of a first type of information from the first text.

[0352] For example, the specific implementation of the current step can be referred to the above content, and will not be repeated here.

[0353] In summary, this embodiment introduces the concept of a first confidence level, allowing for a more refined evaluation of the quality of the second lexical unit. Furthermore, a mapping relationship is established between the first confidence level and the first acceptance probability of a lexical unit. Using this mapping relationship, the first acceptance probability of a lexical unit can be quantitatively determined based on its first confidence level. Further, the first acceptance probability of each lexical unit is compared with a set first threshold to determine whether further reasoning is needed for that lexical unit. This allows for the effective selection of appropriate lexical units for reasoning, thereby improving the flexibility of lexical unit reasoning. Simultaneously, by timely filtering and discarding of reasoned lexical units, unnecessary computation can be reduced, improving the efficiency of the first model during the reasoning process, thereby reducing the overall system latency and increasing response speed.

[0354] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0355] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0356] The text generation method of this application has been described above. The apparatus for performing the above method provided in this application is described below. Those skilled in the art will understand that the methods and apparatus can be combined and referenced with each other, and the related apparatus provided in this application can perform the steps in the above text generation method.

[0357] The text generation method provided in this application can be applied to electronic devices with data processing capabilities. Electronic devices include terminal devices, and the specific device form of the terminal device can be referred to the above-mentioned descriptions, which will not be repeated here.

[0358] In one implementation, this application provides an electronic device. Figure 18 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0359] like Figure 18 As shown, the electronic device 1800 includes: a processor 1801 and a memory 1802; the memory 1802 stores computer execution instructions; the processor 1801 executes the computer execution instructions stored in the memory 1802, causing the electronic device 1800 to perform the above-described method.

[0360] When the memory 1802 is set up independently, the electronic device also includes a bus 1803 for connecting the memory 1802 and the processor 1801.

[0361] This application provides a chip. The chip includes a processor, which is used to call a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.

[0362] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0363] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0364] This application provides a computer program product, which includes a computer program that, when run, causes the computer to perform the above-described method.

[0365] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0366] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A text generation method, characterized in that, include: The first feature information, the first instruction, the first word element, and the first text are input into the first model so that the first model infers the first word element and obtains a plurality of second word elements output by the first model. The plurality of second word elements correspond to their respective first prediction probabilities. The first prediction probabilities are used to indicate the probability that the second word element is the next word of the first word element. The first feature information is used to reflect the feature information of the first text, the feature information of the first instruction, and the feature information of at least one historical word element obtained by the first model inference. The first instruction is used to instruct the extraction of a first type of information from the first text. For any one of the plurality of second word elements, a first acceptance probability is determined based on the first prediction probability corresponding to the second word element. The first acceptance probability is used to indicate the probability that the second word element is accepted by the second model. The second model is used to verify the word elements output by the first model. The first acceptance probability corresponding to the second word is compared with the first threshold to determine whether to perform inference for the second word.

2. The method according to claim 1, characterized in that, Based on the first predicted probability corresponding to the second word, the first acceptance probability corresponding to the second word is determined, including: Based on the first predicted probability corresponding to the second word, a first confidence level is determined for the second word, wherein the first confidence level is proportional to the first predicted probability; Based on the first confidence level corresponding to the second term and the preset mapping relationship, the first acceptance probability corresponding to the second term is determined, wherein the preset mapping relationship is used to indicate the correspondence between the confidence level and the acceptance probability.

3. The method according to claim 1 or 2, characterized in that, The step of comparing the first predicted probability corresponding to the second word with the first threshold to determine whether to perform inference for the second word includes: The first threshold is determined based on the first delay and the second delay, wherein the first delay is the time required for one inference by the first model, and the second delay is the time required for one inference by the second model; The first predicted probability corresponding to the second word is compared with the first threshold to determine whether to perform inference for the second word.

4. The method according to claim 3, characterized in that, Determining the first threshold based on the first delay and the second delay includes: Determine the difference between the first delay and the second delay, and determine the first threshold based on the difference and the first delay; The first threshold is directly proportional to the first time delay and inversely proportional to the difference.

5. The method according to claim 3 or 4, characterized in that, The method further includes: When the first acceptance probability is greater than the first threshold, the second feature information, the first instruction, and the second word are input into the first model so that the first model can reason about the second word to obtain multiple third words output by the first model. The multiple third words correspond to their respective second prediction probabilities, and the second prediction probabilities are used to indicate the probability that the third word is the next word of the second word. The second feature information is used to reflect the feature information of the first text, the feature information of the first instruction, and the feature information of at least one historical word element obtained by the first model and the second model. The first instruction is used to instruct the extraction of a first type of information from the first text.

6. The method according to any one of claims 3-5, characterized in that, The first model performs inference on the first lexical unit, including: The first model infers the first word element based on the target vocabulary; The method further includes: A first decoding speed is determined based on the first delay, the second delay, the average acceptance length, and the first length. The first length is the number of inferences performed by the first model, and the average acceptance length is the average length of the tokens output by the first model that are accepted by the second model. The first delay, the second delay, and the first length are all inversely proportional to the first decoding speed, and the average acceptance length is directly proportional to the first decoding speed. The first vocabulary is truncated based on the first decoding speed and the first dataset to obtain the target vocabulary, which contains multiple texts in the first dataset.

7. The method according to claim 6, characterized in that, Determining the first decoding speed based on the first delay, the second delay, the average reception length, and the first length includes: The first product is calculated based on the first length and the first time delay; The first value is determined by summing the first product and the second delay; The first decoding speed is determined based on the ratio of the average received length to the first value.

8. The method according to claim 6 or 7, characterized in that, The step of pruning the first vocabulary based on the first decoding speed and the first dataset to obtain the target vocabulary includes: The first vocabulary is trimmed based on the first dataset to obtain the second vocabulary; Update the first delay and the average reception length based on the first vocabulary and the second vocabulary; The second decoding speed is determined based on the second delay, the updated first delay, the first length, and the updated average receiving length; If the second decoding speed is greater than the first decoding speed, the second vocabulary is pruned according to the first dataset to obtain a third vocabulary, and it is determined whether to perform the next round of pruning based on the decoding speed corresponding to the third vocabulary. If the second decoding speed is less than or equal to the first decoding speed, the vocabulary obtained after the previous round of pruning is determined as the target vocabulary.

9. The method according to claim 8, characterized in that, The step of pruning the first vocabulary based on the first dataset to obtain the second vocabulary includes: For any fourth word element in the first vocabulary, determine the first word frequency of the fourth word element in the first dataset; The first word frequencies corresponding to each of the multiple word elements in the first vocabulary are sorted to obtain a first sorting result; In the first sorting result, the word elements whose first word frequency is located in the first position interval are retained to obtain the second word list. The first position interval is the interval located before the nth sorting position. The nth sorting position is located at position p% in the first sorting result. Both n and p are integers greater than or equal to 1.

10. The method according to claim 8 or 9, characterized in that, The step of updating the average acceptance length based on the first vocabulary and the second vocabulary includes: The total first word frequency corresponding to the first word list is determined based on the sum of the word frequencies corresponding to each word element in the first word list. The total second word frequency corresponding to the second word list is determined based on the sum of the word frequencies corresponding to each word element in the second word list. The first adjustment parameter is determined based on the ratio of the second total word frequency to the first total word frequency. The average acceptance length is updated based on the first adjustment parameter.

11. The method according to claim 8 or 9, characterized in that, The step of updating the first time delay based on the first vocabulary and the second vocabulary includes: The second adjustment parameter is determined based on the ratio of the number of lexical units contained in the second vocabulary to the number of lexical units contained in the first vocabulary; The first delay is updated according to the second adjustment parameter.

12. The method according to any one of claims 1-11, characterized in that, The method further includes: The first text and the first instruction are input into the second model to obtain the first word and the third feature information output by the second model. The third feature information is used to reflect the feature information of the first text and the feature information of the first instruction.

13. The method according to any one of claims 1-12, characterized in that, The method further includes: At least one word to be verified is input into the second model so that the second model can reason about the at least one word to be verified to obtain the second reasoning word corresponding to each of the at least one word to be verified. The word to be verified is the word output by the reasoning of the first model. For any one of the at least one to-be-verified lexical units, if the second inference lexical unit output by the second model is the same as the first inference lexical unit inferred by the first model for the to-be-verified lexical unit, then the verification result of the to-be-verified lexical unit is determined to be verified as passed; or, If the second inference term output by the second model is different from the first inference term obtained by the first model for the term to be verified, the verification result of the term to be verified is determined to be verification failure. Based on the verified tokens that have passed the verification, the target information extracted from the first text is determined.

14. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 13.

15. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 13.

17. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 13.

Images