Large language model-based push decoding method and device, and computer device

Abstract

The present disclosure relates to a large language model-based prediction decoding method and device and a computer device. The method comprises: obtaining a text sequence input by a user; performing a decoding operation on the text sequence by using a text generation model to generate a first word piece, and sending the first word piece and the text sequence to a text prediction model; predicting a second word piece from the first word piece and the text sequence by using the text prediction model, and sending the second word piece to the text generation model; verifying the second word piece by using the text generation model, and if the second word piece is verified, concatenating the first word piece and the second word piece into a first output sequence in a preset order by using the text generation model; and feeding back the first output sequence to the user. Thus, parallel pipeline execution of decoding and prediction is realized, and the step-by-step waiting delay of autoregressive decoding is effectively reduced.

Description

Technical Field

[0001] This disclosure relates to the field of natural language processing, and in particular to a reasoning decoding method, apparatus, and computer device based on a large language model. Background Technology

[0002] In related technologies, large language models (LLMs) obtained based on the Transformer architecture have been widely used in tasks such as natural language processing, dialogue systems, and real-time translation. They are usually deployed using graphics processing units (GPUs) or GPU clusters for text inference and use an autoregressive approach to generate output step by step.

[0003] However, since each output token depends on the previous output token, the decoding process must be executed sequentially. This results in low GPU utilization and high inference latency in single-user, single-request scenarios. To address this issue, although the overall throughput can be improved by merging the requests from multiple users into a batch, this often increases the queuing time for individual requests and sacrifices the response latency of a single task. Therefore, it is not suitable for latency-sensitive application scenarios such as real-time dialogue or online translation. Summary of the Invention

[0004] In view of this, the present disclosure provides a reasoning decoding method, apparatus, and computer device based on a large language model to solve the problems existing in the related art.

[0005] A first aspect of this disclosure provides an inference decoding method based on a large language model, applied to a server. The server is equipped with a first computing unit and a second computing unit. The first computing unit is used to run a text generation model, and the second computing unit is used to run a text prediction model. The method includes: acquiring a text sequence input by a user; performing a decoding operation on the text sequence using the text generation model to generate a first word, and sending the first word and the text sequence to the text prediction model; predicting a second word based on the first word and the text sequence using the text prediction model, and sending the second word to the text generation model; verifying the second word using the text generation model, and if the second word passes verification, concatenating the first word and the second word in a preset order to form a first output sequence using the text generation model; and feeding back the first output sequence to the user.

[0006] A second aspect of this disclosure provides an inference decoding device based on a large-scale language model, applied to a server. The server is equipped with a first computing unit and a second computing unit. The first computing unit is used to run a text generation model, and the second computing unit is used to run a text prediction model. The device includes: an acquisition module for acquiring a text sequence input by a user; a decoding module for performing a decoding operation on the text sequence using the text generation model to generate a first word element, and sending the first word element and the text sequence to the text prediction model; a prediction module for predicting a second word element based on the first word element and the text sequence using the text prediction model, and sending the second word element to the text generation model; a verification module for verifying the second word element using the text generation model, and if the second word element passes verification, concatenating the first word element and the second word element in a preset order to form a first output sequence using the text generation model; and a sending module for feeding back the first output sequence to the user.

[0007] A third aspect of this disclosure provides a computer device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described reasoning and decoding method based on a large language model.

[0008] A fourth aspect of this disclosure provides a computer program product that, when executed by a processor, implements the steps of the above-described inference decoding method based on a large language model.

[0009] The above-mentioned at least one technical solution adopted in the embodiments of this disclosure can achieve the following beneficial effects: by running the text generation model in the first computing unit and the text prediction model in the second computing unit, the parallel pipeline execution of decoding and prediction is realized, effectively reducing the stepwise waiting delay of autoregressive decoding; at the same time, through the prediction mechanism and the verification mechanism, the text generation model can skip repeated calculations when the prediction is correct, reducing computational overhead and improving inference throughput; at the same time, the verification mechanism ensures that all output tokens are confirmed by the text generation model, and the output quality is completely consistent with the text generation model, thereby ensuring the accuracy of the generated tokens; at the same time, the division of labor and cooperation of heterogeneous computing units also improves the hardware energy efficiency ratio, and has significant performance advantages and practical value in application scenarios with low latency and high real-time requirements. Attached Figure Description

[0010] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 A schematic diagram of a reasoning decoding based on a large language model is provided as an embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of a server provided in one embodiment of the present disclosure; Figure 3 A schematic diagram of the structure of an inference decoding device based on a large language model provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the structure of a computer system provided in an embodiment of the present disclosure; Figure 6 This is a schematic diagram of a computer program product provided according to an embodiment of the present disclosure. Detailed Implementation

[0012] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0013] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0014] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0015] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0016] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0017] In recent years, large language models (LLMs) obtained based on the Transformer architecture have been widely used in tasks such as natural language processing, dialogue systems, and real-time translation. They are typically deployed using graphics processing units (GPUs) or GPU clusters for text inference, and employ an autoregressive approach to progressively generate output. However, since each output token depends on the tokens from the previous output, the decoding process must be executed sequentially. This leads to low GPU utilization and high inference latency in single-user, single-request scenarios. While merging requests from multiple users into a single batch can improve overall throughput, this often increases the queuing time for individual requests and sacrifices the response latency of a single task. Therefore, it is not suitable for latency-sensitive applications such as real-time dialogue or online translation.

[0018] Besides optimizing throughput through scheduling strategies, related technologies have also proposed various optimization methods based on the LLM's own structure. For example, they reduce the computational overhead of each decoding through model compression, pruning, quantization, or early exit. However, model compression and pruning require changing the original network structure of the model, quantization reduces the numerical precision of weights and activation values, and early exit may truncate the model's proper inference depth. It is evident that these methods all sacrifice speed for changes in model structure or reduced computational precision, making it difficult to effectively reduce latency while maintaining the original model's output quality.

[0019] In summary, existing technologies that optimize throughput through scheduling strategies are limited by the sequential nature of autoregressive decoding, making it difficult to improve inference speed while maintaining single-task response latency. Furthermore, acceleration methods through model structure optimization inevitably involve modifications to the original LLM, leading to output distribution deviations and impacting the quality of the final LLM output. Moreover, related technologies all use the GPU as the sole computing core to handle all decoding tasks. This means that GPU computing power cannot be fully utilized when user requests are limited, resulting in idle hardware resources and unnecessary energy consumption due to the high power consumption of GPUs.

[0020] The inference and decoding method based on a large language model provided in this disclosure can be executed by a terminal or by a chip applied to the terminal.

[0021] For example, the aforementioned terminals may include one or more of the following: mobile phones, tablets, wearable devices, in-vehicle devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, handheld computers (PDAs), and wearable devices based on augmented reality (AR) and / or virtual reality (VR) technologies. They may also include, but are not limited to, remote control devices, wearable devices, streetlights, home appliances, and other smart terminals. This disclosure does not impose specific limitations on these aspects.

[0022] Figure 1 This is a schematic diagram illustrating a reasoning decoding process based on a large language model, as provided in one embodiment of this disclosure. Figure 1 As shown, this is applied to a server, which is equipped with a first computing unit and a second computing unit. The first computing unit is used to run a text generation model, and the second computing unit is used to run a text prediction model, including: S101, Obtain the text sequence input by the user.

[0023] Figure 2 This is a schematic diagram of the structure of a server provided in one embodiment of this disclosure. Figure 2 As shown, the server can be deployed with a first computing unit and a second computing unit. The first computing unit can be a GPU (Graphics Processing Unit), and it runs a text generation model. This text generation model can be a large language model (LLM) based on the Transformer architecture, used to perform the main computational tasks of inference and decoding. The second computing unit can be a FPGA (Field Programmable Gate Array), and it runs a text prediction model. This text prediction model can be a Transformer model with a similar structure to the text generation model but a smaller scale, used to perform subsequent word prediction tasks. The Transformer architecture is a neural network architecture based on a self-attention mechanism, which captures contextual semantics by computing the dependencies between all positions in a sequence in parallel, unlike traditional recurrent neural networks which must process sequentially.

[0024] In some embodiments, the first computing unit and the second computing unit can establish a point-to-point communication connection via a Peripheral Component Interconnect Express (PCIe) bus. This point-to-point communication means that the first computing unit and the second computing unit, as two peer devices, directly exchange data via the PCIe bus. The entire data transmission process does not require CPU scheduling intervention, nor does it require data to be temporarily stored in other storage spaces before being read by the other party. That is, data is directly transmitted from the storage space of the first computing unit to the storage space of the second computing unit via the PCIe bus, or from the storage space of the second computing unit to the storage space of the first computing unit via the PCIe bus. The entire transmission path does not involve the participation of other storage spaces, thereby effectively avoiding the latency overhead caused by multiple data copies and host memory bandwidth contention in the transmission path in related technologies. This significantly reduces the communication latency between the first computing unit and the second computing unit, ensuring high-bandwidth, low-latency data interaction capabilities and providing a reliable hardware foundation for frequent data exchange between the two in real-time inference scenarios. It should be noted that the PCIe bus is a high-speed serial bus standard based on point-to-point serial communication. It adopts a layered protocol architecture and provides a high-bandwidth, low-latency data interconnection channel between the central processing unit (CPU) and peripheral devices through a multi-channel parallel transmission mechanism. It is currently the mainstream high-performance I / O expansion bus in computer and server systems.

[0025] In some embodiments, users can interact through applications installed on mobile devices (e.g., smart customer service apps, translation tools, or document processing software). The application's front-end interface provides a text input interface (e.g., a dialog input box, a translation input box, or a document upload portal). Users can enter text content in the text input interface and click the send control on the text input interface. The application responds to the user's click of the send control by acquiring the text sequence and sending the text sequence to the server. After receiving the text sequence, the server can use it as input to the text generation model of the first computing unit deployed on the server for subsequent inference and decoding.

[0026] S102, the text generation model is used to perform a decoding operation on the text sequence to generate the first word, and the first word and the text sequence are sent to the text prediction model.

[0027] In some embodiments, the text generation model running on the first computing unit can perform a decoding operation on the text sequence input by the user after acquiring the text sequence. Taking the user input "What will the temperature be like tomorrow?" as an example, the text generation model can first encode the text sequence, converting it into a vector containing semantic information to capture the contextual semantics of the text sequence; then, it performs autoregressive decoding based on the vector. In the first decoding step, the text generation model can calculate the probability distribution of each candidate word in the first vocabulary based on the vector of the text sequence through forward inference, and select the word with the highest probability as the output of the first decoding step to generate the first word (in the above example, the first word can be "temperature").

[0028] After the first word is generated, the first computing unit can send the generated first word and the original text sequence to the second computing unit via the PCIe bus. This allows the text prediction model deployed in the second computing unit to update its internal state based on the original text sequence and the generated first word, and predict the next word accordingly. In this process, the sent original text sequence provides the text prediction model with complete contextual information, while the sent first word provides the model with the current decoding progress, serving as input for predicting the next word. Together, they ensure that the text prediction model can make accurate predictions while maintaining state synchronization with the text generation model.

[0029] In this way, through data interaction between the first and second computing units on the PCIe bus, on the one hand, the original text sequence provides the text prediction model with a complete global context, enabling the model to accurately grasp user intent and dialogue background; on the other hand, the currently generated first word unit serves as a fast synchronization signal for the incremental state, allowing the text prediction model to obtain the latest decoding progress and semantic evolution path in real time. Both together ensure state synchronization between the text prediction model and the text generation model, thus achieving an efficient advance prediction and verification mechanism. This working mode based on dual-model state collaboration maintains the stability and accuracy of the text generation model's output while effectively reducing the cumulative latency of single-step waiting in autoregressive decoding through advance prediction, thereby significantly improving decoding efficiency while ensuring inference quality.

[0030] S103, using a text prediction model to predict a second word based on the first word and the text sequence, and sending the second word to the text generation model.

[0031] In some embodiments, continuing the previous example, the original text sequence received by the second computing unit is "How's the temperature tomorrow?", and the first word element is "temperature". Based on this, the text prediction model can use both the original text sequence and the first word element as input to predict subsequent words. In this process, the text prediction model can first encode the original text sequence to obtain a complete contextual semantic representation, and then use the first word element (temperature) as the currently generated word element, inputting it into the decoder of the text prediction model for autoregressive forward inference. The decoder of the text prediction model calculates the probability distribution of each candidate word in the second vocabulary and selects the word element with the highest probability as the prediction result, i.e., the second word element, for example, "moderate". After the second word element (moderate) is generated, the second computing unit can send this second word element to the first computing unit via the PCIe bus for the text generation model to verify the accuracy of the second word element.

[0032] In some embodiments, the text generation model running on the first computing unit may be configured with a first vocabulary, and the text prediction model running on the second computing unit may be configured with a second vocabulary. The first vocabulary and the second vocabulary may be the same, so as to ensure that the second word token predicted by the text prediction model can be compared and verified with the word token generated by the text generation model under the same word token numbering system.

[0033] In this way, after the text generation model generates the first word, it can send the first word and the original text sequence to the text prediction model, enabling the text prediction model to predict the second word in advance. This achieves efficient collaboration between the text generation and text prediction models. Simultaneously, by having the text prediction model perform the prediction task, the prediction process and the text generation model's own decoding computation can be pipelined in parallel. Their time consumption overlaps, effectively reducing the additional latency caused by waiting for prediction results at each step in the autoregressive decoding process, thus improving the overall efficiency of inference decoding. Furthermore, the text prediction model uses the same first vocabulary as the text generation model, ensuring that the words output by both models are based on the same numbering system. This allows for direct and accurate comparison and verification of the prediction results, further guaranteeing the reliability of the prediction and verification mechanism and providing a stable foundation for subsequent accelerated decoding.

[0034] S104, use a text generation model to verify the second word element, and if the second word element passes the verification, use the text generation model to concatenate the first word element and the second word element in a preset order to form the first output sequence.

[0035] In some embodiments, verifying the second word element using a text generation model includes: using the text generation model to perform a decoding operation on the text sequence based on the first word element to generate a third word element; comparing the second word element with the third word element, and determining that the second word element has passed verification if the comparison result shows that the second word element and the third word element are the same.

[0036] In some embodiments, continuing the previous example, after the text generation model running on the first computing unit generates the first word (temperature), it can send the first word (temperature) and the original text sequence (How's the temperature tomorrow?) together to the second computing unit so that the text prediction model running on the second computing unit can predict the second word. Simultaneously, the text generation model can continue to perform autoregressive decoding operations based on the first word (temperature) and the original text sequence without waiting for the prediction result from the text prediction model, to generate the third word. It should be noted that the decoding operation of the text generation model to generate the third word and the prediction operation of the text prediction model to generate the second word are executed in parallel.

[0037] Based on this, after the text prediction model completes its prediction, the second term (moderate) can be sent to the text generation model via the PCIe bus. If the text generation model then generates a third term that is also moderate, it can be determined that the second and third terms are identical, meaning the second term has passed verification and the prediction result of the text prediction model is acceptable. Subsequently, the text generation model can concatenate the first term (temperature) and the second term (moderate) in the order they were decoded to form the first output sequence, i.e., moderate temperature.

[0038] It should be noted that the embodiments of this disclosure use the two words "temperature" and "moderate" as examples for illustration only, in order to facilitate understanding of the technical principles of the embodiments of this disclosure, and do not constitute any limitation on the number of output words. In practical applications, the final output of the text generation model may contain more words (e.g., "moderate temperature, suitable for going out for a walk"). In this case, the text prediction model will continue to perform the above prediction operation until a complete answer sequence is generated.

[0039] In some embodiments, if the third word element generated by the text generation model itself is "moderate," while the second word element predicted by the text prediction model is "scientific," the text generation model confirms that the second and third word elements are different, i.e., the verification fails. In this case, the text generation model discards the second word element (scientific) predicted by the text prediction model, takes its own generated third word element (moderate) as the correct second word element, and concatenates it with the first word element (temperature) in the order they were generated during decoding to form "moderate temperature." Simultaneously, the correct third word element is fed back to the text prediction model so that the text prediction model updates its internal state based on the correct word element, ensuring the accuracy of subsequent predictions.

[0040] As can be seen, in this embodiment, after generating the first word, the text generation model can send the first word and the original text sequence to the text prediction model, which then begins to predict the second word. Simultaneously, the text generation model does not wait for the prediction result from the text prediction model, but immediately continues its own decoding calculation based on the first word and the original text sequence to generate the third word. In this way, the process of the text prediction model predicting the second word and the decoding process of the text generation model generating the third word are parallel in time, with their time consumption overlapping. When the text generation model completes the generation of the third word and receives the second word sent by the text prediction model, it can compare and verify the second and third words without waiting for additional prediction or decoding results, thus effectively reducing the time overhead caused by the cumulative delay of each step in the sequential execution method. Meanwhile, the verification is completed by comparing the prediction results of the text prediction model with the word units decoded by the text generation model itself. The prediction results are only adopted when the comparison is consistent, which ensures that every word unit in the output sequence is confirmed by the text generation model. This makes the final output completely consistent with the result of the independent decoding of the text generation model, thereby improving the inference decoding efficiency without sacrificing the generation quality.

[0041] S105, the first output sequence is fed back to the user.

[0042] In some embodiments, following the previous example, during the decoding process, the text generation model directly uses the word units predicted by the text prediction model whenever the verification passes, and uses the correct word units generated by its own decoding when the verification fails. This process is repeated until a complete end marker is generated or the preset maximum output length is reached, and finally a complete first output sequence is obtained, such as "moderate temperature".

[0043] Based on this, the text generation model uses the complete output sequence as the text content to be fed back. The server encapsulates this first output sequence into a response message and sends it to the client device over the network. Upon receiving the response message, the client parses it, extracts the text content corresponding to the output sequence, and displays it to the user through the application's front-end interface. For example, in a smart customer service scenario, if the user initially enters "How will the temperature be tomorrow?", the front-end interface will display "The temperature is moderate" as the response.

[0044] Therefore, after a user sends a text sequence through a mobile device, the heterogeneous computing of the server accelerates the inference, and the user can finally obtain an accurate and complete response on the front-end interface of the mobile device in a timely manner. This improves the user experience in real-time interactive scenarios while ensuring low-latency response and high-quality output.

[0045] In some embodiments, the first output sequence further includes a fourth word element. If the second word element verification passes, the text generation model concatenates the first and second word elements in a preset order to form the first output sequence. This includes: using a text prediction model to predict the fourth word element based on the second word element and the text sequence, and sending the fourth word element to the text generation model; and using the text generation model to concatenate the first, second, and fourth word elements into the first output sequence. Here, if the second word element verification passes, the text generation model pauses the decoding calculation of the fourth word element; if the verification fails, the decoding calculation of the fourth word element resumes.

[0046] In some embodiments, after the second word element passes validation, the text prediction model can predict the next candidate word element (the fourth word element) based on the context of the second word element and the original text sequence. Taking the user input "How's the weather today?" as an example, after the second word element "very" passes validation, the text prediction model continues to predict the next fourth word element (sunny) based on the original text sequence (How's the weather today?) and the first word element (weather) and the second word element (very), and sends this candidate word element to the text generation model. At this point, the text generation model can directly concatenate the first, second, and fourth word elements to generate the first output sequence (The weather is very sunny).

[0047] In other embodiments, the text generation model can also verify the fourth word element. After successful verification, it is concatenated with the already generated first, second, and fourth word elements in a preset order to obtain "the weather is very sunny". Here, the verification of the fourth word element does not have to be done in the same way as the initial verification (the second word element verification process). That is, the text generation model does not need to re-execute the complete decoding calculation to generate the actual word element and then compare it with the fourth word element. Instead, it can utilize the probability output characteristics of the text generation model itself to directly obtain the probability distribution of the predicted word element on the second vocabulary of the text generation model, and complete the verification by judging whether the probability meets the preset acceptance threshold.

[0048] For example, if the probability of the fourth word element reaches or exceeds a preset threshold, the verification passes, and the text generation model directly adopts the fourth word element as output without performing any decoding calculations. If the probability of the fourth word element is lower than the preset threshold, the verification fails, the text generation model discards the fourth word element, performs decoding calculations to generate the correct word element as output, and simultaneously feeds the correct word element back to the text prediction model to correct its internal state. Since subsequent verification only involves probability queries and threshold comparisons, and does not involve complete decoding calculations, and the computational cost of the verification process is much less than that of complete decoding, the average computational cost of each decoding step can be further reduced while ensuring output quality.

[0049] As can be seen, in this embodiment, if the second word token verification passes, the text generation model can simplify the verification process when verifying the fourth word token and pause the decoding calculation of the fourth word token, deciding whether to adopt it only by judging the probability threshold; if the second word token verification fails, the text generation model resumes the complete decoding calculation of the fourth word token, regenerates the correct word token, and feeds the correct word token back to the text prediction model to correct its internal state. In short, subsequent verifications can be simplified when the previous verification passes, and subsequent verifications need to resume complete decoding when the previous verification fails.

[0050] This dynamically adjusted verification strategy further reduces the average computational cost of each decoding step while ensuring output quality. Specifically, when subsequent word verification only requires probability threshold judgment, the text generation model does not need to perform the most time-consuming matrix multiplication and attention calculation processes in autoregressive decoding, but only needs to perform lightweight probability query and numerical comparison operations. When prediction deviation occurs, although resuming full decoding calculation introduces additional computational overhead, since full decoding is only triggered in the few cases of verification failure, the text generation model can still significantly reduce the computational load by simplifying the verification process when most verifications pass. Therefore, this strategy effectively reduces the average computational cost of each word in the decoding process in an overall statistical sense, especially in scenarios with high prediction accuracy, where the acceleration effect is particularly significant.

[0051] In some embodiments, if the second word verification fails, the text generation model performs a decoding operation based on the first word and the text sequence to generate a fifth word; the fifth word is sent to the text prediction model to update the context information used by the text prediction model to generate the next word; the text generation model concatenates the first word and the fifth word into a second output sequence, and the second output sequence is fed back to the user.

[0052] In some embodiments, the text generation model can feed back the verification result of the second word to the text prediction model: if the second word passes the verification, the text prediction model is only notified that the prediction is correct, and the text prediction model continues to predict subsequent words based on the current internal state; if the second word fails the verification, in subsequent steps, the text generation model will continue to perform decoding operations based on the first word, the correct second word, and the text sequence to generate the correct word (i.e., the fifth word), and feed the fifth word back to the text prediction model, so that the text prediction model updates its internal state based on the correct fifth word before proceeding to the next prediction step.

[0053] At this point, the text prediction model can predict the sixth word based on the correct fifth word and send it to the text generation model. The text generation model receives the sixth word and verifies it. If the verification passes, it adopts the prediction and concatenates the words; if the verification fails, it discards the prediction and uses its own decoding result. This process iterates repeatedly, with each word undergoing the same prediction and verification process, until an end marker is generated or the preset maximum output length is reached, ultimately resulting in the complete first output sequence.

[0054] In some embodiments, there are multiple users and multiple text sequences. The method includes: using a text generation model to perform a decoding operation on each text sequence to generate a first word element corresponding to each text sequence, and sending multiple text sequences and the first word element corresponding to each text sequence to a text prediction model; using the text prediction model to predict a second word element based on each text sequence and the corresponding first word element, and sending multiple second word elements to the text generation model; using the text generation model to verify each second word element, and if the second word element passes verification, using the text generation model to concatenate the verified second word element with the corresponding first word element to form a first output sequence; and feeding back the first output sequence to the corresponding user.

[0055] In some embodiments, the server can simultaneously receive text sequences sent by multiple mobile devices. For example, if three users simultaneously initiate requests, user A inputs "What will the temperature be like tomorrow?", user B inputs "Is this movie good?", and user C inputs "How is this book?". Based on this, the server can send these text sequences to a first computing unit, where a text generation model running on the first computing unit performs a decoding operation on each text sequence to generate a first word unit corresponding to each text sequence.

[0056] For example, the first word element "temperature" is generated for user A's text sequence, "movie" for user B's text sequence, and "content" for user C's text sequence. After generating the first word elements for each text sequence, the text generation model can send multiple text sequences and their corresponding first word elements to the second computing unit via the PCIe bus.

[0057] After receiving the multiple text sequences, the second computing unit runs a text prediction model that can predict the second term based on each text sequence and its corresponding first term. For example, the text prediction model predicts the second term "moderate" based on user A's "How will the temperature be tomorrow?" and "temperature"; it predicts "good" based on user B's "Is this movie good?" and "movie"; and it predicts "wonderful" based on user C's "How is this book?" and "content". After prediction, the text prediction model sends multiple second terms to the text generation model via the PCIe bus. It should be noted that the prediction process for multiple text sequences can be performed in parallel using batch processing, meaning multiple text sequences share the same model parameters and perform forward inference simultaneously, thus fully utilizing the parallel computing capabilities of the second computing unit and improving overall throughput.

[0058] In some embodiments, while the text prediction model predicts the second word, the text generation model running on the first computing unit also performs parallel decoding calculations on each text sequence to generate the third word corresponding to each text sequence for subsequent verification. Thus, after the text prediction model completes its prediction, multiple second words can be sent to the text generation model, which then verifies each second word individually.

[0059] For each text sequence, if the second word element passes validation, the text generation model concatenates the validated second word element with the corresponding first word element to form the first output sequence. For example, user A's first word element "temperature" is concatenated with the second word element "moderate" to form "temperature moderate"; user B's first word element "movie" is concatenated with the second word element "good-looking" to form "movie good-looking"; and user C's first word element "content" is concatenated with the second word element "wonderful" to form "content wonderful". If validation fails, the text generation model discards the corresponding second word element, uses its own decoded third word element as the correct second word element for concatenation, and feeds the correct word element back to the text prediction model to correct its internal state, ensuring the accuracy of subsequent predictions for each text sequence.

[0060] After concatenating the first output sequence, if the response requires multiple terms, the above process continues to loop until each text sequence generates an end marker or reaches the preset maximum output length, ultimately obtaining the complete output sequence corresponding to each text sequence. Subsequently, the server feeds back the first output sequence corresponding to each text sequence to the corresponding user, that is, displays the respective response content on the front-end interface of each mobile device.

[0061] In this way, through the collaborative operation of the first and second computing units, efficient response to concurrent requests from multiple users is achieved. On one hand, the text prediction model on the second computing unit uses batch processing to perform forward inference on multiple text sequences in parallel. By leveraging the parallel characteristics of model parameter sharing and matrix operations, the total prediction time for multiple requests is significantly shortened, and the system throughput is significantly improved. On the other hand, the text generation model on the first computing unit continues to maintain the decoding progress of each text sequence while interacting with the second computing unit, so that the verification process and the decoding process overlap, forming a multi-parallel pipeline operation mode. Compared with the traditional approach of queuing and processing multiple user requests sequentially, this embodiment effectively reduces the average processing latency in multi-user concurrent scenarios while ensuring the quality of single-path responses. It is particularly suitable for large-scale application scenarios such as online customer service and real-time Q&A that require simultaneous response to a large number of user requests.

[0062] In some embodiments, the first output sequence further includes a fourth word element, and the method further includes: using a text prediction model to simultaneously predict a second word element and a fourth word element based on the first word element and the text sequence, and sending the second word element and the fourth word element to a text generation model; using the text generation model to verify the second word element and the fourth word element in sequence, and if both the second word element and the fourth word element pass the verification, using the text generation model to concatenate the first word element, the second word element, and the fourth word element in a preset order to form a first output sequence.

[0063] In some embodiments, the text prediction model described above can continuously generate n candidate words in a single forward inference, and send all n candidate words to the text generation model for batch verification. For example, when predicting the second word word mentioned above, the fourth word word can be predicted simultaneously, and the second and fourth word words can be sent to the text generation model for verification in sequence. If both the second and fourth word words pass verification, the text generation model concatenates the first, second, and fourth word words into a first output sequence in a preset order. Here, n is a positive integer, and n≥1, representing the length of a single prediction window. The value of n determines the upper limit of the acceleration effect; the larger the value, the more decoding steps can be skipped at once when the prediction is accurate, and the more significant the acceleration effect. It should be noted that the prediction window length n represents the number of words continuously generated by the text prediction model in a single forward inference.

[0064] In some embodiments, the difficulty of predicting lexical terms varies significantly at different positions during the inference and decoding process. Specifically, for positions with high contextual certainty, such as everyday expressions and common sentence structures, prediction accuracy is high, and a larger prediction window can bring significant skipping benefits. However, for positions with low contextual certainty, such as those with dense technical terms and complex logical reasoning, prediction accuracy is low. If a large prediction window is still used, most predicted lexical terms will be rejected. In this case, the text prediction model performs a large amount of invalid forward inference and transmits invalid data via the PCIe bus, which becomes a system overhead. Therefore, a fixed prediction window length cannot achieve optimal results in different contexts simultaneously, and the value of n needs to be dynamically adjusted according to the runtime context state.

[0065] In some embodiments, the core logic of adjusting the n-value is to expand the prediction window to pursue greater acceleration benefits when the prediction quality is high, and to shrink the prediction window to avoid resource waste when the prediction quality is low. Prediction quality can be judged using various metrics. One approach is based on prediction confidence: each time the text prediction model generates a candidate word, it simultaneously records the prediction probability value of that word as a confidence score. If the confidence scores of multiple consecutive words within the historical prediction window are all higher than a first preset threshold, it indicates that the current context is in a high-certainty region, and the n-value is increased; if they are lower than a second preset threshold, it indicates that it is in a low-certainty region, and the n-value is decreased. Another approach is based on historical acceptance rate: the text generation model records the acceptance rate of words predicted by the text prediction model within the historical prediction window, which is the ratio of the number of words that passed verification (the predicted word matches the word generated by its own decoding) to the total number of words submitted in that batch. This acceptance rate is fed back to the text prediction model; when the acceptance rate is higher than a first acceptance rate threshold, the n-value is increased; when it is lower than a second acceptance rate threshold, the n-value is decreased. Another approach is based on system latency budgeting, which involves determining the initial value of n based on the target latency constraints of the current inference task and the average time taken for single-step decoding of the text generation model, and then making adaptive adjustments based on actual time fluctuations during the decoding process.

[0066] It should be noted that the above-mentioned dynamic adjustment methods can be used individually or in combination. For example, a latency budget can be used as a hard constraint to determine the upper limit of the value of n, while fine-tuning can be performed based on confidence or acceptance rate within the upper limit. Through the above dynamic adjustment mechanism, the embodiments of this disclosure can achieve an optimal balance between latency and efficiency in different types of input text and context environments, maintaining stable and efficient inference and decoding performance.

[0067] The foregoing primarily describes the solutions provided by the embodiments of this disclosure from the perspective of the server. It is understood that, in order to implement the above functions, the server includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0068] This disclosure embodiment can divide the server into functional units according to the above method example. For example, it can divide each function into separate functional modules, or it can integrate two or more functions into one management module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this disclosure embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0069] By dividing each functional module according to its corresponding function, an exemplary embodiment of this disclosure provides an inference decoding device based on a large language model, which can be a server or a chip applied to a server. Figure 3 This is a schematic diagram of the structure of an inference decoding device based on a large language model, provided as an embodiment of this disclosure. Figure 3 As shown, the inference decoding device 300 based on a large language model is applied to a server. The server is equipped with a first computing unit and a second computing unit. The first computing unit is used to run a text generation model, and the second computing unit is used to run a text prediction model. The inference decoding device 300 based on a large language model includes: The acquisition module 301 is used to acquire the text sequence input by the user; The decoding module 302 is used to perform a decoding operation on the text sequence using the text generation model to generate a first word element, and send the first word element and the text sequence to the text prediction model; Prediction module 303 is used to predict a second word based on the first word and the text sequence using the text prediction model, and send the second word to the text generation model; The first verification module 304 is used to verify the second word element using the text generation model, and if the second word element passes the verification, to use the text generation model to concatenate the first word element and the second word element in a preset order to form a first output sequence. Feedback module 305 is used to feed back the first output sequence to the user.

[0070] In an optional manner, the first verification module 304 is further configured to use the text generation model to perform a decoding operation on the text sequence based on the first word element to generate a third word element; compare the second word element with the third word element, and if the comparison result determines that the second word element and the third word element are the same, determine that the second word element has passed verification.

[0071] In an alternative embodiment, the first output sequence further includes a fourth word element, and the first verification module 304 is further configured to use the text prediction model to predict the fourth word element based on the second word element and the text sequence, and send the fourth word element to the text generation model; and use the text generation model to concatenate the first word element, the second word element and the fourth word element into a first output sequence.

[0072] In one alternative approach, if the second word token verification passes, the text generation model pauses the decoding calculation of the fourth word token; if the verification fails, the decoding calculation of the fourth word token resumes.

[0073] In one alternative embodiment, the inference decoding device 300 based on a large language model further includes a second verification module 306, configured to, in the event that the second lexical verification fails, utilize the text generation model to perform a decoding operation based on the first lexical and the text sequence to generate a fifth lexical; send the fifth lexical to the text prediction model to update the context information used by the text prediction model to generate the next lexical; use the text generation model to concatenate the first lexical and the fifth lexical into a second output sequence, and feed the second output sequence back to the user.

[0074] In one optional embodiment, there are multiple users, and the text sequences include multiple components. The decoding module 302 is further configured to perform a decoding operation on each text sequence using the text generation model to generate a first word element corresponding to each text sequence, and send multiple text sequences and the first word element corresponding to each text sequence to the text prediction model. The prediction module 303 is further configured to predict a second word element based on each text sequence and the corresponding first word element using the text prediction model, and send multiple second word elements to the text generation model. The first verification module 304 is further configured to verify each second word element using the text generation model, and if the second word element passes verification, concatenate the verified second word element with the corresponding first word element using the text generation model to form the first output sequence. The feedback module 305 is further configured to feed back the first output sequence to the corresponding user.

[0075] This disclosure also provides an electronic device, including: at least one processor; a memory for storing at least one processor-executable instruction; wherein the at least one processor is used to execute the instruction to implement the steps of the method disclosed in this disclosure.

[0076] Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present disclosure. Figure 4 As shown, the electronic device 400 includes at least one processor 401 and a memory 402 coupled to the processor 401, which can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0077] The processor 401 described above can also be called a Central Processing Unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the processor 401 or by software instructions. The processor 401 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 402, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 401 reads information from the memory 402 and, in conjunction with its hardware, completes the steps of the method described above.

[0078] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, for example, Figure 5 The computer system 500 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including functions such as those mentioned above. Figure 5 This is a schematic diagram of the structure of a computer system provided in an embodiment of the present disclosure.

[0079] Computer system 500 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of this disclosure described and / or claimed herein.

[0080] like Figure 5As shown, the computer system 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the computer system 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0081] Multiple components in the computer system 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 can be any type of device capable of inputting information into the computer system 500. The input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. The storage unit 508 may include, but is not limited to, a hard disk and an optical disk. The communication unit 509 allows the computer system 500 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, a modem, network card, infrared communication device, wireless communication transceiver, and / or chipset, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0082] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose temperature (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 502 and / or communication unit 509. In some embodiments, the computing unit 501 can be configured to perform the methods disclosed in this disclosure by any other suitable means (e.g., by means of firmware).

[0083] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.

[0084] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0085] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0086] Figure 6 This is a schematic diagram of a computer program product provided according to an embodiment of the present disclosure. Figure 6 As shown, the computer program product 600 includes a computer program 601, which, when executed by a processor, implements the methods disclosed in the embodiments of this disclosure.

[0087] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0088] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0089] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0090] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0091] The above description is merely an illustration of some embodiments of this disclosure and the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0092] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A reasoning and decoding method based on a large-scale language model, characterized in that, Applied to a server, the server being deployed with a first computing unit and a second computing unit, the first computing unit being used to run a text generation model, and the second computing unit being used to run a text prediction model, the method includes: Obtain the text sequence input by the user; The text generation model is used to perform a decoding operation on the text sequence to generate a first word element, and the first word element and the text sequence are sent to the text prediction model. The text prediction model is used to predict a second word based on the first word and the text sequence, and the second word is sent to the text generation model. The text generation model is used to verify the second word element, and if the second word element passes the verification, the text generation model is used to concatenate the first word element and the second word element in a preset order to form a first output sequence. The first output sequence is fed back to the user.

2. The method according to claim 1, characterized in that, The step of validating the second lexical unit using the text generation model includes: The text generation model is used to perform a decoding operation on the text sequence based on the first word element to generate a third word element; The second word element is compared with the third word element, and if the comparison results determine that the second word element and the third word element are the same, the second word element is determined to have passed the verification.

3. The method according to claim 1, characterized in that, The first output sequence further includes a fourth word element. The step of concatenating the first word element and the second word element into a first output sequence according to a preset order using the text generation model, provided the second word element has passed verification, includes: The text prediction model is used to predict the fourth word based on the second word and the text sequence, and the fourth word is sent to the text generation model. The first word, the second word, and the fourth word are concatenated into a first output sequence using the text generation model.

4. The method according to claim 3, characterized in that, If the second word element verification passes, the text generation model pauses the decoding calculation of the fourth word element; if the verification fails, the decoding calculation of the fourth word element resumes.

5. The method according to claim 1, characterized in that, The method further includes: If the second word token verification fails, the text generation model is used to perform a decoding operation based on the first word token and the text sequence to generate the fifth word token; The fifth word is sent to the text prediction model to update the context information used by the text prediction model to generate the next word. The text generation model is used to concatenate the first word and the fifth word into a second output sequence, and the second output sequence is fed back to the user.

6. The method according to claim 1, characterized in that, The users are multiple, the text sequence includes multiple components, and the method includes: The text generation model is used to perform decoding operations on each text sequence to generate a first word element corresponding to each text sequence, and multiple text sequences and the first word element corresponding to each text sequence are sent to the text prediction model. The text prediction model is used to predict the second word element based on each text sequence and the corresponding first word element, and multiple second word elements are sent to the text generation model. The text generation model is used to verify each second word element, and if the second word element passes the verification, the text generation model is used to concatenate the verified second word element with the corresponding first word element to form the first output sequence. The first output sequence is fed back to the corresponding user.

7. The method according to claim 1, characterized in that, The first output sequence further includes a fourth word element, and the method further includes: The text prediction model is used to simultaneously predict the second and fourth word elements based on the first word element and the text sequence, and the second and fourth word elements are sent to the text generation model. The text generation model is used to verify the second word and the fourth word in sequence. If both the second word and the fourth word pass the verification, the text generation model is used to concatenate the first word, the second word, and the fourth word in a preset order to form the first output sequence.

8. A reasoning and decoding device based on a large-scale language model, characterized in that, Applied to a server, the server is equipped with a first computing unit and a second computing unit, the first computing unit being used to run a text generation model, and the second computing unit being used to run a text prediction model, including: The acquisition module is used to acquire the text sequence input by the user; The decoding module is used to perform a decoding operation on the text sequence using the text generation model to generate a first word element, and send the first word element and the text sequence to the text prediction model; The prediction module is used to predict a second word based on the first word and the text sequence using the text prediction model, and to send the second word to the text generation model; The verification module is used to verify the second word element using the text generation model, and if the second word element passes the verification, to use the text generation model to concatenate the first word element and the second word element in a preset order to form a first output sequence. A sending module is used to send the first output sequence back to the user.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images