Multi-lexical-element parallel prediction method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-lexical-element parallel prediction method and device, electronic equipment and a storage medium, and the method comprises the steps: inputting a current context into a large language model for multi-lexical-element parallel prediction, and generating a candidate sequence containing a plurality of candidate lexical elements and the prediction probability of each candidate lexical element; screening the candidate sequence according to the prediction probability of each candidate lexical element so as to identify a to-be-optimized lexical element and generate a to-be-optimized sequence; inputting the sequence to be optimized into a pre-trained bidirectional attention model to generate a draft sequence; inputting the draft sequence into the large language model for verification, and generating a verification sequence; and comparing lexical elements at corresponding positions in the verification sequence and the draft sequence by using a speculation sampling algorithm to obtain a target sequence. According to the method, the accuracy and semantic coherence of parallel generation of lexical elements can be effectively improved, and the problem that the draft lexical element acceptance rate is low is solved.

Description

Technical Field

[0001] This disclosure relates to the field of large model inference acceleration technology, and in particular to a multi-word parallel prediction method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have been widely applied in various fields such as natural language generation, machine translation, and intelligent question answering, becoming a core supporting technology in the field of natural language processing. In practical applications of LLMs, current mainstream LLMs generally adopt a token-by-token autoregressive generation method during inference, which inherently suffers from high inference latency. To address this inference latency issue, the industry has proposed Multi-Token Parallel Prediction (MTP) technology. Its core design idea is to predict multiple subsequent tokens simultaneously in a single forward propagation, thereby reducing the number of inference steps and accelerating LLM inference.

[0003] However, the aforementioned MTP methods still have significant technical shortcomings. Traditional MTP methods suffer from high error rates due to a lack of sufficient semantic understanding and logical association capabilities, which offsets the speedup benefits brought by parallel prediction. Moreover, existing MTP technologies are mostly limited to the model training stage, focusing on improving the efficiency of training data utilization and model convergence speed, while failing to meet the high requirements for generation quality in the inference stage. Summary of the Invention

[0004] This disclosure provides a multi-lexical parallel prediction method, apparatus, electronic device, and storage medium. It features dual optimization of bidirectional attention model correction and speculative sampling verification, which improves the accuracy and semantic coherence of parallel generated lexical units, solves the problem of low acceptance rate of draft lexical units, retains the core acceleration of MTP single forward propagation multi-lexical parallel prediction, and combines the speculative sampling logic of single parallel verification of the main model to reduce the number of inference steps, thus truly achieving performance acceleration in the inference stage of large language models.

[0005] According to one aspect of this disclosure, a multi-lexical parallel prediction method is provided, comprising: inputting the current context into a large language model for multi-lexical parallel prediction to generate a candidate sequence containing multiple candidate lexical units and a prediction probability of each candidate lexical unit; filtering the candidate sequence according to the prediction probability of each candidate lexical unit to identify lexical units to be optimized and generate a sequence to be optimized; inputting the sequence to be optimized into a pre-trained bidirectional attention model to generate a draft sequence; inputting the draft sequence into the large language model for verification to generate a verification sequence; and comparing the lexical units at corresponding positions in the verification sequence and the draft sequence using a speculative sampling algorithm to obtain a target sequence.

[0006] Optionally, filtering the candidate sequence based on the predicted probability of each candidate word to identify the word to be optimized includes: determining a dynamic threshold based on the predicted probability of at least one selected word from the candidate sequence; and identifying candidate words in the candidate sequence whose predicted probability is less than the dynamic threshold as the word to be optimized.

[0007] Optionally, determining the correction probability threshold based on the predicted probability of the selected word element includes: determining the dynamic threshold based on the predicted probability of at least one selected word element from the candidate sequence includes: obtaining the predicted probability of at least one selected word element as a baseline probability; and determining the dynamic threshold based on the baseline probability and a preset correction coefficient.

[0008] Optionally, determining the dynamic threshold based on the predicted probability of at least one selected word from the candidate sequence includes: obtaining the predicted probability of at least one selected word as a baseline probability; determining an adjustment coefficient based on the semantic complexity of the current context; and determining the dynamic threshold based on the baseline probability, the adjustment coefficient, and a preset correction coefficient; wherein the semantic complexity is calculated based on at least one of the lexical diversity, syntactic structural complexity, or semantic coherence of the current context.

[0009] Optionally, generating the sequence to be optimized includes replacing the identified tokens to be optimized with mask characters to form the sequence to be optimized.

[0010] Optionally, the sequence to be optimized is input into a pre-trained bidirectional attention model to generate a draft sequence, including: predicting and filling the mask symbols in the sequence to be optimized using the bidirectional attention model to obtain multiple corrected lexical units; and fusing the multiple corrected lexical units with the retained lexical units in the candidate sequence that were not identified as lexical units to be optimized to generate the draft sequence.

[0011] Optionally, fusing the plurality of corrected lexical units with the retained lexical units in the candidate sequence that were not identified as lexical units to be optimized to generate the draft sequence includes: obtaining a first prediction score corresponding to the corrected lexical unit and a second prediction score of the lexical unit at the corresponding position in the candidate sequence; and determining the lexical unit at the corresponding position in the draft sequence by performing a weighted summation based on the first prediction score and the second prediction score.

[0012] Optionally, the weighting coefficients of the weighted summation are dynamically determined based on the overall predicted probability distribution of the candidate sequence or the semantic features of the current context.

[0013] Optionally, determining the lexical unit at the corresponding position in the draft sequence by weighted summation of the first predicted score and the second predicted score includes: obtaining a first score vector corresponding to the first predicted score and a second score vector corresponding to the second predicted score; determining a first fusion parameter corresponding to the corrected lexical unit and a second fusion parameter corresponding to the lexical unit at the corresponding position in the candidate sequence; calculating a fusion score vector based on the first fusion parameter, the first score vector, the second fusion parameter, and the second score vector; and determining the lexical unit at the corresponding position in the draft sequence based on the fusion score vector.

[0014] Optionally, determining the first fusion parameter corresponding to the corrected lexical and the second fusion parameter corresponding to the lexical at the corresponding position in the candidate sequence includes: obtaining the third score vectors corresponding to all retained lexicals in the candidate sequence; determining the target score vector with the largest value from all the second score vectors and the third score vectors; calculating the first fusion parameter based on a preset fusion coefficient and the target score vector; and determining the second fusion parameter according to the first fusion parameter.

[0015] Optionally, the speculative sampling algorithm is used to compare the corresponding words in the verification sequence and the draft sequence to obtain the target sequence, which includes: starting from the starting position, comparing the words in the same position in the verification sequence and the draft sequence in turn; if they are consistent up to the kth word, then the first k words are adopted into the target sequence; if the first inconsistency occurs at the rth word, then the rth word in the verification sequence is adopted into the target sequence, and the comparison in the current round is terminated; where k and r are both positive integers and less than the number of words in the candidate sequence.

[0016] According to one aspect of this disclosure, a multi-lexical parallel prediction device is provided, comprising: a parallel prediction module, configured to input the current context into a large language model for multi-lexical parallel prediction to generate a candidate sequence; the candidate sequence includes multiple candidate lexicals, each corresponding to a multiple prediction probability; a determination module, configured to determine the lexical to be optimized from the multiple candidate lexicals and generate a sequence to be optimized; a first generation module, configured to input the sequence to be optimized into a pre-trained bidirectional attention model to generate a draft sequence; a second generation module, configured to input the draft sequence into the large language model to generate a verification sequence; and a verification module, configured to verify the draft sequence based on the verification sequence using a speculative sampling algorithm to obtain a target sequence.

[0017] According to one aspect of this disclosure, an electronic device is proposed, the electronic device comprising a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory, wherein the program is executed by the processor to implement the multi-word parallel prediction method as described above.

[0018] According to one aspect of this disclosure, a computer-readable storage medium is proposed that stores one or more programs, which can be executed by one or more processors to implement the multi-word parallel prediction method as described above.

[0019] This disclosure proposes a multi-lexical parallel prediction method, apparatus, electronic device, and storage medium. The method includes: inputting the current context into a large language model for multi-lexical parallel prediction, generating a candidate sequence containing multiple candidate lexical units and the prediction probability of each candidate lexical unit; filtering the candidate sequence according to the prediction probability of each candidate lexical unit to identify lexical units to be optimized and generating a sequence to be optimized; inputting the sequence to be optimized into a pre-trained bidirectional attention model to generate a draft sequence; inputting the draft sequence into the large language model for verification to generate a verification sequence; and comparing the lexical units at corresponding positions in the verification sequence and the draft sequence using a speculative sampling algorithm to obtain the target sequence. This disclosure improves the accuracy and semantic coherence of parallel generated lexical units through dual optimization of bidirectional attention model correction and speculative sampling verification, solves the problem of low acceptance rate of draft lexical units, retains the core acceleration of multi-lexical parallel prediction in MTP single forward propagation, and reduces the number of inference steps by combining the speculative sampling logic of single parallel verification in the main model, truly achieving performance acceleration in the inference stage of the large language model.

[0020] Furthermore, a dynamic threshold is determined based on the predicted probability of the selected word element. The word elements to be optimized are screened based on the quantitative comparison between the predicted probability and the dynamic threshold, so as to achieve accurate identification of the word elements to be optimized. Only low-confidence word elements are subsequently corrected, reducing invalid calculations and improving correction efficiency. The word elements to be optimized are distinguished from the retained word elements, and high-confidence word elements are retained. This ensures the basic semantic coherence of the candidate sequence and makes subsequent correction and optimization more targeted, reducing the risk of over-correction.

[0021] Furthermore, by selecting the prediction probability of at least one selected word in the candidate sequence as the baseline probability, the determination of the dynamic threshold has a clear and objective quantitative basis, avoiding the problem that the traditional fixed threshold is out of touch with the current prediction scenario and has too large a screening bias.

[0022] Furthermore, by combining the semantic complexity quantification of the current context with the adjustment coefficient, the correction probability threshold can be adapted to different semantic scenarios. Specifically, the threshold can be lowered in complex scenarios to more leniently select words to be optimized, while the threshold can be raised in simple scenarios to reduce unnecessary corrections, thereby improving the dynamic adaptability of the threshold. Semantic complexity is quantified from multiple dimensions, including lexical density, syntactic level, and semantic reasoning depth, making the complexity judgment more objective and accurate, avoiding the one-sidedness of single-dimensional judgment. The threshold is determined by multiplying the baseline probability, the preset correction coefficient, and the scenario-based adjustment coefficient, achieving multi-level fine-grained control of the threshold and further improving the accuracy of word selection to be optimized.

[0023] Furthermore, the words to be optimized are replaced with mask symbols to adapt to the mask filling capability of the bidirectional attention model, providing standardized input for subsequent accurate correction and ensuring the compatibility of the correction process. Only low-confidence words to be optimized are masked, while retaining the original information of the retained words. This allows the sequence to be optimized to clearly define the correction position while retaining high-confidence semantic information, providing a complete contextual reference for the bidirectional attention model.

[0024] Furthermore, the mask-filling capability of the pre-trained bidirectional attention model is used to accurately correct the mask symbols, compensating for the insufficient semantic understanding capability of the MTP module in the large language model and improving the generation quality of low-confidence words. A draft sequence is generated by fusing corrected words with retained words, taking into account both the parallel generation features of the original MTP module and the correction advantages of the bidirectional attention model, avoiding the limitations of a single model. The fusion process preserves the semantic logic of the original sequence, ensuring that the generated draft sequence not only corrects errors but also maintains consistency with the core semantics of the original candidate sequence, thus improving the matching degree between the draft sequence and the main model validation.

[0025] Furthermore, the word units in the draft sequence are determined by weighted fusion based on the predicted scores. The confidence of the word units is measured by quantified score values, making the fusion process more objective and quantifiable, and avoiding the randomness of subjective substitution.

[0026] Furthermore, by dynamically determining the weight coefficients through the overall prediction probability distribution or the semantic features of the current context, the weighted summation process can adaptively match the current prediction scenario, thereby improving the accuracy and rationality of word fusion.

[0027] Furthermore, fusion parameters are configured for both the corrected and unoptimized lexical units, enabling flexible adjustment of their weights. This allows for adjustment of the contributions of the corrected model and the original MTP model based on the actual scenario, enhancing the flexibility of the fusion process. Fusion calculations are performed word-by-word to determine the lexical units in the draft sequence, achieving refined word-level fusion. This ensures that the lexical unit at each position is the optimal choice for the current confidence level, further improving the overall quality of the draft sequence.

[0028] Furthermore, the fusion parameters are determined based on the maximum score vector among the retained and candidate words, allowing the fusion parameters to adapt to the overall confidence level of the current candidate sequence, thus achieving dynamic generation of fusion parameters. The first fusion parameter is determined based on the product of the preset fusion coefficient and the target score vector, and then the second fusion parameter is derived, ensuring the correlation and rationality of the two fusion parameters. This makes the weight allocation more closely match the actual confidence level of the words. The determination of the fusion parameters relies on the scoring characteristics of the candidate sequence itself, requiring no additional manual intervention, thus improving the automation level and scenario adaptability of the fusion process.

[0029] Furthermore, by sequentially matching the tokens of the verification sequence and the draft sequence position by position, the first mismatch position is accurately located, thereby achieving refined verification of the draft sequence and ensuring the generation quality of the target sequence. The matched tokens are directly adopted as target tokens, and the tokens verified by the main model are adopted at the first mismatch position and the verification stops. This retains the correct tokens with high confidence in the draft sequence and uses the main model to correct the first error, thus balancing generation quality and inference efficiency.

[0030] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0031] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0032] Figure 1 This is a system architecture diagram of the multi-word parallel prediction method applied in the embodiments of this disclosure;

[0033] Figure 2 This is a main flowchart of a multi-word parallel prediction method according to an embodiment of the present disclosure;

[0034] Figure 3 This is a schematic diagram of a multi-word parallel prediction method according to an embodiment of the present disclosure;

[0035] Figure 4 This is a schematic diagram of step S220 of an embodiment of the present disclosure;

[0036] Figure 5 This is a schematic diagram of step S230 of an embodiment of the present disclosure;

[0037] Figure 6 This is a schematic diagram of step S502 of an embodiment of the present disclosure;

[0038] Figure 7 yes Figure 2 Flowchart of step S250;

[0039] Figure 8 This is a schematic diagram of the structure of a multi-word parallel prediction device proposed in one embodiment of the present disclosure;

[0040] Figure 9 This is a schematic diagram of the structure of an electronic device proposed in one embodiment of the present disclosure. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0042] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0043] A token is the smallest semantic unit used in large-scale language processing models. It represents the basic input unit formed after the original input text has been segmented. A token can be a character, a word, or a phrase, and it is the basic unit for constructing sequences, generating vector representations, and performing attention calculations.

[0044] Large Language Model (MLM): Also known as a large-scale language model, it is a deep learning model pre-trained on massive text corpora. It is a core model architecture in the field of natural language processing, representing an artificial intelligence model that uses the Transformer as its backbone and achieves understanding and generation of natural language with hundreds of billions of parameters. A single large language model can support multiple tasks such as text generation, semantic understanding, machine translation, and question-answering interaction. It is a foundational model for realizing human-computer interaction in natural language, multi-lexical parallel prediction, and complex semantic reasoning.

[0045] Bidirectional Attention Model: This is an attention mechanism model that integrates bidirectional contextual semantic capture capabilities. It is the core model architecture for natural language understanding tasks. It represents a deep learning model that uses a bidirectional self-attention mechanism as its core and achieves accurate semantic encoding by simultaneously learning the semantic information of the preceding and following context of the input sequence.

[0046] Query Vector: A query vector is a vector representation generated from the current lexical unit, used in self-attention mechanisms to proactively request attention from other lexical units. This vector is obtained through the model's embedding layer and linear transformations, representing the semantic direction of the current lexical unit.

[0047] Prediction Probability: Prediction probability is the confidence score assigned by the large model to each candidate word in the candidate sequence. It is obtained by normalizing the original score output by the model and ranges from 0 to 1. It is used to characterize the confidence of the model in generating the corresponding word at that position and is the core quantitative basis for screening words to be optimized.

[0048] Context: The context is the reference text sequence when the large language model generates lexical units. It contains the user's original input and the effective lexical units that the model has generated. It is the semantic basis for the model to perform lexical unit prediction. Multi-lexical unit parallel prediction is performed based on the latest context.

[0049] Score Vector: The score vector is the original numerical vector output for each word in the sequence during word prediction in large language models, bidirectional attention models, etc. It is also often called logits / original prediction score. It is the original output result of the model after the last linear layer mapping without Softmax normalization. It is the core quantitative basis for characterizing the strength and confidence of the model's preference for word prediction, and it is also the key numerical basis in the process of word selection, fusion, and optimization.

[0050] In related technologies, traditional MTP methods suffer from high error rates due to a lack of sufficient semantic understanding and logical association capabilities, which offsets the speedup benefits brought by parallel prediction. Moreover, existing MTP technologies are mostly limited to the model training stage, focusing on improving the utilization efficiency of training data and the model convergence speed, while failing to meet the high requirements for generation quality in the inference stage.

[0051] Based on this, this disclosure proposes a multi-lexical parallel prediction method, device, electronic device, and storage medium. Through dual optimization of bidirectional attention model correction and speculative sampling verification, it improves the accuracy and semantic coherence of parallel generated lexical units, solves the problem of low acceptance rate of draft lexical units, retains the core acceleration of MTP single forward propagation multi-lexical parallel prediction, and combines the speculative sampling logic of single parallel verification of the main model to reduce the number of inference steps, truly achieving performance acceleration in the inference stage of large language models.

[0052] System architecture description applied in the embodiments of this disclosure

[0053] Figure 1This is a system architecture diagram of the multi-terminal parallel prediction method used in this embodiment of the disclosure. It includes: object terminal 110, Internet 120, gateway 130, and server 140.

[0054] Object terminal 110 is a device used by an object to input query information. It includes various forms such as desktop computers, laptops, PDAs (personal digital assistants), mobile phones, in-vehicle terminals, home theater terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple devices can be connected via a local area network, sharing a single display device to work collaboratively, forming a single terminal. Object terminal 110 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0055] Gateway 130, also known as an internetwork connector or protocol converter, is a computer system or device that enables network interconnection at the transport layer and acts as a translator. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateway 130 also provides filtering and security functions. Messages sent from target terminal 110 to server 140 are forwarded to the corresponding server 140 via gateway 130. Messages sent from server 140 to target terminal 110 are also forwarded to the corresponding target terminal 110 via gateway 130.

[0056] Server 140 refers to a computer system that provides services to target terminal 110. Compared to target terminal 110, server 140 has higher requirements in terms of stability, security, and performance. Server 140 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 140 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0057] Overall Implementation of the Multi-Word Parallel Prediction Method of the Embodiments of this Disclosure

[0058] This disclosure proposes a multi-word parallel prediction method, applied to a multi-word parallel prediction device, with reference to... Figure 2 and Figure 3 The multi-word parallel prediction method includes:

[0059] Step S210: Input the current context into the large language model for multi-lexical parallel prediction, and generate a candidate sequence containing multiple candidate lexical units and the predicted probability of each candidate lexical unit.

[0060] Step S220: The candidate sequences are filtered according to the predicted probability of each candidate word to identify the word to be optimized and generate the sequence to be optimized.

[0061] Step S230: Input the sequence to be optimized into a pre-trained bidirectional attention model to generate a draft sequence;

[0062] Step S240: Input the draft sequence into the large language model for verification to generate a verification sequence;

[0063] Step S250: The tokens at corresponding positions in the verification sequence and the draft sequence are compared using a speculative sampling algorithm to obtain the target sequence.

[0064] In the lexical prediction and natural language generation process of the large language model, the query information is the original request text initiated by the user to the large language model. It is the only initial input for the model to start inference and the original input content that triggers the model's inference. The current context is the complete semantic sequence formed by the model continuously concatenating the effective target lexical units generated by the model based on the query during the inference process. It is the direct input basis for the model to perform each multi-lexical parallel prediction and will be dynamically updated with the inference process. The query is the core component of the current context. The current context is the dynamic iterative result of the query combined with the effective lexical units generated by the model. Lexical prediction is performed based on the real-time updated current context.

[0065] If this round of prediction is the first round of prediction, then the current context of the input large language model is the query input by the user. If it is a prediction after the first round of prediction, then before step S210, there is also a step of predicting the current context based on the query input by the user.

[0066] In step S210, the multi-term parallel prediction (MTP) module of the large language model (LLM) is used to perform a forward propagation based on the generated current context, and a candidate sequence [t1, t2, ..., tN] containing N terms is generated synchronously, and the prediction probability [p1, p2, ..., pN] corresponding to each term is recorded.

[0067] Where N is the preset number of parallel prediction tokens, which can be set according to the model inference performance and hardware resource adaptation, and is a positive integer.

[0068] like Figure 3 As shown in the embodiment of this application, the parallel prediction of 4 words is used as an example for explanation. Therefore, the words 1, 2, 3 and 4 in the candidate sequence are [t1, t2, t3, t4] and the prediction probabilities corresponding to [t1, t2, t3, t4] are [p1, p2, p3, p4].

[0069] In step S220, the word units that need to be corrected are selected from multiple candidate word units, and an optimization sequence is generated. In this application, the word units that need to be corrected are low-confidence word units that have the risk of generation errors or semantic incoherence, while high-confidence word units are retained. Finally, an optimization sequence is generated for subsequent correction processing, providing a clear target position and basic sequence for the accurate correction of the bidirectional attention model.

[0070] Specifically, see Figure 4 In one embodiment, step S220, which involves filtering the candidate sequences based on the predicted probabilities of each candidate lexical to identify the lexical to be optimized, includes the following steps.

[0071] In step S401, a dynamic threshold is determined based on the predicted probability of at least one selected word from the candidate sequence.

[0072] In this embodiment, the first word in the candidate sequence is selected as the chosen word. That is, the prediction probability p1 of the first word is used as the benchmark to filter low-confidence words and high-confidence words. The first word in the candidate sequence is the first subsequent word directly predicted by the large language model based on the current context. It has the highest semantic relevance and fit with the current context, and its prediction result is most tightly constrained by the semantics of the context. Compared with words in subsequent positions in the candidate sequence, it is less likely to have semantic disconnect or prediction errors.

[0073] For example, if the current context is "The weather is nice today, we can go", the candidate sequences generated in parallel by the MTP module are [park, walk, take a picture, check in], with corresponding prediction probabilities of 0.92, 0.72, 0.51, and 0.41, respectively. Among them, the closest word in the example context, "park", is the selected word, with p1 of 0.92.

[0074] In one possible embodiment, determining the dynamic threshold based on the predicted probability of at least one selected word from the candidate sequence includes: obtaining the predicted probability of at least one selected word as a baseline probability; and determining the dynamic threshold based on the baseline probability and a preset correction coefficient.

[0075] The correction probability threshold can be calculated using the formula: Threshold = λ × p1, where Threshold represents the correction probability threshold, λ is a preset empirical coefficient, for example, it can be 0.6, and its value range can be flexibly adjusted according to actual operation and model characteristics, usually set to 0.4-0.8, and p1 represents the baseline probability corresponding to the selected word determined in step S401. This correction probability threshold calculation method is suitable for application scenarios with high threshold calculation efficiency requirements and relatively simple semantic scenarios, realizing the rapid screening of words to be optimized.

[0076] For example, if the baseline probability of the selected word "park" is 0.92 and the preset correction coefficient is 0.6, then the correction probability threshold = 0.92 × 0.6 = 0.552.

[0077] In another possible embodiment, the correction probability threshold needs to be adjusted in conjunction with the semantic complexity of the current context. The step of determining the dynamic threshold based on the predicted probability of at least one selected word from the candidate sequence includes: obtaining the predicted probability of at least one selected word as a baseline probability; determining an adjustment coefficient based on the semantic complexity of the current context; and determining the dynamic threshold based on the baseline probability, the adjustment coefficient, and a preset correction coefficient; wherein the semantic complexity is calculated based on at least one of the lexical diversity, syntactic structural complexity, or semantic coherence of the current context.

[0078] First, the predicted probability corresponding to the selected word is determined as the baseline probability. That is, the predicted probability of the first word in the candidate sequence is determined as the baseline probability.

[0079] Then, semantic complexity is quantified based on the current context to obtain a complexity coefficient, wherein the semantic complexity includes at least one of lexical density, syntactic level, and semantic reasoning depth.

[0080] For the current context of the input large language model in step S210, a quantitative analysis of semantic complexity is performed, and a quantitative value, namely the complexity coefficient, is finally output. The complexity coefficient is used to characterize the semantic difficulty of the current context. The larger the coefficient, the more complex the semantic context is, and the higher the difficulty of the model in predicting subsequent words. The selection criteria need to be relaxed, specifically by lowering the correction probability threshold. The smaller the coefficient, the simpler the semantic context is, and the lower the difficulty of the model in prediction. The selection criteria can be improved, specifically by raising the correction probability threshold.

[0081] The quantitative analysis of semantic complexity needs to be combined with multi-dimensional evaluation, specifically including at least one of lexical density, syntactic hierarchy, and semantic reasoning depth. The quantification methods for each dimension are as follows (they can be used individually or in combination):

[0082] Lexical density is the ratio of the number of effective words in the current context to the total length of the context (total number of lexical units). The larger the ratio, the higher the lexical density and the more complex the semantics.

[0083] Syntactic hierarchy is determined by parsing the syntactic structure of the current context using a syntactic analysis model, such as a dependency parsing model, and counting the number of syntactic levels. For example, a simple sentence has 1 level, a complex sentence has 2 or more levels, and the more levels there are, the more complex the syntax.

[0084] Semantic reasoning depth assesses whether the current context requires cross-sentence reasoning or implicit semantic interpretation. If no reasoning is required, the semantic reasoning depth is 1; if simple reasoning is required, it is 2; and if complex cross-sentence reasoning is required, it is 3 or higher. The higher the depth value, the more complex the semantics.

[0085] In this embodiment, the complexity coefficient is compared with a preset complexity threshold to determine whether the current context is complex. The preset complexity threshold is a set value of 0.5.

[0086] In one possible embodiment, if the complexity coefficient is greater than a preset complexity threshold, a first adjustment coefficient is determined as the adjustment coefficient; the first adjustment coefficient is less than one.

[0087] If the complexity coefficient is greater than the preset complexity threshold, it indicates that the current context semantics are complex and the model prediction is difficult. In this case, the first adjustment coefficient is determined as the adjustment coefficient for this calculation. The first adjustment coefficient is less than 1, for example, 0.8 (the value range can be limited to 0.7-0.95). It is used to reduce the correction probability threshold, relax the screening criteria for the words to be optimized, and avoid missing low-confidence words.

[0088] In one possible embodiment, if the complexity coefficient is less than or equal to the preset complexity threshold, a second adjustment coefficient is determined as the adjustment coefficient; the first adjustment coefficient is greater than one.

[0089] If the complexity coefficient is less than or equal to the preset complexity threshold, it indicates that the current context semantics are simple and the model prediction difficulty is low. In this case, the second adjustment coefficient is determined as the adjustment coefficient for this calculation. The second adjustment coefficient is greater than 1, for example, 1.1 (the value range can be limited to 1.05-1.2), which is used to increase the correction probability threshold, strictly screen the words to be optimized, and avoid invalid calculations caused by over-correction.

[0090] It should be noted that the specific values ​​of the first and second adjustment coefficients can be flexibly adjusted according to the number of parameters of the large language model and the number of lexical units predicted in parallel. This is mainly to achieve a more lenient screening for semantically complex scenarios and a stricter screening for semantically simple scenarios, thus ensuring a balance between screening accuracy and efficiency.

[0091] Finally, the correction probability threshold is determined based on the product of the baseline probability, the preset correction coefficient, and the adjustment coefficient.

[0092] For example, if the baseline probability of the selected word "park" is 0.92, the preset correction coefficient is 0.6, the current context complexity coefficient is 0.6, which is greater than the preset complexity threshold of 0.5, and the first adjustment coefficient of 0.8 is selected, then the correction probability threshold = 0.92 × 0.6 × 0.8 = 0.4416.

[0093] In step S402, candidate words in the candidate sequence whose predicted probability is less than the dynamic threshold are identified as the words to be optimized.

[0094] Specifically, after determining the correction probability threshold, the following steps are performed on each of the plurality of candidate lexical units:

[0095] In step S4021, the candidate word currently being executed is determined as the current candidate word.

[0096] In this embodiment, each candidate word is selected as the current candidate word in the order of the word elements in the candidate sequence. The subsequent probability judgment and type judgment operations are performed on the current candidate word. After the judgment of the current candidate word is completed, the next candidate word is selected as the new current candidate word. The above process is repeated until all candidate words have been judged.

[0097] In step S4022, it is determined whether the predicted probability of the current candidate word is less than the correction probability threshold. If yes, proceed to step S4023; otherwise, proceed to step S4024.

[0098] In step S4023, the current candidate lexical is determined to be a lexical to be optimized.

[0099] In one possible embodiment, if the predicted probability of the current candidate word is less than the correction probability threshold, the current candidate word is determined to be a word to be optimized, indicating that the prediction confidence of the current candidate word is insufficient and belongs to a low-confidence word, which needs to be corrected later.

[0100] In step S4024, the current candidate word is determined to be a reserved word.

[0101] In one possible embodiment, if the predicted probability of the current candidate word is greater than or equal to the correction probability threshold, the current candidate word is determined to be a retained word, indicating that the prediction confidence of the current candidate word meets the standard and belongs to a high-confidence word, so no correction is required.

[0102] In the first embodiment of step S401, the correction probability threshold is 0.552. In [park, walk, take a photo, check in], the predicted probability of "take a photo" is 0.51, which is less than the correction probability threshold of 0.552, and the predicted probability of "check in" is 0.41, which is less than the correction probability threshold of 0.552. They are low-confidence words, which are words to be optimized. The remaining two words, "park" and "walk", are retained words.

[0103] In the second embodiment of step S401, the correction probability threshold is 0.4416. Among [park, take a walk, take pictures, check in], the predicted probability of "check in" is 0.41, which is less than the correction probability threshold of 0.4416. The remaining three tokens, "park", "take a walk", and "take pictures", are all retained tokens.

[0104] In the following text of this application, the second embodiment of step S401 is taken as an example, where "check in" is the token to be optimized, and the remaining three tokens, "park", "take a walk", and "take pictures", are all retained tokens. The subsequent processing steps of the first embodiment are the same and will not be elaborated here.

[0105] In one embodiment, generating the sequence to be optimized in step S220 includes: replacing the identified token to be optimized with a mask token to form the sequence to be optimized.

[0106] In this embodiment, the token to be optimized is replaced with a mask token, and based on the replaced mask token and the retained tokens, the sequence to be optimized is formed.

[0107] Specifically, the low-confidence token "check in" is replaced with [MASK] to form the sequence to be optimized [park, take a walk, take pictures, [MASK]].

[0108] In step S230, the sequence to be optimized is input into a pre-trained bidirectional attention model to generate a draft sequence. Specifically, the sequence to be optimized is input into a pre-trained bidirectional attention model (such as BERT). This model uses its powerful context semantic understanding ability to predict these [MASK] positions and finally generates a corrected draft sequence.

[0109] Specifically, refer to Figure 5 , in the first possible embodiment of step S230, step S230 includes the following steps.

[0110] In step S501, the mask tokens in the sequence to be optimized are predicted and filled by the bidirectional attention model to obtain multiple corrected tokens.

[0111] In this embodiment, the bidirectional attention model (such as BERT) simulates this scenario of "knowing some tokens and predicting the tokens at the masked positions" during the pre-training stage. During the training process, some tokens in the input sequence are randomly replaced with [MASK] tokens. The model learns the context semantics to predict the real tokens corresponding to the [MASK] positions. After pre-training with a large-scale corpus, it already has a powerful ability to predict [MASK] positions and can directly adapt to the [MASK] position prediction requirements of the sequence to be optimized in this application without additional large-scale fine-tuning, and has extremely strong adaptability.

[0112] For example, in this application, the mask symbol is located at the end of the sequence. The sequence to be optimized is [park, walk, take a picture, [MASK]]. The bidirectional attention model can predict the word that best fits the position of [MASK] based on the three words predicted above. The final predicted corrected word is "enjoy the scenery".

[0113] In other possible embodiments, the mask symbol is located in the middle of the sequence. For example, if the sequence to be optimized is [A,[MASK],B], the model can combine the semantics of the preceding "A" and the following "B" to predict the word that best fits the position of [MASK] and correct the sequence to [A,C,B]. This avoids prediction bias caused by relying solely on one-way semantics and meets the core requirement of this application to correct low-confidence words while ensuring the semantic coherence of the sequence.

[0114] In this embodiment, the word "viewing the scenery" in the corrected sequence [park, stroll, take photos, enjoy the scenery] is the corrected word element, which is the word element after masking and filling [MASK].

[0115] In step S502, the plurality of corrected lexical units are fused with the retained lexical units in the candidate sequence that were not identified as lexical units to be optimized to generate the draft sequence.

[0116] In step S502, multiple corrected lexical units are fused with multiple lexical units to be optimized to generate a draft sequence. This is to achieve the optimal combination of corrected lexical units and lexical units to be optimized, ensuring that each fused draft lexical unit has both the semantic accuracy of the bidirectional attention model and the contextual association of the multi-lexical parallel prediction of the large language model, thereby further improving the quality of the draft sequence.

[0117] In a second possible embodiment of step S230, refer to Figure 3 As shown, a portion of the current context and the sequence to be optimized can be input into the bidirectional attention model for prediction and filling, resulting in multiple corrected words. By inputting a portion of the current context and the sequence to be optimized together, the bidirectional attention model can simultaneously capture the dual semantics of the sequence to be optimized and the current context, making the corrected words in the prediction more in line with the real semantic logic of the current prediction scenario, and significantly reducing the probability of the corrected words being out of sync with the context.

[0118] Specifically, see Figure 6 In one embodiment, step S502 includes the following steps.

[0119] In step S601, the first prediction score corresponding to the corrected word and the second prediction score of the word at the corresponding position in the candidate sequence are obtained.

[0120] In this embodiment, the first prediction score represents the confidence level of the corrected lexical output by the bidirectional attention model, and the second prediction score represents the confidence level of the candidate lexical output by the large language model.

[0121] In step S602, the word at the corresponding position in the draft sequence is determined by weighted summation of the first prediction score and the second prediction score.

[0122] By weighted summing of the confidence scores of the two lexical units at the same position, the most reasonable lexical unit with the highest confidence score at that position can be determined, and a more semantically accurate and reliable draft sequence can be constructed based on this.

[0123] In one possible embodiment, the weighting coefficients of the weighted summation are dynamically determined based on the overall predicted probability distribution of the candidate sequence or the semantic features of the current context.

[0124] In this embodiment, the weighting coefficients of the weighted summation are not fixed, but are calculated based on the overall prediction probability distribution of the candidate sequences in this prediction or the semantic features of the current context. The overall prediction probability distribution of the candidate sequences is used to illustrate the reliability of the large language model's predictions, while the semantic features of the current context are used to illustrate the prediction difficulty of the current context.

[0125] When the prediction reliability of a large language model is high, the weight coefficient of the second prediction score of the candidate sequence can be increased to improve the confidence of the candidate words output by the large language model and retain more candidate words.

[0126] When the prediction difficulty of the current context is high, the weight coefficient of the first prediction score of the corrected word can be increased to improve the confidence of the candidate words output by the bidirectional attention model and retain more corrected words.

[0127] By dynamically adjusting the weight coefficients based on the overall prediction probability distribution of the candidate sequence or the semantic features of the current context, the weight coefficients can adapt to different prediction scenarios and flexibly adjust the fusion ratio of corrected words and candidate words, thereby improving the accuracy and applicability of draft sequence generation.

[0128] In another possible embodiment, step S602 includes the following steps.

[0129] In step S6021, the first score vector corresponding to the first predicted score and the second score vector corresponding to the second predicted score are obtained.

[0130] In this embodiment, the first score vectors corresponding to multiple corrected lexical units are obtained, which are the original score vectors (logits) output by the bidirectional attention model when masking each [MASK] position in step S501. Each corrected lexical unit corresponds to a unique first score vector. The second score vectors corresponding to multiple candidate lexical units are obtained, which are the original score vectors (logits) output by the large language model for each candidate lexical unit.

[0131] It should be noted that the dimensions of the first score vector and the second score vector may differ due to the size of the model vocabulary. In this embodiment, they can be converted into vectors of the same dimension through linear mapping to ensure the feasibility of subsequent weighted calculations.

[0132] In step S6022, a first fusion parameter corresponding to the corrected lexical and a second fusion parameter corresponding to the lexical at the corresponding position in the candidate sequence are determined.

[0133] In this embodiment, the first fusion parameter is the weight coefficient of the corrected word element, and the second fusion parameter is the weight coefficient of the candidate word element.

[0134] In this embodiment, determining the first fusion parameter corresponding to the corrected lexical unit and the second fusion parameter corresponding to the lexical unit at the corresponding position in the candidate sequence includes:

[0135] First, obtain the third score vector corresponding to each retained word in the candidate sequence, which is the original score vector (logits) of each retained word output by the large language model. From all the second score vectors and the third score vectors, determine the score vector with the largest value among the retained words and the corrected words as the target score vector max(ST).

[0136] Based on a preset fusion coefficient and the target score vector, the first fusion parameter is calculated. In this embodiment, the preset fusion coefficient is 0.5, so the first fusion parameter μ = 0.5 * max(ST). The sum of the first fusion parameter and the second fusion parameter is 1. Therefore, the second fusion parameter 1-μ can be determined based on the first fusion parameter.

[0137] In another possible embodiment, the first fusion parameter can be directly taken as a given value, without the need to calculate it through the target score vector and the preset fusion coefficient. This embodiment is suitable for application scenarios with extremely high requirements for fusion efficiency and relatively fixed semantic scenarios, which simplifies the process of determining the fusion parameter and reduces computational overhead.

[0138] In step S6023, a fusion score vector is calculated based on the first fusion parameter, the first score vector, the second fusion parameter, and the second score vector.

[0139] In this embodiment, S = μS T +(1-μ)S T’ S T Let S be the first score vector, μ be the first fusion parameter, and S be the second score vector. T’ Let be the second score vector, 1-μ be the second fusion parameter, and S be the fusion score vector.

[0140] In step S6024, the lexical units at the corresponding positions in the draft sequence are determined based on the fusion score vector.

[0141] In this implementation, firstly, for a given position, the fusion score vector calculated for that position and the second score vector of the candidate words at that position are obtained. Secondly, Softmax normalization is performed on the fusion score vector and the second score vector respectively, converting them into probability vectors between 0 and 1, resulting in a fusion probability vector and a second probability vector. Then, the two probability vectors are compared, and the word with the higher probability value is selected as the draft word. The draft word will serve as the final word for the current position and will be used to generate subsequent draft sequences.

[0142] For example, the candidate sequence in this application is [park, stroll, take photos, check-in], where the retained lexical units are "park", "stroll", and "take photos", the corrected lexical unit is "enjoy the scenery", and the candidate lexical unit at the corresponding position of the corrected lexical unit is "check-in". Now, it is necessary to merge "enjoy the scenery" and "check-in" to determine the most suitable lexical unit at that position. Assuming that after fusion and normalization calculation, if the fusion probability vector value corresponding to "enjoy the scenery" is 0.78 and the second probability vector value corresponding to "check-in" is 0.7, then "enjoy the scenery" with the higher probability value is determined as the final draft lexical unit.

[0143] Finally, based on the draft lexical units and the retained lexical units, the draft sequence is generated. That is, the draft sequence is [park, walk, take a picture, enjoy the scenery].

[0144] In step S240, the draft sequence is input into the large language model for verification to generate a verification sequence.

[0145] In this embodiment, the draft sequence [park, walk, take a picture, enjoy the scenery] is re-input into the large language model for a single-step, parallel verification to generate a verification sequence. The sequence after correction by the bidirectional language model is verified again by the large language model. The main model will output the "true" prediction result for each position in the draft sequence as the verification sequence.

[0146] In step S250, a speculative sampling algorithm is used to compare the corresponding word units in the verification sequence and the draft sequence to obtain the target sequence. Specifically, the speculative sampling algorithm is used to select the optimal word unit combination through word-by-word matching verification between the verification sequence and the draft sequence, generating the final target sequence. This ensures that the target sequence has both the high efficiency of multi-word unit parallel prediction and the high accuracy and semantic coherence after the main model verification.

[0147] Specifically, see Figure 7 In one embodiment, step S250 includes the following steps.

[0148] In step S701, starting from the starting position, the tokens at the same positions in the verification sequence and the draft sequence are compared sequentially.

[0149] In this embodiment, each word is selected as the current verification word in the order of each word in the verification sequence. Subsequent verification operations are performed on the current verification word. After the verification of the current verification word is completed, the next word is selected as the new current verification word. The above process is repeated until the process ends.

[0150] In step S702, if all consecutive comparisons up to the kth word are consistent, then the first k words are adopted into the target sequence.

[0151] In step S703, if an inconsistency occurs for the first time at the r-th word, the r-th word in the verification sequence is adopted into the target sequence, and the comparison in the current round is terminated.

[0152] Where k and r are both positive integers and are less than the number of lexical units in the candidate sequence.

[0153] In one possible embodiment, if the current verified term matches a term at a corresponding position in the draft sequence, the current term is identified as the target term and adopted into the target sequence. The next term of the current verified term is then matched against the term at a corresponding position in the draft sequence. If the current verified term matches the term at a corresponding position in the draft sequence, it indicates that the term at that position in the draft sequence has been verified by the main model, and its semantic accuracy and credibility meet the standards. Therefore, no replacement is needed, and the next term can be verified.

[0154] In one possible embodiment, if the current verification term does not match the term at the corresponding position in the draft sequence, that is, the first inconsistency occurs at the r-th term, the current verification term, that is, the r-th term, is determined as the target term, and the verification is stopped. This indicates that the term at the corresponding position in the draft sequence has not passed the main model verification and there is a semantic bias or prediction error. The verification term generated by the main model should be used as the standard, and all subsequent verification operations are stopped to avoid invalid verification and improve efficiency.

[0155] In this embodiment, after verification stops, all unverified words after the currently verified word no longer participate in the selection of target words. The target sequence is generated only based on the determined target words, the successfully matched words, and the last unmatched word. This avoids invalid calculations and ensures the accuracy of the target sequence.

[0156] For example, the verified sequence after the draft sequence is verified by the large language model is [park, stroll, enjoy the scenery, cool off], while the draft sequence is [park, stroll, take photos, enjoy the scenery]. "Park" and "stroll" are the same, so they are retained as target words and entered into the target sequence. "Enjoy the scenery" and "take photos" at position 3 are different. At this time, "enjoy the scenery" in the verified sequence is retained as a target word and entered into the target sequence. "Cool off" and "enjoy the scenery" are no longer verified. The final target sequence is [park, stroll, enjoy the scenery].

[0157] This invention creatively introduces pre-trained bidirectional attention models such as BERT as a correction module to intelligently complete and correct low-confidence lexical units in the initial MTP prediction results, similar to a "cloze test." The bidirectional attention mechanism fully utilizes global information from the context surrounding the position to be optimized, thereby generating lexical units that better conform to linguistic logic and semantic coherence, fundamentally solving the bottleneck problem of poor subsequent lexical unit prediction quality in traditional MTP. Furthermore, this invention does not use a fixed threshold for judgment but designs a scientific dynamic threshold calculation mechanism, realizing intelligent and scenario-based quality control strategies, ensuring precise intervention in the correction process, and achieving precise and efficient quality control and dynamic threshold adjustment.

[0158] Furthermore, due to the improved prediction quality of MTP and the further assurance of draft quality by the correction mechanism, the main model's acceptance rate of the draft sequence is improved in the subsequent speculative sampling verification stage, optimizing the speculative decoding process and comprehensively improving the inference speedup ratio.

[0159] Furthermore, a dynamic threshold is determined based on the predicted probability of the selected word element. The word elements to be optimized are screened based on the quantitative comparison between the predicted probability and the dynamic threshold, so as to achieve accurate identification of the word elements to be optimized. Only low-confidence word elements are subsequently corrected, reducing invalid calculations and improving correction efficiency. The word elements to be optimized are distinguished from the retained word elements, and high-confidence word elements are retained. This ensures the basic semantic coherence of the candidate sequence and makes subsequent correction and optimization more targeted, reducing the risk of over-correction.

[0160] Furthermore, by selecting the prediction probability of at least one selected word in the candidate sequence as the baseline probability, the determination of the dynamic threshold has a clear and objective quantitative basis, avoiding the problem that the traditional fixed threshold is out of touch with the current prediction scenario and has too large a screening bias.

[0161] Furthermore, by combining the semantic complexity quantification of the current context with the adjustment coefficient, the correction probability threshold can be adapted to different semantic scenarios. Specifically, the threshold can be lowered in complex scenarios to more leniently select words to be optimized, while the threshold can be raised in simple scenarios to reduce unnecessary corrections, thereby improving the dynamic adaptability of the threshold. Semantic complexity is quantified from multiple dimensions, including lexical density, syntactic level, and semantic reasoning depth, making the complexity judgment more objective and accurate, avoiding the one-sidedness of single-dimensional judgment. The threshold is determined by multiplying the baseline probability, the preset correction coefficient, and the scenario-based adjustment coefficient, achieving multi-level fine-grained control of the threshold and further improving the accuracy of word selection to be optimized.

[0162] Furthermore, the words to be optimized are replaced with mask symbols to adapt to the mask filling capability of the bidirectional attention model, providing standardized input for subsequent accurate correction and ensuring the compatibility of the correction process. Only low-confidence words to be optimized are masked, while retaining the original information of the retained words. This allows the sequence to be optimized to clearly define the correction position while retaining high-confidence semantic information, providing a complete contextual reference for the bidirectional attention model.

[0163] Furthermore, the mask-filling capability of the pre-trained bidirectional attention model is used to accurately correct the mask symbols, compensating for the insufficient semantic understanding capability of the MTP module in the large language model and improving the generation quality of low-confidence words. A draft sequence is generated by fusing corrected words with retained words, taking into account both the parallel generation features of the original MTP module and the correction advantages of the bidirectional attention model, avoiding the limitations of a single model. The fusion process preserves the semantic logic of the original sequence, ensuring that the generated draft sequence not only corrects errors but also maintains consistency with the core semantics of the original candidate sequence, thus improving the matching degree between the draft sequence and the main model validation.

[0164] Furthermore, the word units in the draft sequence are determined by weighted fusion based on the predicted scores. The confidence of the word units is measured by quantified score values, making the fusion process more objective and quantifiable, and avoiding the randomness of subjective substitution.

[0165] Furthermore, by dynamically determining the weight coefficients through the overall prediction probability distribution or the semantic features of the current context, the weighted summation process can adaptively match the current prediction scenario, thereby improving the accuracy and rationality of word fusion.

[0166] Furthermore, fusion parameters are configured for both the corrected and unoptimized lexical units, enabling flexible adjustment of their weights. This allows for adjustment of the contributions of the corrected model and the original MTP model based on the actual scenario, enhancing the flexibility of the fusion process. Fusion calculations are performed word-by-word to determine the lexical units in the draft sequence, achieving refined word-level fusion. This ensures that the lexical unit at each position is the optimal choice for the current confidence level, further improving the overall quality of the draft sequence.

[0167] Furthermore, the fusion parameters are determined based on the maximum score vector among the retained and candidate words, allowing the fusion parameters to adapt to the overall confidence level of the current candidate sequence, thus achieving dynamic generation of fusion parameters. The first fusion parameter is determined based on the product of the preset fusion coefficient and the target score vector, and then the second fusion parameter is derived, ensuring the correlation and rationality of the two fusion parameters. This makes the weight allocation more closely match the actual confidence level of the words. The determination of the fusion parameters relies on the scoring characteristics of the candidate sequence itself, requiring no additional manual intervention, thus improving the automation level and scenario adaptability of the fusion process.

[0168] Furthermore, by sequentially matching the tokens of the verification sequence and the draft sequence position by position, the first mismatch position is accurately located, thereby achieving refined verification of the draft sequence and ensuring the generation quality of the target sequence. The matched tokens are directly adopted as target tokens, and the first mismatch position adopts the main model's verification token and stops verification. This approach retains the correct tokens with high confidence in the draft sequence while using the main model to correct the first error, thus balancing generation quality and inference efficiency.

[0169] Description of apparatus and devices according to embodiments of this disclosure

[0170] See Figure 8 This disclosure also provides a multi-terminal parallel prediction device 800, including a parallel prediction module 810, a sequence generation module 820, a correction module 830, a verification module 840, and an output module 850.

[0171] The parallel prediction module 810 is used to input the current context into the large language model for multi-lexical parallel prediction, and generate a candidate sequence containing multiple candidate lexical units and the prediction probability of each candidate lexical unit.

[0172] The sequence generation module 820 is used to filter the candidate sequences according to the predicted probability of each candidate word to identify the word to be optimized and generate the sequence to be optimized.

[0173] Correction module 830 is used to input the sequence to be optimized into a pre-trained bidirectional attention model to generate a draft sequence;

[0174] Verification module 840 is used to input the draft sequence into the large language model for verification and generate a verification sequence;

[0175] The output module 850 is used to compare the corresponding words in the verification sequence and the draft sequence using a speculative sampling algorithm to obtain the target sequence.

[0176] The multi-word parallel prediction device 800 disclosed herein is used to execute the multi-word parallel prediction method as described in the above embodiments. Its specific processing procedure is the same as that of the multi-word parallel prediction method in the above embodiments, and will not be repeated here.

[0177] This disclosure also provides an electronic device 900, including:

[0178] At least one processor, and,

[0179] A memory that is communicatively connected to at least one processor; wherein,

[0180] The memory stores instructions that are executed by at least one processor to cause the at least one processor to perform the method as described in any of the above embodiments of this application when executing the instructions.

[0181] The following is combined Figure 9 The hardware structure of the electronic device is described in detail. The electronic device includes: a processor 910, a memory 920, an input / output interface 930, a communication interface 940, and a bus 950.

[0182] The processor 910 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0183] The memory 920 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 920 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 920 and is called and executed by the processor 910 using the multi-word parallel prediction method of the embodiments of this disclosure.

[0184] The input / output interface 930 is used to implement information input and output;

[0185] Communication interface 940 is used to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0186] Bus 950 transmits information between various components of the device (e.g., processor 910, memory 920, input / output interface 930, and communication interface 940);

[0187] The processor 910, memory 920, input / output interface 930 and communication interface 940 are connected to each other within the device via bus 950.

[0188] This application also provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the multi-word parallel prediction method of the above embodiments, which will not be described again here.

[0189] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0190] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0191] It should be understood that in the description of the embodiments disclosed herein, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0192] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0193] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0195] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0196] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A multi-word parallel prediction method, characterized in that, include: The current context is input into a large language model for multi-lexical parallel prediction, generating a candidate sequence containing multiple candidate lexical units and the predicted probability of each candidate lexical unit; The candidate sequences are filtered based on the predicted probability of each candidate word to identify the word to be optimized and generate the sequence to be optimized. The sequence to be optimized is input into a pre-trained bidirectional attention model to generate a draft sequence; The draft sequence is input into the large language model for verification, generating a verification sequence; The target sequence is obtained by comparing the corresponding words in the verification sequence and the draft sequence using a speculative sampling algorithm.

2. The multi-terminal parallel prediction method according to claim 1, characterized in that, The candidate sequences are filtered based on the predicted probabilities of each candidate lexical unit to identify the lexical units to be optimized, including: A dynamic threshold is determined based on the predicted probability of at least one selected word from the candidate sequence; Candidate words in the candidate sequence whose predicted probability is less than the dynamic threshold are identified as the words to be optimized.

3. The multi-terminal parallel prediction method according to claim 2, characterized in that, Determining the dynamic threshold based on the predicted probability of at least one selected word from the candidate sequence includes: Obtain the predicted probability of at least one selected word as the baseline probability; The dynamic threshold is determined based on the baseline probability and the preset correction coefficient.

4. The multi-terminal parallel prediction method according to claim 2, characterized in that, Determining the dynamic threshold based on the predicted probability of at least one selected word from the candidate sequence includes: Obtain the predicted probability of at least one selected word as the baseline probability; The adjustment coefficient is determined based on the semantic complexity of the current context; The dynamic threshold is determined based on the baseline probability, the adjustment coefficient, and the preset correction coefficient. The semantic complexity is calculated based on at least one of the lexical diversity, syntactic structural complexity, or semantic coherence of the current context.

5. The multi-terminal parallel prediction method according to claim 2, 3, or 4, characterized in that, The generation of the sequence to be optimized includes: The identified terms to be optimized are replaced with mask characters to form the sequence to be optimized.

6. The multi-terminal parallel prediction method according to claim 5, characterized in that, The sequence to be optimized is input into a pre-trained bidirectional attention model to generate a draft sequence, including: The bidirectional attention model is used to predict and fill in the mask characters in the sequence to be optimized, resulting in multiple corrected lexical units. The multiple corrected lexical units are fused with the retained lexical units in the candidate sequence that were not identified as lexical units to be optimized to generate the draft sequence.

7. The multi-terminal parallel prediction method according to claim 6, characterized in that, The process of fusing the plurality of corrected lexical units with the retained lexical units in the candidate sequence that were not identified as lexical units to be optimized, to generate the draft sequence, includes: Obtain the first prediction score corresponding to the corrected word, and the second prediction score of the word at the corresponding position in the candidate sequence; The corresponding word in the draft sequence is determined by weighted summation of the first prediction score and the second prediction score.

8. The multi-terminal parallel prediction method according to claim 7, characterized in that, The weighting coefficients of the weighted summation are dynamically determined based on the overall predicted probability distribution of the candidate sequence or the semantic features of the current context.

9. The multi-terminal parallel prediction method according to claim 7, characterized in that, The step of determining the corresponding word in the draft sequence by weighted summation of the first prediction score and the second prediction score includes: Obtain the first score vector corresponding to the first predicted score, and the second score vector corresponding to the second predicted score; Determine the first fusion parameter corresponding to the corrected lexical unit, and the second fusion parameter corresponding to the lexical unit at the corresponding position in the candidate sequence; Calculate the fusion score vector based on the first fusion parameter, the first score vector, the second fusion parameter, and the second score vector; The lexical units at the corresponding positions in the draft sequence are determined based on the fusion score vector.

10. The multi-terminal parallel prediction method according to claim 9, characterized in that, Determining the first fusion parameter corresponding to the corrected lexical unit and the second fusion parameter corresponding to the lexical unit at the corresponding position in the candidate sequence includes: Obtain the third score vector corresponding to each retained word in the candidate sequence; From all the second score vectors and the third score vectors, determine the target score vector with the largest value; The first fusion parameter is calculated based on a preset fusion coefficient and the target score vector; The second fusion parameter is determined based on the first fusion parameter.

11. The multi-terminal parallel prediction method according to claim 1, characterized in that, By comparing the corresponding words in the verification sequence and the draft sequence using a speculative sampling algorithm, the target sequence is obtained, including: Starting from the beginning position, compare the words at the same positions in the verification sequence and the draft sequence sequentially; If all consecutive comparisons up to the kth word are consistent, then the first k words are adopted into the target sequence; If an inconsistency occurs for the first time at the r-th word, the r-th word in the verification sequence is adopted into the target sequence, and the comparison in the current round is terminated. Where k and r are both positive integers and are less than the number of lexical units in the candidate sequence.

12. A multi-term parallel prediction device, characterized in that, include: The parallel prediction module is used to input the current context into the large language model for multi-lexical parallel prediction, generating candidate sequences containing multiple candidate lexical units and the predicted probability of each candidate lexical unit; The sequence generation module is used to filter the candidate sequences based on the predicted probability of each candidate word to identify the word to be optimized and generate the sequence to be optimized. The correction module is used to input the sequence to be optimized into a pre-trained bidirectional attention model to generate a draft sequence; The verification module is used to input the draft sequence into the large language model for verification and generate a verification sequence; The output module is used to compare the corresponding words in the verification sequence and the draft sequence using a speculative sampling algorithm to obtain the target sequence.

13. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for establishing communication between the processor and the memory. The program is executed by the processor to implement the multi-word parallel prediction method as described in any one of claims 1 to 11.

14. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 11.

Images