A dialogue content generation method and an electronic device

Abstract

The application discloses a dialogue content generation method and an electronic device, and relates to the technical field of artificial intelligence, and comprises the following steps: obtaining a target vector constructed based on sampling probabilities of each word element in a word table and output by an activation function of a dialogue model created based on a pre-training model; obtaining a plurality of data blocks by blocking the target vector; filtering out first word elements in each data block based on a target sampling probability threshold to construct a corresponding target set in parallel; performing merge sorting on each target set to obtain a target list; and generating and outputting corresponding target dialogue content based on the target list. By blocking the target vector output by the activation function, filtering out the word elements in each data block to construct the corresponding target set in parallel, and then performing merge sorting to obtain the target list to generate the corresponding target dialogue content, the hardware utilization rate is effectively improved, the data granularity of single processing is reduced, and the dialogue generation efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for generating dialogue content and an electronic device. Background Technology

[0002] As large-scale models expand to mobile devices, sampling optimization is crucial in edge computing scenarios, especially in current technologies for accelerating large language model inference. However, in speculative inference, generating each candidate token requires 1+m sampling operations (where m is the prediction length), significantly increasing the sampling frequency. Traditional sampling takes 5-10ms on a 100K vocabulary, severely limiting inference throughput. Furthermore, current methods require full sorting (O(|V|log|V|)) and prefix sum calculation (O(|V|)) on the complete vocabulary probability distribution (vocabulary length |V|, typically >100,000), which easily leads to resource waste and computational redundancy. Summary of the Invention

[0003] This application provides a dialogue content generation method and electronic device to at least solve the resource waste and computational redundancy caused by performing full sorting of the probability distribution of the complete vocabulary in related technologies. By splitting the probability vector of the complete vocabulary into several independent data blocks, it helps to improve hardware utilization and reduce the data granularity of a single processing, thereby reducing memory bandwidth pressure.

[0004] This application provides a method for generating dialogue content, including:

[0005] Obtain the target vector output by the preset activation function of the dialogue model; the target vector is a vector constructed based on the sampling probabilities of each word in the preset vocabulary of the dialogue model; the dialogue model is a dialogue model created based on a pre-trained model.

[0006] The target vector is divided into blocks based on a preset number of blocks to obtain multiple data blocks;

[0007] The first word element in each data block is selected in parallel based on the target sampling probability threshold, and a target set corresponding to each data block is constructed based on the first word element; the target set is used to store the first word element and the sampling probability corresponding to each first word element.

[0008] The target sets are merged and sorted to obtain a target list. Based on the first word in the target list and its corresponding sampling probability, the corresponding target dialogue content is generated and output.

[0009] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described dialogue content generation methods.

[0010] This application first obtains the target vector output by the preset activation function of the dialogue model created based on the pre-trained model, and constructs it based on the sampling probability of each word in the preset vocabulary of the dialogue model. The target vector is then divided into multiple data blocks based on a preset number of blocks. Then, the first word in each data block is selected in parallel based on the target sampling probability threshold, and a target set corresponding to each data block is constructed based on the first word. After that, the target sets are merged and sorted to obtain a target list. Based on the first word in the target list and the corresponding sampling probability, the corresponding target dialogue content is generated and output.

[0011] This application allows for the partitioning of the target vector output by the activation function into blocks based on a preset number of blocks. Then, based on a target sampling probability threshold, the first word in each data block is selected in parallel to construct the target set corresponding to each data block. Subsequently, the target sets are merged and sorted to obtain a target list. Based on the words in the target list and their corresponding sampling probabilities, the corresponding target dialogue content is generated and output. This approach, by splitting the probability vector of the entire vocabulary into several independent data blocks, and processing each data block in parallel through corresponding processing units, improves hardware utilization, avoids idle hardware computing power, and helps reduce the data granularity of a single processing operation, reducing memory bandwidth pressure. This, through fast data read and write, reduces memory access latency, solving the problem of resource waste and computational redundancy caused by performing full sorting on the probability distribution of the entire vocabulary. Furthermore, merging and sorting the target sets of multiple data blocks to generate a globally ordered target list effectively reduces computational complexity and improves dialogue generation efficiency. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart for generating dialogue content is provided in an embodiment of this application;

[0014] Figure 2 A system architecture diagram for generating dialogue content provided in this application embodiment;

[0015] Figure 3 A specific flowchart for generating dialogue content is provided in this application embodiment;

[0016] Figure 4 A hierarchical architecture diagram for merge sorting is provided in an embodiment of this application;

[0017] Figure 5A hierarchical merging architecture diagram provided for embodiments of this application;

[0018] Figure 6 This is a schematic diagram of a dialogue content generation device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0020] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0021] As large-scale models expand to mobile devices, sampling optimization is crucial in edge computing scenarios for accelerating dialogue model inference. However, traditional sampling takes 5-10ms on a 100K vocabulary, severely limiting inference throughput. Furthermore, current methods require full sorting of the probability distribution of the complete vocabulary, easily leading to resource waste and computational redundancy. This application addresses this by dividing the target vector output by the activation function into blocks, then parallelly selecting the lexical units from each block to construct the corresponding target set. Finally, a merge sort is performed to obtain the target list and generate the corresponding target dialogue content. This effectively improves hardware utilization, reduces the data granularity of a single processing iteration, and increases dialogue generation efficiency.

[0022] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Next, this embodiment will describe the execution flow of the dialogue content generation method in detail, such as... Figure 1 As shown, embodiments of this application provide a method for generating dialogue content, including:

[0024] Step S11: Obtain the target vector output by the preset activation function of the dialogue model; the target vector is a vector constructed based on the sampling probability of each word in the preset vocabulary of the dialogue model; the dialogue model is a dialogue model created based on the pre-trained model.

[0025] In this embodiment, the target vector output by the preset activation function of the dialogue model is first obtained. The dialogue model mentioned above is a dialogue model created based on a pre-trained model, and the target vector is a vector constructed based on the sampling probabilities of each word in the preset vocabulary of the dialogue model. Pre-training is a strategy for training deep learning models. Its core lies in using a large-scale dataset to initially train the model, enabling it to learn general feature representations. In this embodiment, the pre-trained model can specifically be a pre-trained language model, such as a Large Language Model (LLM). Specifically, it involves optimizing the probability sampling algorithm during the inference process of the large language model, and is particularly suitable for scenarios requiring high-frequency sampling, such as speculative decoding, real-time dialogue systems, and code generation, such as dialogue generation scenarios based on in-vehicle voice assistants. A pre-trained language model generally refers to designing a language model training task based on a large-scale corpus, training a large-scale neural network algorithm structure to learn its implementation, and ultimately obtaining the large-scale neural network algorithm structure and parameters. By pre-training on large-scale corpora, neural language representation models can learn powerful language representation capabilities, extracting rich syntactic and semantic information from text. Furthermore, pre-trained language models can provide tokens rich in semantic information and sentence-level features for downstream tasks. Fine-tuning can also be performed directly on the pre-trained model for downstream tasks, facilitating the rapid and convenient acquisition of downstream-specific models. The neural network algorithm structure used to train the pre-trained language model can be CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), or an attention network-based model such as Transformer or BERT; this application does not impose any limitations on this approach.

[0026] Specifically, such as Figure 2The example shown first requires an input probability distribution, which is the raw probability distribution output by the model's activation function, Softmax. This is typically a vector of length |V|, where each element represents the probability of the corresponding token. It's understood that the Softmax function is a core activation function whose core function is to transform the raw scores (Logits) output by the model into a probability distribution that conforms to probability rules. For example, for an input vector of length LLM and vocabulary size |V|, such as 128K, Softmax will calculate for each element in the vector and output a vector of the same length, where each element represents the probability of the corresponding token. It's understood that obtaining the sampling probability can be achieved using a sampling accelerator, specifically PCIe Gen5 (Peripheral Component Interconnect Express 5.0) high-speed interconnect (200GB / s).

[0027] Step S12: Divide the target vector into blocks based on the preset number of blocks to obtain multiple data blocks.

[0028] In this embodiment, the target parameters are first obtained and used as the preset number of blocks. These target parameters are parameters pre-configured according to preset instructions. Alternatively, the number of units in the target processing unit is determined, and the preset number of blocks is determined based on this number of units. The target processing unit is the processor running the dialogue model. Specifically, the vocabulary probability vector is split into N data blocks, where N = BLOCK_SIZE, or N = the number of GPU (Graphics Processing Unit) thread blocks, so that each data block independently executes the subsequent streaming filtering process. It is understood that the BLOCK_SIZE block size can be adaptively obtained, as detailed below:

[0029] ;

[0030] in, It is the maximum block size limited by hardware, such as the GPU thread block limit, with a typical value of 1024; H(p) is the Shannon Entropy of the current probability distribution, calculated in the range of 0 (deterministic distribution) to 15 (uniform distribution). This is a precision control factor used to balance computational efficiency and load balancing, with a value of 0.1 (high precision) to 0.5 (low latency). The number of hardware streaming multiprocessors (SM for GPUs, Core for TPUs (tensor processing units)) for discrete probability distributions. Its Shannon entropy H(P) is defined as:

[0031] ;

[0032] in: It is the probability of the i-th event, satisfying and log2 is the logarithm to the base 2, expressed in bits. In this embodiment, it is defined as follows: It is used to handle events with zero probability.

[0033] Once the preset number of blocks is determined, the target vector can be divided into multiple data blocks based on the preset number of blocks. In other words, this embodiment can divide the entire probability vector into multiple smaller data blocks, and each data block can be assigned to a processing unit for parallel processing, such as using a thread block of the GPU as a processing unit.

[0034] Step S13: Based on the target sampling probability threshold, the first word element in each data block is selected in parallel, and the target set corresponding to each data block is constructed based on the first word element; the target set is used to store the first word element and the sampling probability corresponding to each first word element.

[0035] In this embodiment, the first word element in each data block can be selected in parallel based on a target sampling probability threshold, and a target set corresponding to each data block can be constructed based on the first word element. The target set is used to store the first word element and the sampling probability corresponding to each first word element. Furthermore, based on the previous step, when selecting the first word element in each data block in parallel based on the target sampling probability threshold, the target processing unit can perform the word element selection operation of the corresponding data block in parallel, so as to select the first word element in each data block based on the target sampling probability threshold through the word element selection operation.

[0036] Specifically, when selecting the first word in each data block in parallel based on the target sampling probability threshold, the initial sampling probability threshold can be used as the target sampling probability threshold, and the first word in the data block can be saved as the first word in the preset candidate set according to the target sampling probability threshold; the initial sampling probability threshold is a floating-point value used to represent negative infinity, i.e. =-float('inf'); Then, based on the sampling probability of the first word, the first word in the corresponding data block is selected and each first word is saved to the preset candidate set. After that, when the first word in the preset candidate set meets the preset selection conditions, the preset candidate set is used as the target set.

[0037] Furthermore, when selecting the first word in the corresponding data block based on the sampling probability of the first word, the target sampling probability threshold can be updated based on the sampling probability of the first word to obtain the updated target sampling probability threshold. The next word in the data block is taken as the target word, and it is determined whether the sampling probability of the target word is not less than the updated target sampling probability threshold. If the sampling probability of the target word is not less than the updated target sampling probability threshold, the target word is saved to the preset candidate set, and the updated target sampling probability threshold is updated again based on the sampling probability of the target word to obtain a new updated target sampling probability threshold. The next word in the data block is taken as the new target word, and then the process jumps to the step of determining whether the sampling probability of the target word is not less than the updated target sampling probability threshold. If the sampling probability of the next word is less than the updated target sampling probability threshold, the next word in the data block is directly taken as the new target word, and then the process jumps to the step of determining whether the sampling probability of the target word is not less than the updated target sampling probability threshold.

[0038] In other words, in this embodiment, dynamic thresholds can be used when performing streaming filtering on each data block. This is used to filter candidate tokens. The dynamic threshold τ can be updated adaptively in real time to achieve efficient filtering. First, [the following is a list of parameters]... Perform initialization, that is The function `=-float('inf')` ensures that the first element unconditionally enters the candidate set, i.e., the target set. Then, a dynamic threshold update is performed based on the probability of the first element, i.e.:

[0039] ;

[0040] in, The threshold of the current time step; The probability value of the token being processed; This is the current candidate set, i.e., the target set; For the k largest elements in set S, only if the probability value is... Only when this happens will it be considered for inclusion in the candidate set.

[0041] It should be noted that in this embodiment, the updating of tokens in the candidate set can be achieved by maintaining a min-heap. Specifically, a corresponding binary tree can be generated based on a preset candidate set, and the sampling probability of the first word can be used as the target value of the binary tree. The binary tree is constructed based on a min-heap, and the target value is the top value of the min-heap. Correspondingly, the target word is saved to the preset candidate set, and after updating the target sampling probability threshold again based on the sampling probability of the target word, the target word can be saved to the binary tree, and the sampling probability of the target word can be used as the new target value. The cumulative sum of the sampling probabilities corresponding to each first word in the binary tree is determined. This embodiment can also update the global tokens through atomic operations. :

[0042] ;

[0043] in, This is a global dynamic threshold, a threshold shared by all processing units, used for candidate filtering across threads / cores; The local dynamic threshold is the minimum heap top value of the candidate set for the current processing unit. This is the updated global threshold. Furthermore, this embodiment allows for predictive threshold adjustment, with the adjusted threshold being:

[0044] ;

[0045] in: , is the smoothing coefficient, with an empirical value of 0.3-0.7; It represents the top value of the min-heap in the local candidate set at the current time step.

[0046] Understandably, as the number of elements processed in the candidate set increases, the minimum value of the heap increases with the addition of larger elements. Gradually rising. When After rising to a certain value, if subsequent elements are less than... These are filtered out, thus reducing unnecessary comparisons and heap operations. Ultimately, It will converge to the Kth largest probability value globally, i.e., the boundary value of TopK. In other words, this embodiment can maintain a min-heap of size K for TopK constraints and a cumulative sum for TopP constraints to track candidate tokens in the current block. When an element is added, the heap and cumulative sum are updated. If the heap size exceeds K, the top of the heap (the minimum value) is popped and subtracted to update the cumulative sum. If the cumulative sum exceeds P, the process terminates. In this way, when traversing elements within a block, only those with probability values ​​greater than or equal to the current threshold are considered. The elements are processed and the candidate set and threshold are updated. If the termination condition is met, such as the candidate set size reaching K or the accumulated probability reaching P, the traversal of that block is terminated early. The TopK constraint maintains a min-heap of capacity K, retaining only those with probabilities greater than or equal to... The elements; the TopP constraint is to calculate the cumulative probability S in real time, and terminate the traversal early when S is greater than or equal to P.

[0047] Specifically, in one embodiment, it can be determined whether the number of tokens in the binary tree corresponding to the preset candidate set meets the preset token number condition, and whether the cumulative sum of the binary tree is greater than the preset cumulative sum threshold. When the cumulative sum is greater than the preset cumulative sum threshold, the preset candidate set is directly used as the target set; when the number of tokens meets the preset token number condition, but the cumulative sum is less than the preset cumulative sum threshold, the token corresponding to the top value in the preset candidate set is deleted, and the process jumps to the step of using the next token after the first token in the data block as the target token, until the cumulative sum is not less than the preset cumulative sum threshold, at which point the preset candidate set is used as the target set.

[0048] In another specific embodiment, it can be determined whether the number of words in the binary tree corresponding to the preset candidate set meets the preset word quantity condition, or whether the cumulative sum corresponding to the binary tree is greater than the preset cumulative sum threshold; when the number of words meets the preset word quantity condition, or the cumulative sum is not less than the preset cumulative sum threshold, the preset candidate set is directly used as the target set.

[0049] In other words, when performing streaming probability filtering in this embodiment, a min-heap is first initialized to maintain the current TopK candidates, an accumulator S=0 of type FP32 used to track the cumulative probability of the candidate set, and a dynamic threshold of type FP32 used for dynamic filtering boundaries. =0, at this point, all elements may be added initially in the stream filtering process because =0, and any probability >= 0. It's understandable that, since the probability of any token >= 0, we can... The probability of each token is then adjusted from -float('inf') to 0. If the current probability value is large enough, exceeding the threshold Or, if the cumulative probability of the candidate set does not reach the target P, then... Add to the heap. Specifically, if the heap size is less than K, add directly; otherwise, if... If the value is greater than the top of the heap, replace the top element by popping it from the heap, then pushing the new element onto the heap, subtracting the popped value, and adding the new element's value to update the accumulator S. If the heap size is not yet K, simply add the new element and add its value. Then update... The loop iterates through the heap, identifying the top element (the Kth largest value when the heap size reaches K) and checking if the accumulator S is greater than or equal to P, and if the length of the min-heap is equal to K. If so, the loop exits. After the loop ends, the elements in the heap form the candidate set. This candidate set is less than or equal to K, and consists of the largest elements, provided the loop exits early and the cumulative sum is greater than or equal to P, or the cumulative sum has not reached P but the K largest probabilities have been selected.

[0050] It should be noted that in this embodiment, a hybrid precision architecture of FP8 (8-bit floating point) pre-screening and FP32 (Single-precision floating-point) refinement is adopted in the block-parallel streaming filtering stage. First, low-precision fast filtering is achieved through FP8 pre-screening, and FP32 probability vectors are batch-converted to FP8 format (E5M2 or E4M3). Threshold comparison is then performed in the FP8 space. Furthermore, in the TopP constraint, the cumulative sum S needs to be calculated in real time. Traditional accumulation still has rounding errors under FP32, and the error can reach 10^10 after 10,000 accumulations. -3 The magnitude of the error is such that normalized probability bias affects sampling fairness, leading to premature termination failures or distribution distortion. In this embodiment, the integrated FP32 refinement in the flow filtering uses Kahan summation to compensate for accumulated errors.

[0051] ;

[0052] The specific implementation is as follows:

[0053] / / When a new element p i When adding to the heap:

[0054] float old_min=heap.replace(p i ); / / Replace the top of the heap

[0055] / / Kahan-style incremental update cumulative sum:

[0056] float y=p i -old_min-err; / / Incremental value + error compensation

[0057] float t = S + y;

[0058] err=(tS)–y; / / Update error term

[0059] S=t.

[0060] Thus, this embodiment utilizes a hybrid precision architecture of FP8 pre-screening and FP32 refining (Kahan), reducing data processing volume by 4 times using FP8, and reducing the error from 10^ -3 Reduced to 10^ -6 This technology improves energy efficiency, enables hybrid precision computing and error control, and is suitable for deployment across cloud, edge, and device scenarios. It supports dynamic voltage and frequency scaling (DVFS) during edge deployment and enables FP8 quantization mode, automatically reducing precision when temperature exceeds 80℃. In practical engineering edge deployment optimizations, DVFS dynamically balances computational performance with power consumption / heat generation. The temperature-triggered FP8 precision reduction mode further reduces computational load at the algorithm level, automatically switching to a lower precision, lower power consumption mode when the temperature threshold is triggered. For example, in edge-cloud collaborative inference scenarios, the edge uses streaming filtering (FP8 quantization) to quickly generate the first token, while the cloud uses full-precision merge sort and dynamic vocabulary for long sequence refinement. For instance, in in-vehicle voice assistants, drafts are generated at the edge in offline mode, and the cloud optimizes the output after connecting to the network, reducing end-to-end latency from approximately 220ms to 80ms.

[0061] Step S14: Merge and sort the target sets to obtain a target list, and generate and output the corresponding target dialogue content based on the first word in the target list and the corresponding sampling probability.

[0062] In this embodiment, the probabilities of each target set, i.e., the candidate set, are merged and sorted to obtain the target list. Based on the first word in the target list and its corresponding sampling probability, the corresponding target dialogue content is generated and output. It can be understood that the local candidate set of each block output in the previous step (…) Each local candidate set consists of candidate tokens and their probability values ​​selected from the corresponding data block after streaming filtering. Because only tokens that meet the conditions are retained, the size of these candidate sets is much smaller than the original data block, and they are unsorted or partially ordered, such as a min-heap structure. Then, the GPU's merge sort API (Application Programming Interface) is used to efficiently merge these candidate sets using a multi-way merge sort algorithm, achieving multi-way merge sort aggregation and combining the local candidate sets from various processing units into a globally ordered candidate set.

[0063] The above technical solution divides the target vector output by the activation function into blocks based on a preset number of blocks. Then, based on the target sampling probability threshold, the first word in each data block is selected in parallel to construct the target set corresponding to each data block. Afterwards, the target sets are merged and sorted to obtain a target list. Based on the words in the target list and their corresponding sampling probabilities, the corresponding target dialogue content is generated and output. In this way, by splitting the probability vector of the entire vocabulary into several independent data blocks, each data block is processed in parallel by corresponding processing units, improving hardware utilization, avoiding idle hardware computing power, and helping to reduce the data granularity of a single processing operation, reducing memory bandwidth pressure. This solves the problem of resource waste and computational redundancy caused by performing full sorting on the probability distribution of the complete vocabulary, and further reduces computational complexity and improves dialogue generation efficiency by merging and sorting the target sets of multiple data blocks to generate a globally ordered target list. Furthermore, this embodiment can extend the streaming filtering strategy to the text-image-speech joint generation scenario through multimodal joint optimization, which is achieved through a dynamic threshold sharing mechanism and cross-modal probability alignment, establishing a unified probability normalization space, so that the sampling thresholds of different modalities can be compared horizontally.

[0064] As described in the previous embodiment, this application can divide the target vector output by the activation function into blocks, then select the lexical units in each data block in parallel to construct the corresponding target set, and then perform merging and sorting to obtain the target list and generate the corresponding target dialogue content. Next, this embodiment will describe in detail the process of merging and sorting the target set and generating the dialogue. See [link to previous embodiment]. Figure 3 As shown in the figure, this application discloses a specific method for generating dialogue content, including:

[0065] Step S21: Sort the sampling probabilities corresponding to the first word in each target set to obtain the sorted target set.

[0066] In this embodiment, when performing multi-way merge sorting on the candidate sets corresponding to each data block, the sampling probabilities corresponding to the first word in each target set, i.e. the candidate set, can be sorted to obtain the sorted target set.

[0067] Step S22: Construct a priority queue based on the first sampling probability in each sorted target set, and merge and sort each sorted target set based on the priority queue to obtain a target list.

[0068] In this embodiment, a priority queue can be constructed based on the first sampled probability in each sorted target set. The sorted target sets are then merged and sorted based on the priority queue to obtain a target list. That is, the min-heap output from streaming filtering can usually be directly used for merging. However, if the local candidate sets are unordered, they are sorted first, and then a priority queue (min-heap or max-heap, depending on the sorting direction) is used to merge multiple ordered lists. Specifically, the current largest (or smallest) element is taken from each local candidate set and placed into the priority queue. Then, elements are taken out one by one from the queue to form a globally ordered list. Afterwards, the global candidate set is truncated according to the TopK or TopP requirements: for TopK, the top K tokens with the highest probabilities are taken; for TopP, the cumulative probability is calculated on the globally ordered list until a threshold P is reached. This results in a global TopK / TopP set after multi-way merge sorting and aggregation, i.e., a global candidate token set that satisfies the TopK or TopP conditions, arranged in descending order of probability. For example, take the first element (i.e. the largest element of the list) from each local ordered list and put it into a priority queue. Then, take the global largest element from the priority queue and add it to the global ordered list. If the list is not empty, take the next element from the local list to which the element belongs and put it into the priority queue. Continue until the priority queue is empty and all elements of the local lists have been processed. At this time, the global ordered list contains the candidate elements of all blocks and is sorted in descending order of probability. In one specific embodiment, if there are 3 local lists, A: [0.5, 0.3, 0.2]; B: [0.4, 0.35, 0.1]; C: [0.6, 0.25, 0.15]), the initial queue is: 0.5 (A), 0.4 (B), 0.6 (C). The maximum value 0.6 (C) is added to the global list. Then, the next element of C, 0.25, is added to the priority queue, i.e.: 0.5 (A), 0.4 (B), 0.25 (C). 0.5 (A) is added to the global list. Then, the next element of A, 0.3, is added to the priority queue, i.e.: 0.4 (B), 0.3 (A), 0.25 (C). 0.4 (B) is added to the global list. The final completely ordered global list is: [0.6, 0.5, 0.4, 0.35, 0.3, 0.25, 0.2, 0.15, 0.1].

[0069] Based on the above technical solution, and combined with the block processing process in the previous embodiment, as follows: Figure 4 The process begins with block-based parallel filtering, where the vocabulary probability vector is split into N data blocks, where N = BLOCK_SIZE or N = the number of GPU thread blocks. Each block is independently stream-filtered, generating a local candidate set. The process involves: each block having a candidate set consisting of a heap and a cumulative sum, but local candidate sets may terminate early because they already meet the TopP condition. After processing each block, elements are retrieved from the heap and sorted in descending order to obtain a locally ordered list. Then, multi-way merging is used to merge these ordered lists into a globally ordered list, and the cumulative sum is calculated on the globally ordered list until P is reached or the top K tokens are selected. It should be noted that both TopK and TopP can be satisfied simultaneously: that is, the top K tokens are selected first, and then the cumulative sum is calculated to reach P without exceeding K. In fact, the cumulative sum can be recalculated from largest to smallest on the merged globally ordered list until P is reached, but at most K tokens can be selected. Therefore, it is only necessary to traverse this globally ordered list. Since the length of the globally ordered list is at most the sum of the sizes of all local candidate sets, but the size of each local candidate set does not exceed K, the total length does not exceed the number of blocks multiplied by K. The number of blocks is generally fixed, such as the number of thread blocks in a GPU, so the total length of the globally ordered list will not be very large. Figure 4 As shown, during parallel execution of streaming filtering, threshold synchronization between threads can be achieved through Warp (Warp Cooperative Groups) level cooperation using CUDA (Compute Unified Device Architecture) Shuffle instructions (speedup of 3.2x). Furthermore, SIMD (Single Instruction, Multiple Data) vectorization and AVX-512 (Advanced Vector Extensions-512) mask loading can reduce conditional branches (speedup of 2.1x). Specifically, in streaming filtering, each thread needs to know the current dynamic threshold. If each thread reads and writes to global memory or shared memory... This can lead to severe lock contention and memory latency, becoming a performance bottleneck. Therefore, streaming filtering needs to quickly find the maximum (or minimum) value among the 32 probability values ​​currently processed by the Warp and use it to update the threshold. This implementation, for example... Figure 4 , Figure 5 The example demonstrates how the CUDA Shuffle instruction (__shfl_sync) allows threads within the same warp (32 threads) to directly read register values ​​from other threads without needing shared or global memory, achieving extremely low-latency inter-thread communication at the register level. It's also understandable that during streaming filtering on the CPU (Central Processing Unit), the most time-consuming part is the conditional branch (if(p...). i >= Currently, CPUs rely on branch prediction, but when the probability distribution is irregular, the prediction failure rate is very high, leading to pipeline stagnation. In this embodiment, AVX-512 introduces a mask register (k0-k7), which allows almost all instructions to become conditional execution. The mask can be set with a comparison operation first, and then loading and calculation are performed only on the channels where the mask is true, without any branch jumps, thus achieving hardware optimization for parallel streaming filtering.

[0070] Furthermore, in this embodiment, when sorting is implemented using the merge sort API, on the GPU, the DeviceMergeSort function of the CUB library (CUDA Unbound, an open-source template library) (or Thrust's sorting) can be used to sort each local candidate set. Since each candidate set is small, multiple candidate sets can be sorted in parallel, with each thread block sorting one candidate set. The CUB library provides DeviceSegmentedSort or MultiMerge functionality; alternatively, a global priority queue can be used to implement an efficient multi-way merge on the GPU. Hardware-optimized merge sort APIs (such as CUDA's merge_sort, TPU's XLA:Merge (XLA, or Accelerated Linear Algebra)) can also be called to perform parallel merging of local candidate sets. XLA:Merge requires input data to be stored contiguously in memory, enabling the TPU's DMA (Direct Memory Access) engine to prefetch large blocks of data and reduce memory access frequency. Specifically, it first generates a locally ordered candidate set. After each TPU core processes a data block, it outputs a locally ordered list (in descending order of probability). Then, XLA:Merge is called to perform multi-way merging. The XLA compiler maps the merging operation to optimized TPU instructions, and then explicitly plans the data layout to optimize memory layout, ensuring contiguous memory access and implementing a hardware-accelerated API. The complexity of the merging operation is reduced from O(|V|log|V|) to O(N*K*logK) (where N is the number of blocks). Finally, after obtaining a globally ordered candidate set in descending order of probability, the cumulative probability, i.e., the prefix sum, is calculated. Then, based on the user-defined TopP value, the prefix with a cumulative probability of P is truncated. This prefix may be smaller than the entire candidate set, and multinomial sampling is performed on this subset of prefixes. In this embodiment, under GPU-TPU heterogeneous computing, image tokens (such as Latent Space in Stable Diffusion) and text tokens share the merge sort pipeline, and the actual generation speed is improved by 20% through hardware collaborative scheduling.

[0071] Furthermore, this embodiment achieves memory bank conflict avoidance through interleaved storage, i.e., buf[threadIdx.x+(bank_id%4)*16]. Here, buf is a shared memory array used to store temporary data; threadIdx.x is the thread index of the CUDA thread (from 0 to blockDim.x-1); bank_id is the memory bank number, calculated as bank_id=threadIdx.x%32; (bank_id%4) is the bank number modulo 4, distributing conflicts across 4 banks; and 16 indicates that the step offset is stored at multiples of 16. It can be understood that in this embodiment, each thread block processes a local candidate set, such as a min-heap of size K. Before merging, this heap needs to be converted into an ordered array, which typically requires sorting or adjusting within shared memory. For example: A thread block has 128 threads, and the shared memory array buf

[128] is used to store the probability values ​​of the candidate set. If each thread simply stores data according to its own thread ID (threadIdx.x) as the index, then because the number of banks is 2 to the power of 5, we look at the [4:0] bits of the address, and the number of the bank is determined by the low bits of the memory address. The addresses of buf[0], buf[1], ..., buf

[31] are exactly distributed on 32 different banks. At this time, threads 0-31 access different banks at the same time without conflict, and the efficiency is the highest. However, when the thread reads data for merge comparison, it usually needs to let thread 0 access buf[0], buf[1], buf[2]... to perform serial merge operation. At this time, the addresses accessed by adjacent threads are no longer staggered, but very close, and it is very easy to access the same bank, resulting in serious bank conflict. Therefore, in order to solve this problem, the following formula is used for interleaved storage to artificially scatter the data to different banks:

[0072] index=threadIdx.x+(bank_id % 4)*16;

[0073] The calculation process is as follows: `bank_id = threadIdx.x % 32` calculates the Bank ID for each thread; `(bank_id % 4)` divides the 32 Banks into 4 groups (0-3, 4-7, ..., 28-31). Threads with `bank_id` values ​​of 0, 4, 8, ..., 28 belong to group 0, and so on, reducing conflict granularity; 16 is a stride. By multiplying by a sufficiently large number (16), it ensures that the memory address intervals accessed by threads within the same group are large enough to map to different Banks; during the calculation of the above index, adding the original index to `threadIdx.x` guarantees address uniqueness. In this way, in the multi-way merge sort aggregation step, by leveraging the GPU's memory architecture characteristics through interleaved storage, the memory access mode that would otherwise lead to severe serialization is transformed into a highly parallel mode, achieving multi-way merge optimization.

[0074] Step S23: Based on the first word in the target list and its corresponding sampling probability, generate and output the corresponding target dialogue content.

[0075] In this embodiment, final sampling can be performed based on the target list, that is, multinomial sampling is performed on the global TopK / TopP set. Firstly, because the probability sum after TopK / TopP operations is not necessarily 1, the token probabilities in the set need to be renormalized. Then, based on the normalized probability distribution, an algorithm such as Roulette Wheel Selection is used to sample a token as the output. In this way, the entire process avoids full sorting and computation through parallel processing and streaming filtering, greatly reducing computational complexity, while maintaining the accuracy of the sampling results without approximation errors. In other words, this embodiment can first select a preset number of first words from the target list as dialogue words; or, select several first words from the target list that meet a preset cumulative sum condition as dialogue words; the preset cumulative sum condition is that the cumulative sum of the sampling probabilities of the dialogue words is not less than a preset cumulative sum threshold; then, corresponding target dialogue content is generated based on each dialogue word. Specifically, the dialogue words are first sorted based on their sampling probabilities to obtain sorted dialogue words. These sorted words are then normalized to obtain normalized words. The corresponding target dialogue content is then generated based on these normalized words. That is: first, a probability interval is constructed based on the sampling probability of each normalized word, and random numbers are generated. Based on the random numbers and probability intervals, corresponding normalized words are determined as second words. Then, the process returns to generating random numbers and determining corresponding normalized words as second words based on the random numbers and probability intervals, until the number of second words meets a preset word quantity condition. Finally, the corresponding target dialogue content is generated based on each second word.

[0076] For example, we can extract all elements from the global candidate set G whose cumulative probability is first greater than or equal to P, forming a TopP candidate subset S. Then, we re-normalize the probability of subset S to ensure sampling fairness. Specifically, we can divide the probability of each element in subset S by the total probability of subset S to obtain the normalized probability. This ensures that the selection probability of each element during sampling is consistent with its relative weight in the global distribution, helping to achieve zero precision loss. Finally, we perform multinomial sampling on the normalized subset S to generate tokens. For example, when using the roulette wheel algorithm, we first construct a probability interval based on the normalized probability, generate a random number between [0, 1), determine the interval corresponding to the random number, and select the token within that interval. In this way, elements with higher probabilities have wider intervals, which better aligns with the logic of selecting high-probability tokens, while also ensuring the diversity of model dialogue generation.

[0077] The above technical solution originally required sorting the entire vocabulary |V|log|V|, but in this embodiment, each block is sorted using an array of size K, meaning each block sorts K elements, resulting in a complexity of KlogK. With N blocks, the total complexity is O(N*K*logK), and the complexity of multi-way merge sort is O(N*K*logN). Since N and K are much smaller than |V|, the total computational cost is significantly reduced. For example, for TopK, it is reduced from O(|V|log|V|) to O(N*K*logK); for TopP, the average number of traversals is reduced to O(P⁻¹) (P is the threshold), far lower than O(|V|). Simultaneously, it is hardware-accelerated, streaming filtering has no data dependency, and is suitable for SIMT (Single Instruction, Multiple Threads) parallelism, reducing computational cost. Furthermore, by calling the vendor's optimized API through merge sort, memory throughput is maximized, accelerating the merging process. At the same time, the output distribution is completely consistent with traditional methods, with no approximate calculations, achieving zero precision loss. In scenarios where speculative inference gain m>10, the improvement in sampling speed directly translates into end-to-end inference acceleration. Through integration and verification using the LLM inference framework (vLLM+TensorRT-LLM), it can be seamlessly deployed to all cloud, edge, and endpoint scenarios. For example, in cloud inference services, the sampling latency for a 100K vocabulary is <1ms (thousands of concurrent users); the inference speed of mobile real-time dialogue-side LLM (such as GPT-3.5Turbo) is improved by 8.7x; and multimodal generation achieves end-to-end acceleration of joint text-image sampling. Furthermore, webpage sorting can be replaced with LLM streaming sampling, reducing the result generation speed from approximately 300ms to 30ms, while also supporting personalized real-time adjustments. In a test environment with |V|=128K, m=5 speculative inference, and Batch=32, TopK (K=50) was accelerated from 8.2ms to 0.9ms, a speedup of 9.1x; TopP (P=0.95) was accelerated from 7.8ms to 0.7ms, a speedup of 11.1x; and the joint sampling of TopK and TopP (K=50, P=0.9) was accelerated from 9.1ms to 1.1ms, a speedup of 8.3x. End-to-end latency compression and lightweighting for edge scenarios were achieved, providing fundamental support for real-time deployment of models with hundreds of billions of records.

[0078] like Figure 6 As shown, embodiments of this application also provide a dialogue content generation apparatus, including:

[0079] The vector acquisition module 11 is used to acquire the target vector output by the preset activation function of the dialogue model; the target vector is a vector constructed based on the sampling probability of each word in the preset vocabulary of the dialogue model; the dialogue model is a dialogue model created based on a pre-trained model.

[0080] Vector segmentation module 12 is used to segment the target vector into multiple data blocks based on a preset number of blocks.

[0081] The lexical filtering module 13 is used to filter out the first lexical in each data block in parallel based on the target sampling probability threshold, and to construct the target set corresponding to each data block based on the first lexical; the target set is used to store the first lexical and the sampling probability corresponding to each first lexical.

[0082] The dialogue generation module 14 is used to merge and sort the target sets to obtain a target list, and generate and output the corresponding target dialogue content based on the first word in the target list and the corresponding sampling probability.

[0083] For a description of the features in the embodiments corresponding to the dialogue content generation device described above, please refer to the relevant descriptions in the embodiments corresponding to the dialogue content generation method, which will not be repeated here.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0085] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described embodiments of the dialogue content generation method.

[0086] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the dialogue content generation method when it is run.

[0087] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0088] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described dialog content generation method embodiments.

[0089] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described dialog content generation method embodiments.

[0090] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0091] The foregoing has provided a detailed description of a dialogue content generation method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for generating dialogue content, characterized in that, include: Obtain the target vector output by the preset activation function of the dialogue model; The target vector is a vector constructed based on the sampling probabilities of each word in the preset vocabulary of the dialogue model; The dialogue model is a dialogue model created based on a pre-trained model; The target vector is divided into multiple data blocks based on a preset number of blocks. The first word element in each data block is selected in parallel based on the target sampling probability threshold, and the target set corresponding to each data block is constructed based on the first word element. The target set is used to store the first word element and the sampling probability corresponding to each first word element; The target sets are merged and sorted to obtain a target list, and the corresponding target dialogue content is generated and output based on the first word element in the target list and the corresponding sampling probability. The step of parallelly filtering the first word element in each data block based on the target sampling probability threshold, and constructing the target set corresponding to each data block based on the first word element, includes: The initial sampling probability threshold is used as the target sampling probability threshold, and the first word in the data block is saved as the first word in the preset candidate set according to the target sampling probability threshold; the initial sampling probability threshold is a floating-point value used to represent negative infinity; Based on the sampling probability of the first word, each of the first words in the corresponding data block is selected, and each of the first words is saved to the preset candidate set; Once the first word in the preset candidate set meets the preset filtering conditions, the preset candidate set is used as the target set.

2. The dialogue content generation method according to claim 1, characterized in that, Before dividing the target vector into multiple data blocks based on a preset number of blocks, the method further includes: Obtain the target parameters and use the target parameters as the preset number of blocks; the target parameters are parameters pre-configured according to preset instructions; Alternatively, determine the number of units in the target processing unit, and determine the preset number of blocks based on the number of units; the target processing unit is a processor that runs the dialogue model; Accordingly, the parallel filtering of the first word element in each of the data blocks based on the target sampling probability threshold includes: The target processing unit performs lexical filtering operations on the corresponding data blocks in parallel to filter out the first lexical in each data block based on the target sampling probability threshold.

3. The dialogue content generation method according to claim 1, characterized in that, The step of filtering out each first word in the corresponding data block based on the sampling probability of the first word includes: The target sampling probability threshold is updated based on the sampling probability of the first word to obtain the updated target sampling probability threshold; The next word after the first word in the data block is taken as the target word, and it is determined whether the sampling probability of the target word is not less than the updated target sampling probability threshold. If the sampling probability of the target word is not less than the updated target sampling probability threshold, then the target word is saved to the preset candidate set, and the updated target sampling probability threshold is updated again based on the sampling probability of the target word to obtain a new updated target sampling probability threshold. The next word of the target word in the data block is taken as the new target word, and then the process jumps to the step of determining whether the sampling probability of the target word is not less than the updated target sampling probability threshold. If the sampling probability of the next word is less than the updated target sampling probability threshold, then the next word of the target word in the data block is directly taken as the new target word, and then the process jumps to the step of determining whether the sampling probability of the target word is not less than the updated target sampling probability threshold.

4. The dialogue content generation method according to claim 3, characterized in that, The step of saving the first word in the data block as the first word in the preset candidate set according to the target sampling probability threshold includes: A corresponding binary tree is generated based on the preset candidate set, and the sampling probability of the first word is used as the target value of the binary tree; the binary tree is a binary tree constructed based on a min-heap, and the target value is the top value of the min-heap; Accordingly, after saving the target lexical unit to the preset candidate set and updating the updated target sampling probability threshold based on the sampling probability of the target lexical unit, the method further includes: The target word is saved into the binary tree, and the sampling probability of the target word is used as the new target value. The cumulative sum of the sampling probabilities corresponding to each first word in the binary tree is determined.

5. The dialogue content generation method according to claim 4, characterized in that, After the first word element in the preset candidate set satisfies the preset filtering conditions, the preset candidate set is used as the target set, including: Determine whether the number of lexical units in the binary tree corresponding to the preset candidate set meets the preset lexical unit number condition, and determine whether the cumulative sum corresponding to the binary tree is greater than the preset cumulative sum threshold; When the cumulative sum is greater than the preset cumulative sum threshold, the preset candidate set is directly used as the target set; When the number of lexical elements meets the preset lexical element quantity condition and the cumulative sum is less than the preset cumulative sum threshold, delete the lexical element corresponding to the top value of the heap in the preset candidate set, and jump to the step of taking the next lexical element of the first lexical element in the data block as the target lexical element, until the cumulative sum is not less than the preset cumulative sum threshold, and then take the preset candidate set as the target set. Alternatively, determine whether the number of lexical units in the binary tree corresponding to the preset candidate set meets the preset lexical unit number condition, or determine whether the cumulative sum corresponding to the binary tree is greater than the preset cumulative sum threshold; When the number of lexical units meets the preset lexical unit number condition, or when the cumulative sum is not less than the preset cumulative sum threshold, the preset candidate set is directly used as the target set.

6. The dialogue content generation method according to any one of claims 1 to 5, characterized in that, The process of merging and sorting the target sets to obtain a target list includes: The sampling probabilities corresponding to the first word in each of the target sets are sorted to obtain sorted target sets; A priority queue is constructed based on the first sampling probability in each sorted target set, and the sorted target sets are merged and sorted based on the priority queue to obtain the target list.

7. The dialogue content generation method according to claim 6, characterized in that, The step of generating and outputting corresponding target dialogue content based on the first word element in the target list and the corresponding sampling probability includes: A preset number of the first word elements in the target list are selected as dialogue word elements; Alternatively, select a number of the first word elements in the target list that meet the preset cumulative sum condition as the dialogue word elements; the preset cumulative sum condition is that the cumulative sum of the sampling probabilities of the dialogue word elements is not less than the preset cumulative sum threshold. The target dialogue content is generated based on each of the aforementioned dialogue lexical units.

8. The dialogue content generation method according to claim 7, characterized in that, The step of generating the corresponding target dialogue content based on each of the dialogue lexical units includes: The dialogue words are sorted based on the sampling probability corresponding to each of the dialogue words to obtain sorted dialogue words. The sorted dialogue lexical units are normalized to obtain normalized lexical units, and a corresponding probability interval is constructed based on the sampling probability of each normalized lexical unit. Generate random numbers, and determine the corresponding normalized lexical as the second lexical based on the random numbers and the probability interval; The process then jumps back to the step of generating random numbers and determining the corresponding normalized lexical units as second lexical units based on the random numbers and the probability interval, until the number of second lexical units meets the preset lexical unit quantity condition, and then generates the corresponding target dialogue content based on each second lexical unit.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the dialogue content generation method as described in any one of claims 1 to 8 when executing the computer program.

Images