Conversation content generation method and electronic equipment

Abstract

The invention discloses a dialogue content generation method and electronic equipment, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a target vector which is output by an activation function of a dialogue model created based on a pre-training model and is constructed based on the sampling probability of each lexical element in a word list, carrying out the partitioning of the target vector, and obtaining a plurality of data blocks, and then, based on a target sampling probability threshold, screening out the first lexical units in each data block in parallel to construct corresponding target sets, then, merging and sorting the target sets to obtain a target list, and generating and outputting corresponding target dialogue content based on the target list. According to the method, the target vector output by the activation function is partitioned, then the lexical units in the data blocks are screened out in parallel to construct the corresponding target set, and then the target set is merged and sorted to obtain the target list to generate the corresponding target dialogue content, so that the hardware utilization rate is effectively improved, the data granularity of single processing is reduced, and the dialogue generation efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a dialogue content generation method and an electronic device. BACKGROUND

[0002] With the expansion of large models to mobile terminals, in the current large language model inference acceleration technology, sampling optimization is crucial in edge computing scenarios. However, in speculative inference, 1+m samplings (m is the prediction length) are performed for each generated candidate token, the frequency of sampling operations is significantly increased, and the traditional sampling requires 5-10 ms on a 100K vocabulary table, which seriously restricts the inference throughput. Moreover, the full sorting (O (|V|log|V|)) and prefix sum calculation (O (|V|)) on the complete vocabulary probability distribution (vocabulary length |V|, usually > 100,000) are prone to cause resource waste and computational redundancy. SUMMARY

[0003] The present application provides a dialogue content generation method and an electronic device to at least solve the resource waste and computational redundancy caused by performing full sorting on the complete vocabulary probability distribution in related technologies. By splitting the probability vector of the full vocabulary table into several independent data blocks, it helps to improve hardware utilization and reduce the data granularity of single processing, thereby reducing the memory bandwidth pressure.

[0004] The present application provides a dialogue content generation method, comprising:

[0005] obtaining a target vector output by a preset activation function of a dialogue model; the target vector is a vector constructed based on the sampling probability of each vocabulary item in a preset vocabulary table of the dialogue model; the dialogue model is a dialogue model created based on a pre-training model;

[0006] performing block division on the target vector based on a preset block number to obtain a plurality of data blocks;

[0007] parallel filtering the first vocabulary item in each data block based on a target sampling probability threshold, and constructing a target set corresponding to each data block based on the first vocabulary item; the target set is used to save the first vocabulary item and the sampling probability corresponding to each first vocabulary item;

[0008] performing merge sorting on each target set to obtain a target list, and generating and outputting the corresponding target dialogue content based on the first vocabulary item and the corresponding sampling probability in the target list.

[0009] The present application also provides an electronic device, comprising: a memory for storing a computer program; a processor for executing the computer program to implement the steps of any one of the dialogue content generation methods described above.

[0010] The preset activation function output of the dialogue model created based on the pre-training model is first acquired, a target vector constructed based on the sampling probability of each word piece in the preset vocabulary of the dialogue model, and the target vector is blocked based on the preset block number to obtain a plurality of data blocks, then the first word piece in each data block is screened out in parallel based on the target sampling probability threshold, and the target set corresponding to each data block is constructed based on the first word piece, then each target set is merged and sorted to obtain a target list, and the corresponding target dialogue content is generated and output based on the first word piece and the corresponding sampling probability in the target list.

[0011] Through the present application, the target vector output by the activation function can be blocked based on the preset block number, then the first word piece in each data block is screened out in parallel based on the target sampling probability threshold to construct the target set corresponding to each data block, then each target set is merged and sorted to obtain a target list, and the corresponding target dialogue content is generated and output based on the word piece and the corresponding sampling probability in the target list. In this way, by splitting the probability vector of the full vocabulary into several independent data blocks, each data block is processed in parallel by the corresponding processing unit, improving the hardware utilization rate, avoiding idle hardware computing power, and helping to reduce the data granularity of single processing, reducing the memory bandwidth pressure, thereby reducing the memory access delay by quickly reading and writing data, solving the problem of resource waste and calculation redundancy caused by full sorting of the probability distribution of the complete vocabulary, and then merging and sorting the target sets of multiple data blocks to generate a globally ordered target list, effectively reducing the computational complexity and improving the generation efficiency of the dialogue. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0013] Figure 1 A dialogue content generation flowchart is provided for the embodiments of the present application.

[0014] Figure 2 A dialogue content generation system architecture diagram is provided for the embodiments of the present application.

[0015] Figure 3 A specific dialogue content generation flowchart is provided for the embodiments of the present application.

[0016] Figure 4 A hierarchical architecture diagram of merge sorting is provided for the embodiments of the present application.

[0017] Figure 5A hierarchical merging architecture diagram provided by an embodiment of the present application;

[0018] Figure 6 A dialogue content generation device structure schematic diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0020] It should be noted that, in the description of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0021] With the expansion of large models to mobile terminals, sampling optimization is crucial in dialogue model reasoning acceleration technology in edge computing scenarios, but traditional sampling requires 5-10 ms on a 100K vocabulary table, which seriously restricts the reasoning throughput, and currently requires full sorting of the complete vocabulary probability distribution, which is prone to resource waste and computational redundancy. The present application effectively improves hardware utilization, reduces the data granularity of single processing, and improves dialogue generation efficiency by dividing the target vector output by the activation function into blocks, then parallel filtering the word units in each data block to construct the corresponding target set, and then performing merge sorting to obtain the target list to generate the corresponding target dialogue content.

[0022] In order to enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0023] Next, the present embodiment will be described in detail in combination with the execution flow of the dialogue content generation method, as shown in Figure 1 The embodiment of the present application provides a dialogue content generation method, which comprises:

[0024] Step S11, obtaining a target vector output by a preset activation function of a dialogue model; the target vector is a vector constructed based on the sampling probability of each word unit in the preset vocabulary table of the dialogue model; the dialogue model is a dialogue model created based on a pre-training model.

[0025] In this embodiment, first, the target vector output by the preset activation function of the dialogue model is obtained. The dialogue model is a dialogue model created based on a pre-trained model, and the target vector is a vector constructed based on the sampling probability of each word in the preset vocabulary of the dialogue model. It can be understood that pre-training is a strategy for training a deep learning model, the core of which is to preliminarily train the model using a large-scale data set, so that the model learns general feature representation. The pre-trained model in this embodiment can be a pre-trained language model, such as a large language model (LLM, Large Language Model). Specifically, it involves optimizing the probability sampling algorithm in the inference process of the large language model, and is particularly suitable for scenarios that require high-frequency sampling, such as speculative decoding, real-time dialogue systems, code generation, and dialogue generation based on in-vehicle voice assistants. The pre-trained language model generally refers to designing a language model training task based on a large-scale corpus, training a large-scale neural network algorithm structure to learn and implement, and finally obtaining a large-scale neural network algorithm structure and parameters. By pre-training on a large-scale corpus, the neural language representation model can learn powerful language representation capabilities, and can extract rich syntactic and semantic information from text. The pre-trained language model can provide tokens containing rich semantic information and sentence-level features for downstream tasks, or can directly fine-tune the pre-trained model for downstream tasks, obtaining a downstream-specific model conveniently and quickly. The neural network algorithm structure trained by the pre-trained language model can also be a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), an LSTM (Long Short-Term Memory), etc., or a model constructed with an attention network, such as a transformer, a bert, etc., which are not limited in this application.

[0026] Specifically, as Figure 2The embodiment shown first requires inputting a probability distribution, i.e., the original probability distribution output by the model activation function Softmax. It is usually a vector with a length of the size of the vocabulary (|V|), where each element represents the probability of the corresponding token. It can be understood that the Softmax function is a core activation function, and its core function is to convert the original score (Logits) output by the model into a probability distribution that meets the probability rules. For example, for an input vector with a length of LLM and a vocabulary size |V|, such as 128K, Softmax will calculate each element in the vector, and output a vector with the same length, where each element represents the probability of the corresponding token. It can be understood that the sampling probability can be obtained based on a sampling accelerator, i.e., a PCIe Gen5 (Peripheral Component Interconnect Express 5.0, Peripheral Component Interconnect Express 5.0) high-speed interconnection (200 GB / s).

[0027] In step S12, the target vector is divided into blocks based on the preset block number, and a plurality of data blocks are obtained.

[0028] In this embodiment, the target parameter can be obtained first, and the target parameter is used as the preset block number. The target parameter is a parameter pre-configured according to a preset instruction; or the number of units of a target processing unit is determined, and the preset block number is determined based on the number of units. The target processing unit is a processor running a dialogue model. Specifically, the vocabulary probability vector is split into N data blocks, N=BLOCK_SIZE, or N=GPU (Graphics Processing Unit, Graphics Processing Unit) thread block number, so that each data block independently executes the subsequent stream filtering process. It can be understood that the block size BLOCK_SIZE can be adaptively obtained, and the specific method is as follows:

[0029] ;

[0030] wherein, is the maximum block size limited by hardware, such as GPU thread block limit, and the typical value is 1024; H(p) is the Shannon Entropy of the current probability distribution, and the calculation range is 0 (deterministic distribution) ~ 15 (uniform distribution), is a precision control coefficient for balancing calculation efficiency and load balancing, and the value is 0.1 (high precision) ~ 0.5 (low delay), is the number of hardware stream multi-processors (GPU is SM, and TPU (tensor processing unit) is Core), and for discrete probability distribution , the Shannon Entropy H(P) is defined as:

[0031] ;

[0032] wherein: pi is the probability of the ith event, satisfying and log2 is the logarithm to the base 2, in bits, and is agreed in this embodiment , and

[0033] After the preset number of blocks is determined, the target vector can be blocked based on the preset number of blocks to obtain a plurality of data blocks. That is, the entire probability vector can be divided into a plurality of smaller data blocks in this embodiment, each data block is assigned to a processing unit for parallel processing, for example, a thread block of a GPU is used as a processing unit.

[0034] In step S13, the first word units in each data block are screened out in parallel based on the target sampling probability threshold, and the target set corresponding to each data block is constructed based on the first word units; the target set is used to save the first word units and the sampling probabilities corresponding to each first word unit.

[0035] In this embodiment, the first word units in each data block can be screened out in parallel based on the target sampling probability threshold, and the target set corresponding to each data block can be constructed based on the first word units. The above target set is used to save the first word units and the sampling probabilities corresponding to each first word unit. And based on the previous step, when the first word units in each data block are screened out in parallel based on the target sampling probability threshold, the word unit screening operation of the corresponding data block can be performed in parallel using the target processing unit to screen out the first word units in each data block based on the target sampling probability threshold through the word unit screening operation.

[0036] Specifically, when the first word units in each data block are screened out in parallel based on the target sampling probability threshold, the initial sampling probability threshold can be used as the target sampling probability threshold first, and the first word unit in the data block is saved to the preset candidate set according to the target sampling probability threshold; the initial sampling probability threshold is a floating point value used to represent negative infinity, that is = -float('inf'); Then, based on the sampling probability of the first word unit, each first word unit in the corresponding data block is screened out, and each first word unit is saved to the preset candidate set. After the first word units in the preset candidate set satisfy the preset screening condition, the preset candidate set is used as the target set.

[0037] And based on the sampling probability of the first word element, each first word element in the corresponding data block is screened out, the target sampling probability threshold can be updated based on the sampling probability of the first word element, to obtain an updated target sampling probability threshold, and the next word element of the first word element in the data block is taken as a target word element, whether the sampling probability of the target word element is not less than the updated target sampling probability threshold is judged; if the sampling probability of the target word element is not less than the updated target sampling probability threshold, the target word element is saved to a preset candidate set, and the updated target sampling probability threshold is updated again based on the sampling probability of the target word element to obtain a new updated target sampling probability threshold, and the next word element of the target word element in the data block is taken as a new target word element, and then jumping to the step of judging whether the sampling probability of the target word element is not less than the updated target sampling probability threshold; if the sampling probability of the next word element is less than the updated target sampling probability threshold, the next word element of the target word element in the data block is directly taken as a new target word element, and then jumping to the step of judging whether the sampling probability of the target word element is not less than the updated target sampling probability threshold.

[0038] That is, in the present embodiment, when performing the stream filtering operation on each data block, the dynamic threshold can be used to screen the candidate token. Wherein, the dynamic threshold τ can be realized by real-time adaptive updating to achieve efficient screening. First, the is initialized, that is, =-float('inf') to ensure that the first element unconditionally enters the candidate set, that is, the target set. After that, the dynamic threshold is updated based on the probability of the first element, that is:

[0039] ;

[0040] Wherein, the threshold value of the current time step; is the probability value of the token being processed; is the current candidate set, that is, the target set; is the top k elements in the set S, and only when the probability value , the candidate set is considered to be added.

[0041] It should be noted that in the embodiment, the updating of the tokens in the candidate set can be implemented by maintaining a minimum heap. Specifically, a corresponding binary tree can be generated based on the preset candidate set, and the sampling probability of the first token is taken as the target value of the binary tree. The binary tree is a binary tree constructed based on the minimum heap, and the target value is the top value of the minimum heap. Then, the target token is saved in the preset candidate set, and the target sampling probability threshold is updated again based on the sampling probability of the target token. After that, the target token is saved in the binary tree, the sampling probability of the target token is taken as a new target value, and the cumulative sum of the sampling probabilities of the first tokens in the binary tree is determined. Meanwhile, the global :

[0042] ;

[0043] wherein, is a global dynamic threshold, which is a threshold shared by all processing units, and is used for candidate filtering across threads / cores; is a local dynamic threshold, which is the top value of the candidate set minimum heap of the current processing unit, is an updated global threshold. In the embodiment, a predicted threshold adjustment can be implemented, and the adjusted threshold is:

[0044] ;

[0045] wherein: is a smoothing coefficient, and the empirical value is 0.3-0.7; is the top value of the local candidate set minimum heap at the current time step.

[0046] It can be understood that, as the number of processing elements in the candidate set increases, the minimum value of the heap increases with the addition of larger elements, gradually increases. When rises to a certain value, subsequent elements that are smaller than are filtered out, thereby reducing unnecessary comparisons and heap operations. Finally, converges to the global Kth largest probability value, i.e., the boundary value of TopK. That is, the embodiment can maintain a minimum heap with a size of K for TopK constraint and a cumulative sum for TopP constraint to track the candidate tokens of the current block. When an element is added, the heap and the cumulative sum are updated. If the size of the heap exceeds K, the top value, i.e., the minimum value, is popped out, and the popped value is subtracted to update the cumulative sum. If the cumulative sum exceeds P, the process is terminated. In this way, when the elements in the block are traversed, only those with a probability value greater than or equal to the current threshold The elements are processed and the candidate set and threshold are updated. If the termination condition is met, such as the candidate set size reaching K or the accumulated probability reaching P, the traversal of that block is terminated early. The TopK constraint maintains a min-heap of capacity K, retaining only those with probabilities greater than or equal to... The elements; the TopP constraint is to calculate the cumulative probability S in real time, and terminate the traversal early when S is greater than or equal to P.

[0047] Specifically, in one embodiment, it can be determined whether the number of tokens in the binary tree corresponding to the preset candidate set meets the preset token number condition, and whether the cumulative sum of the binary tree is greater than the preset cumulative sum threshold. When the cumulative sum is greater than the preset cumulative sum threshold, the preset candidate set is directly used as the target set; when the number of tokens meets the preset token number condition, but the cumulative sum is less than the preset cumulative sum threshold, the token corresponding to the top value in the preset candidate set is deleted, and the process jumps to the step of using the next token after the first token in the data block as the target token, until the cumulative sum is not less than the preset cumulative sum threshold, at which point the preset candidate set is used as the target set.

[0048] In another specific embodiment, it can be determined whether the number of words in the binary tree corresponding to the preset candidate set meets the preset word quantity condition, or whether the cumulative sum corresponding to the binary tree is greater than the preset cumulative sum threshold; when the number of words meets the preset word quantity condition, or the cumulative sum is not less than the preset cumulative sum threshold, the preset candidate set is directly used as the target set.

[0049] In other words, when performing streaming probability filtering in this embodiment, a min-heap is first initialized to maintain the current TopK candidates, an accumulator S=0 of type FP32 used to track the cumulative probability of the candidate set, and a dynamic threshold of type FP32 used for dynamic filtering boundaries. =0, at this point, all elements may be added initially in the stream filtering process because =0, and any probability >= 0. It's understandable that, since the probability of any token >= 0, we can... The probability of each token is then adjusted from -float('inf') to 0. If the current probability value is large enough, exceeding the threshold Or, if the cumulative probability of the candidate set does not reach the target P, then... Add to the heap. Specifically, if the heap size is less than K, add directly; otherwise, if... If the value is greater than the top of the heap, replace the top element by popping it from the heap, then pushing the new element onto the heap, subtracting the popped value, and adding the new element's value to update the accumulator S. If the heap size is not yet K, simply add the new element and add its value. Then update... is the current K-th largest value when the heap size reaches K, and check if the accumulator S is greater than or equal to P, and the length of the minimum heap is equal to K, if so, break the loop. After the loop ends, the elements in the heap are the candidate set, at this time the size of this candidate set <= K, and is the current largest number of elements, and the cumulative sum >= P or the cumulative sum does not reach P but the largest K probability has been taken under the premise of early exit of the loop.

[0050] And it should be pointed out that in the block parallel streaming filtering stage in the embodiment, a mixed precision architecture of FP8 (8-bit floating point, 8-bit floating point) pre-screening + FP32 (Single-precision floating-point, 32-bit binary single-precision floating-point) refinement is adopted. First, the FP8 pre-screening realizes low-precision rapid filtering, and the FP32 probability vector is converted into FP8 format (E5M2 or E4M3) in batches, and threshold comparison is performed in the FP8 space. And in the TopP constraint, the cumulative sum S needs to be calculated in real time. The traditional accumulation still has rounding error under FP32, and after ten thousand times of accumulation, the error can reach 10 -3 orders of magnitude, which affects the fairness of sampling and leads to early termination failure or distribution distortion. In the embodiment, the integrated FP32 refinement in the streaming filtering compensates for the cumulative error through Kahan summation:

[0051] ;

[0052] The specific implementation is as follows:

[0053] / / When a new element p i joins the heap:

[0054] float old_min = heap.replace(p i ); / / replace the top of the heap

[0055] / / Kahan incremental update cumulative sum:

[0056] float y = p i -old_min - err; / / incremental value + error compensation

[0057] float t = S + y;

[0058] err = (t - S) - y; / / update error term

[0059] S = t.

[0060] In this way, the embodiment uses FP8 to reduce the data processing amount by 4 times through the mixed precision architecture of FP8 pre-screening and FP32 refinement (Kahan), and reduces the error from 10^ -3 to 10^ -6 through Kahan summation, improves energy efficiency, realizes mixed precision calculation and error control, is suitable for cloud-edge-end full-scene deployment, and supports dynamic voltage and frequency scaling (DVFS) in edge deployment, and enables FP8 quantization mode, and automatically reduces precision when T>80℃. In the optimization of actual engineering edge deployment, DVFS dynamically balances between calculation performance and power consumption / heat, and the FP8 precision reduction mode triggered by temperature can further reduce the calculation load from the algorithm level on the basis of DVFS. When the temperature threshold is triggered, it can automatically switch to a lower precision and lower power consumption working mode. For example, in the edge-cloud collaborative inference scene, the edge end uses stream filtering (FP8 quantization) to process the first token to generate quickly, and the cloud end uses full-precision merge sorting and dynamic vocabulary to refine long sequences. For example, applied to a vehicle-mounted voice assistant, in offline mode, the edge end generates a draft, and after networking, the cloud end optimizes the output, and the end-to-end delay can be reduced from about 220ms to 80ms.

[0061] In step S14, the target list is obtained by performing merge sorting on each target set, and the corresponding target dialogue content is generated and output based on the first word element in the target list and the corresponding sampling probability.

[0062] In the embodiment, the probabilities in each target set, i.e., the candidate set, are sorted by merging, i.e., the target list is obtained, and the corresponding target dialogue content is generated and output based on the first word element in the target list and the corresponding sampling probability. It can be understood that the local candidate set of each block is output in the previous step. Each local candidate set is a candidate token and its probability value selected from the corresponding data block after stream filtering, because only tokens meeting the conditions are retained, the size of these candidate sets is much smaller than the original data block, and they are unsorted or partially sorted, such as a minimum heap structure. Then, the multi-way merge sorting algorithm is used to efficiently merge these candidate sets using the merge sorting API (Application Programming Interface) of the GPU, and the multi-way merge sorting aggregation is realized to combine the local candidate sets from each processing unit into a globally ordered candidate set.

[0063] The above technical solution divides the target vector output by the activation function into blocks based on a preset number of blocks. Then, based on the target sampling probability threshold, the first word in each data block is selected in parallel to construct the target set corresponding to each data block. Afterwards, the target sets are merged and sorted to obtain a target list. Based on the words in the target list and their corresponding sampling probabilities, the corresponding target dialogue content is generated and output. In this way, by splitting the probability vector of the entire vocabulary into several independent data blocks, each data block is processed in parallel by corresponding processing units, improving hardware utilization, avoiding idle hardware computing power, and helping to reduce the data granularity of a single processing operation, reducing memory bandwidth pressure. This solves the problem of resource waste and computational redundancy caused by performing full sorting on the probability distribution of the complete vocabulary, and further reduces computational complexity and improves dialogue generation efficiency by merging and sorting the target sets of multiple data blocks to generate a globally ordered target list. Furthermore, this embodiment can extend the streaming filtering strategy to the text-image-speech joint generation scenario through multimodal joint optimization, which is achieved through a dynamic threshold sharing mechanism and cross-modal probability alignment, establishing a unified probability normalization space, so that the sampling thresholds of different modalities can be compared horizontally.

[0064] As described in the previous embodiment, this application can divide the target vector output by the activation function into blocks, then select the lexical units in each data block in parallel to construct the corresponding target set, and then perform merging and sorting to obtain the target list and generate the corresponding target dialogue content. Next, this embodiment will describe in detail the process of merging and sorting the target set and generating the dialogue. See [link to previous embodiment]. Figure 3 As shown in the figure, this application discloses a specific method for generating dialogue content, including:

[0065] Step S21: Sort the sampling probabilities corresponding to the first word in each target set to obtain the sorted target set.

[0066] In this embodiment, when performing multi-way merge sorting on the candidate sets corresponding to each data block, the sampling probabilities corresponding to the first word in each target set, i.e. the candidate set, can be sorted to obtain the sorted target set.

[0067] Step S22: Construct a priority queue based on the first sampling probability in each sorted target set, and merge and sort each sorted target set based on the priority queue to obtain a target list.

[0068] In this embodiment, a priority queue can be constructed according to the first sampling probability in each sorted target set, and the priority queue is used for merge sorting of each sorted target set to obtain a target list. That is, the min-heap of the general stream filtering output can be directly used for merging in this embodiment, but if the local candidate set is unordered, the local candidate set is sorted first, and then a priority queue (min-heap or max-heap, depending on the sorting direction) is used to merge multiple ordered lists. That is, the current maximum (or minimum) element is taken out of each local candidate set and put into the priority queue, and then the elements in the queue are taken out one by one to form a globally ordered list. Then, according to the TopK or TopP requirement, the global candidate set is truncated: for TopK, the top K maximum probability tokens are taken; for TopP, the cumulative probability is calculated on the global ordered list until the threshold P is reached. Thus, the globally TopK / TopP set after multi-way merge sorting aggregation is obtained, that is, the global candidate token set satisfying the TopK or TopP condition, arranged in descending order of probability. For example, the first element (i.e., the maximum element of the list) is taken out of each local ordered list and put into the priority queue, and then the global maximum element is taken out of the priority queue in a loop and added to the global ordered list, and the next element from the local list to which the element belongs is taken out of the priority queue and added to the priority queue while the list is not empty. When the priority queue is empty and all elements of the local list have been processed, the global ordered list contains all the candidate elements of the blocks and is sorted in descending order of probability. In a specific embodiment, if there are three local lists A: [0.5, 0.3, 0.2], B: [0.4, 0.35, 0.1], and C: [0.6, 0.25, 0.15], the queue is initialized as 0.5 (A), 0.4 (B), and 0.6 (C), the maximum value 0.6 (C) is taken into the global list, and then the next element 0.25 of C is added to the priority queue, i.e., 0.5 (A), 0.4 (B), and 0.25 (C), and 0.5 (A) is taken into the global list. Then the next element 0.3 of A is added to the priority queue, i.e., 0.4 (B), 0.3 (A), and 0.25 (C), and 0.4 (B) is taken into the global list, and finally the completely ordered global list is obtained, i.e., [0.6, 0.5, 0.4, 0.35, 0.3, 0.25, 0.2, 0.15, 0.1].

[0069] Based on the above technical solution, in combination with the block processing process in the previous embodiment, as shown in Figure 4 first, the table probability vector is split into N data blocks, N = BLOCK_SIZE, or N = the number of GPU thread blocks, each block independently performs stream filtering to generate a local candidate set. }. Wherein: the candidate set of each block is a heap, and a cumulative sum, but the local candidate set may have met the TopP condition and terminated early. After each block is processed, the elements in the heap are taken out, sorted in descending order to get a local ordered list. Then, using multi-way merging, these ordered lists are merged into a global ordered list, and the cumulative sum is calculated on the global ordered list until P is reached or the top K is taken. It should be pointed out that TopK and TopP can also be met at the same time: that is, the top K is taken first, and then the cumulative sum is taken to reach P without exceeding K. In fact, the cumulative sum can be recalculated from large to small on the merged global ordered list until P is reached, but at most K tokens are taken. Therefore, only the global ordered list needs to be traversed, and at this time the total length of the global ordered list is at most the sum of the sizes of all local candidate sets, but the size of each local candidate set is not more than K, so the total length is not more than the block number multiplied by K, and the block number is generally fixed, such as the number of thread blocks of GPU, so the total length of the global ordered list will not be very large. As shown in Figure 4 When performing stream filtering in parallel, threshold synchronization between threads can be achieved by using CUDA (Compute Unified Device Architecture) Shuffle instructions at the Warp (Warp Cooperative Groups) level (speedup ratio 3.2x), and by using SIMD (Single Instruction, Multiple Data) vectorization, using AVX-512 (Advanced Vector Extensions-512, a processor instruction set) mask loading to reduce conditional branching (speedup ratio 2.1x). Specifically, in stream filtering, each thread needs to know the current dynamic threshold If each thread reads and writes the global memory or shared memory , serious lock contention and memory delay will occur, becoming a performance bottleneck, so in stream filtering, the maximum value (or minimum value) of the 32 probability values processed by the current Warp needs to be quickly found and used to update the threshold, and the embodiment shown in Figure 4 , Figure 5 uses CUDA Shuffle instructions (__shfl_sync) to allow threads within the same Warp (32 threads) to directly read the register values of other threads without going through shared memory or global memory, achieving extremely low-latency inter-thread communication at the register level. And it can be understood that when performing stream filtering on a CPU (Central Processing Unit), the most time-consuming part is the conditional branching (if(p i >= ) ), at present CPU relies on branch prediction, but when the probability distribution is irregular, the prediction failure rate is high, resulting in pipeline stall (Pipeline Stall), and the embodiment AVX-512 introduces a mask register (k0-k7), which allows almost all instructions to become conditional execution, which can set the mask with a comparison operation first, and then execute the load and calculation only on the channel where the mask is true, without any branch jump, realizing the hardware optimization of parallel stream filtering.

[0070] And in this embodiment, when sorting is implemented through the merge sort API, on the GPU, the DeviceMergeSort (or the sorting of Thrust) of the CUB library (CUDA Unbound, an open source template library) can be used to sort each local candidate set. At this time, because each candidate set is small, multiple candidate sets can be sorted in parallel, and each thread block sorts a candidate set. The CUB library provides the functions of DeviceSegmentedSort (segmented sorting) or MultiMerge (multi-way merge), or a global priority queue can be used to implement an efficient multi-way merge on the GPU. And a hardware-optimized merge sort API (such as CUDA's merge_sort and TPU's XLA:Merge (XLA, Accelerated Linear Algebra)) can also be called to perform parallel merging on the local candidate set. Among them, XLA:Merge requires the input data to be stored continuously in memory, so that the DMA (Direct Memory Access) engine of TPU can prefetch large blocks of data, reducing the number of memory accesses. Specifically, first, a locally ordered candidate set can be generated, and after each TPU core processes a data block, a locally ordered list (in descending order of probability) is output, and then XLA:Merge is called for multi-way merging. The merging operation is mapped to optimized TPU instructions by the XLA compiler, and then the data layout is explicitly planned to optimize the memory layout, ensure continuous memory access, and realize hardware acceleration API. The complexity of the merging operation is reduced from O (|V|log|V|) to O (N*K*logK) (N is the number of blocks). Finally, after obtaining the globally ordered candidate set in descending order of probability, the cumulative probability, i.e., the prefix sum, is calculated, and then according to the TopP value set by the user, the prefix whose cumulative probability reaches P is truncated, which may be smaller than the entire candidate set, and then polynomial sampling is performed on this prefix subset. In this embodiment, under GPU-TPU heterogeneous computing, image tokens (such as Latent Space of Stable Diffusion) and text tokens share the merge sort pipeline, and the generation speed is improved by 20% through hardware cooperative scheduling.

[0071] And the embodiment can be achieved by staggered storage, that is, buf[threadIdx.x+(bank_id%4)*16], to avoid memory Bank conflict, wherein buf is a shared memory array used to store temporary data; threadIdx.x is a thread index of a CUDA thread (0 to blockDim.x-1); bank_id is a number of a memory Bank, and the calculation method is bank_id=threadIdx.x%32; (bank_id%4) is a modulo of the Bank number by 4, which disperses the conflict into four groups of Banks; and 16 represents a step offset, and the data is stored at intervals of 16 times. It can be understood that each thread block (Block) processes a local candidate set in the embodiment, for example, a minimum heap with a size of K. Before merging, the heap needs to be converted into an ordered array, which usually needs to be sorted or adjusted in the shared memory. For example: a thread block has 128 threads, and a shared memory array buf

[128] is used to store the probability values of the candidate set. If each thread simply stores data by using its own thread ID (threadIdx.x) as an index, the number of Banks is 2 to the power of 5 at this time, so the address [4:0] bit is observed, and the number of Banks is determined by the low bit of the memory address. The addresses of buf[0], buf[1],..., buf

[31] are distributed on 32 different Banks, at this time, threads 0-31 access different Banks at the same time, and there is no conflict, and the efficiency is the highest. However, when the threads read data for merging and comparison, the thread 0 usually needs to access buf[0], buf[1], buf[2]... to perform a serial merging operation, at this time, the addresses accessed by adjacent threads are no longer staggered, but very close, and it is easy to access the same Bank, resulting in serious Bank conflict. Therefore, in order to solve this problem, the following formula is used for staggered storage in the embodiment, and the data is artificially dispersed to different Banks:

[0072] index=threadIdx.x+(bank_id % 4)*16;

[0073] The calculation process is as follows: bank_id = threadIdx.x % 32, calculate the Bank ID of each thread; (bank_id % 4): divide the 32 Banks into 4 groups (0-3, 4-7,..., 28-31), so that the threads with bank_id of 0, 4, 8,..., 28 belong to the 0th group, and so on. This can reduce the conflict granularity; 16 is a step (Stride), by multiplying a large enough number 16, it can be ensured that the memory addresses accessed by the threads in the same group are large enough apart, thereby mapped to different Banks; the calculation process of the above index can ensure the uniqueness of the address by adding the original index to threadIdx.x. In this way, in the multi-way merge sorting aggregation step, by using the memory architecture characteristics of the GPU, the memory access mode that would cause serious serialization is converted into a highly parallel mode, and the multi-way merge optimization is realized.

[0074] In step S23, the corresponding target dialogue content is generated and output based on the first word unit in the target list and the corresponding sampling probability.

[0075] In this embodiment, the final sampling can be performed based on the target list, that is, multinomial sampling is performed on the global TopK / TopP set. First, because after the TopK / TopP operation, the probability sum is not necessarily 1, the token probability in the set needs to be re-normalized, and then according to the normalized probability distribution, a token is sampled as the output using an algorithm such as Roulette Wheel Selection. In this way, the entire process avoids full sorting and calculation through parallel processing and streaming filtering, greatly reducing the computational complexity, while there is no approximation error, maintaining the accuracy of the sampling result. That is, the embodiment can first screen out a preset number of first word elements in the target list as dialogue word elements; or, screen out a plurality of first word elements in the target list that meet a preset cumulative sum condition as dialogue word elements; the preset cumulative sum condition is that the cumulative sum of the sampling probabilities of the dialogue word elements is not less than a preset cumulative sum threshold; and then generate corresponding target dialogue content based on each dialogue word element. Specifically, first, sort the dialogue word elements based on the sampling probabilities corresponding to the dialogue word elements to obtain sorted dialogue word elements, and normalize the sorted dialogue word elements to obtain normalized word elements, and generate corresponding target dialogue content according to the normalized word elements. That is, first, construct a corresponding probability interval according to the sampling probabilities of each normalized word element, and generate a random number, determine the corresponding normalized word element as the second word element based on the random number and the probability interval, then jump back to the step of generating a random number, determine the corresponding normalized word element as the second word element based on the random number and the probability interval, until the number of second word elements meets the preset word element number condition, and then generate corresponding target dialogue content according to each second word element.

[0076] For example, all elements with cumulative probability first≥P can be intercepted from the global candidate set G to form a TopP candidate subset S, and then the probabilities of the subset S are re-normalized to ensure sampling fairness. Specifically, the probability of each element in the subset S can be divided by the total probability of the subset S to obtain a normalized probability, so that the probability of each element being selected during sampling is consistent with its relative weight in the global distribution, which helps to achieve zero precision loss. Finally, multinomial sampling is performed on the normalized subset S to generate tokens. For example, when using the Roulette Wheel Selection algorithm, first construct a probability interval according to the normalized probability, generate a random number between 0 and 1, and determine the interval corresponding to the random number to select the token in the interval. In this way, the higher the probability of an element, the wider the interval length, which is more consistent with the logic of selecting high-probability tokens, while ensuring the diversity of model dialogue generation.

[0077] By the technical solution, the whole vocabulary |V|log|V| needs to be sorted originally, and each block of the embodiment sorts an array of size K, that is, each block sorts K elements, the complexity is KlogK, there are N blocks, the total complexity is O(N*K*logK), and the complexity of multi-way merging is O(N*K*logN). Since N and K are much smaller than |V|, the total amount of calculation is greatly reduced, for example, for TopK, from O(|V|log|V|) to O(N*K*logK); for TopP, the average traversal times are reduced to O(P-1) (P is a threshold), which is much lower than O(|V|). At the same time, it is friendly to hardware acceleration, stream filtering has no data dependency, is suitable for SIMT (Single Instruction, Multiple Threads) parallel, reduces the amount of calculation, and calls the vendor optimization API through merging sorting, maximizes the memory throughput, and accelerates merging. At the same time, the output distribution is completely consistent with the traditional method, there is no approximate calculation, and zero precision loss is realized. In the scene of speculative inference gain m>10, the sampling speed improvement directly translates into end-to-end inference acceleration. Through LLM inference framework integration verification (vLLM+TensorRT-LLM), it can be seamlessly deployed to cloud, edge and end full scene, for example, in cloud inference service, the sampling delay of 100K vocabulary is <1ms (thousands of concurrent); the inference speed of mobile end real-time dialogue end-side LLM (such as GPT-3.5Turbo) is improved by 8.7x; multi-modal generation realizes end-to-end acceleration of joint text-image sampling. And the web page ranking can be replaced by LLM stream sampling, so that the result generation speed is reduced from about 300ms to 30ms, and personalized real-time adjustment is supported. In the test environment of |V|=128K, m=5 speculative inference, Batch=32, TopK (K=50) is accelerated from 8.2ms to 0.9ms, with a speedup ratio of 9.1x; TopP (P=0.95) is accelerated from 7.8ms to 0.7ms, with a speedup ratio of 11.1x; TopK, TopP joint sampling (K=50, P=0.9) is accelerated from 9.1ms to 1.1ms, with a speedup ratio of 8.3x. End-to-end delay compression and edge scene lightweight are realized, which provides a basic support for real-time deployment of hundred-billion-level models.

[0078] As Figure 6 shown, the embodiment of the present application also provides a dialogue content generation device, comprising:

[0079] The vector acquisition module 11 is configured to acquire a target vector output by a preset activation function of a dialogue model; the target vector is a vector constructed based on sampling probabilities of each vocabulary in a preset vocabulary of the dialogue model; and the dialogue model is a dialogue model created based on a pre-training model.

[0080] The vector block module 12 is configured to block the target vector based on a preset block number to obtain a plurality of data blocks.

[0081] The word screening module 13 is configured to screen the first word in each data block in parallel based on a target sampling probability threshold, and construct a target set corresponding to each data block based on the first word. The target set is configured to save the first word and the sampling probability corresponding to each first word.

[0082] The dialogue generation module 14 is configured to perform merge sort on each target set to obtain a target list, and generate and output corresponding target dialogue content based on the first word and the corresponding sampling probability in the target list.

[0083] The above description of the features of the embodiment corresponding to the dialogue content generation device can refer to the related description of the embodiment corresponding to the dialogue content generation method, which will not be repeated here.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment.

[0085] The embodiment of the present application also provides an electronic device, including a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above dialogue content generation method embodiments.

[0086] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any one of the above dialogue content generation method embodiments when running.

[0087] In an exemplary embodiment, the above computer readable storage medium can include but is not limited to: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, magnetic disk or optical disk and various computer program storage media.

[0088] The embodiment of the present application also provides a computer program product, the above computer program product includes a computer program, and the computer program is executed by the processor to realize the steps in any one of the above dialogue content generation method embodiments.

[0089] The embodiment of the present application further provides another computer program product, comprising a nonvolatile computer readable storage medium, the nonvolatile computer readable storage medium stores a computer program, the computer program is executed by a processor to implement the steps in any of the above dialogue content generation method embodiments.

[0090] Those skilled in the art will further appreciate that the functions of the examples described herein, including any related steps of a method, can be implemented using electronic hardware, computer software, or any combination thereof. To clearly illustrate this interchangeability of hardware and software, various examples have been described herein in terms of their functionality, which has been described generally and symbolically in flow charts. Having thus described the functionality of the examples, a person of ordinary skill in the art will appreciate that these and / or other examples can be implemented by a variety of hardware and software configurations, and that the described examples are not limited to any particular type of hardware or software. Software implementations of the described examples can be stored in any computer readable medium, which can be any available media that can be accessed by a computer. The computer readable medium can include, but is not limited to, RAM, ROM, EEPROM, solid state drives, FLASH memory, phase-change memory, optical storage, magnetic storage devices, other storage devices, and / or any suitable combination thereof.

[0091] The above has carried out the detailed introduction to the dialogue content generation method and the electronic equipment provided by the present application. The principle and implementation mode of the present application are described by applying specific examples in the present article. The above embodiment description is only applicable to help understand the method of the present application and its core idea. It should be pointed out that, for the ordinary skilled in the art, under the premise of not departing from the principle of the present application, the present application can be improved and modified in several ways. These improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A method for generating dialogue content, characterized in that, include: Obtain the target vector output by the preset activation function of the dialogue model; The target vector is a vector constructed based on the sampling probabilities of each word in the preset vocabulary of the dialogue model; The dialogue model is a dialogue model created based on a pre-trained model; The target vector is divided into multiple data blocks based on a preset number of blocks. The first word element in each data block is selected in parallel based on the target sampling probability threshold, and the target set corresponding to each data block is constructed based on the first word element. The target set is used to store the first word element and the sampling probability corresponding to each first word element; The target sets are merged and sorted to obtain a target list. Based on the first word in the target list and the corresponding sampling probability, the corresponding target dialogue content is generated and output.

2. The dialogue content generation method according to claim 1, characterized in that, Before dividing the target vector into multiple data blocks based on a preset number of blocks, the method further includes: Obtain the target parameters and use the target parameters as the preset number of blocks; the target parameters are parameters pre-configured according to preset instructions; Alternatively, determine the number of units in the target processing unit, and determine the preset number of blocks based on the number of units; the target processing unit is a processor that runs the dialogue model; Accordingly, the parallel filtering of the first word element in each of the data blocks based on the target sampling probability threshold includes: The target processing unit performs lexical filtering operations on the corresponding data blocks in parallel to filter out the first lexical in each data block based on the target sampling probability threshold.

3. The dialogue content generation method according to claim 1, characterized in that, The step of parallelly filtering the first word element in each data block based on the target sampling probability threshold, and constructing the target set corresponding to each data block based on the first word element, includes: The initial sampling probability threshold is used as the target sampling probability threshold, and the first word in the data block is saved as the first word in the preset candidate set according to the target sampling probability threshold; the initial sampling probability threshold is a floating-point value used to represent negative infinity; Based on the sampling probability of the first word, each of the first words in the corresponding data block is selected, and each of the first words is saved to the preset candidate set; Once the first word in the preset candidate set meets the preset filtering conditions, the preset candidate set is used as the target set.

4. The dialogue content generation method according to claim 3, characterized in that, The step of filtering out each first word in the corresponding data block based on the sampling probability of the first word includes: The target sampling probability threshold is updated based on the sampling probability of the first word to obtain the updated target sampling probability threshold; The next word after the first word in the data block is taken as the target word, and it is determined whether the sampling probability of the target word is not less than the updated target sampling probability threshold. If the sampling probability of the target word is not less than the updated target sampling probability threshold, then the target word is saved to the preset candidate set, and the updated target sampling probability threshold is updated again based on the sampling probability of the target word to obtain a new updated target sampling probability threshold. The next word of the target word in the data block is taken as the new target word, and then the process jumps to the step of determining whether the sampling probability of the target word is not less than the updated target sampling probability threshold. If the sampling probability of the next word is less than the updated target sampling probability threshold, then the next word of the target word in the data block is directly taken as the new target word, and then the process jumps to the step of determining whether the sampling probability of the target word is not less than the updated target sampling probability threshold.

5. The dialogue content generation method according to claim 4, characterized in that, The step of saving the first word in the data block as the first word in the preset candidate set according to the target sampling probability threshold includes: A corresponding binary tree is generated based on the preset candidate set, and the sampling probability of the first word is used as the target value of the binary tree; the binary tree is a binary tree constructed based on a min-heap, and the target value is the top value of the min-heap; Accordingly, after saving the target lexical unit to the preset candidate set and updating the updated target sampling probability threshold based on the sampling probability of the target lexical unit, the method further includes: The target word is saved into the binary tree, and the sampling probability of the target word is used as the new target value. The cumulative sum of the sampling probabilities corresponding to each first word in the binary tree is determined.

6. The dialogue content generation method according to claim 5, characterized in that, After the first word element in the preset candidate set satisfies the preset filtering conditions, the preset candidate set is used as the target set, including: Determine whether the number of lexical units in the binary tree corresponding to the preset candidate set meets the preset lexical unit number condition, and determine whether the cumulative sum corresponding to the binary tree is greater than the preset cumulative sum threshold; When the cumulative sum is greater than the preset cumulative sum threshold, the preset candidate set is directly used as the target set; When the number of lexical elements meets the preset lexical element quantity condition and the cumulative sum is less than the preset cumulative sum threshold, delete the lexical element corresponding to the top value of the heap in the preset candidate set, and jump to the step of taking the next lexical element of the first lexical element in the data block as the target lexical element, until the cumulative sum is not less than the preset cumulative sum threshold, and then take the preset candidate set as the target set. Alternatively, determine whether the number of lexical units in the binary tree corresponding to the preset candidate set meets the preset lexical unit number condition, or determine whether the cumulative sum corresponding to the binary tree is greater than the preset cumulative sum threshold; When the number of lexical units meets the preset lexical unit number condition, or when the cumulative sum is not less than the preset cumulative sum threshold, the preset candidate set is directly used as the target set.

7. The dialogue content generation method according to any one of claims 1 to 6, characterized in that, The process of merging and sorting the target sets to obtain a target list includes: The sampling probabilities corresponding to the first word in each of the target sets are sorted to obtain sorted target sets; A priority queue is constructed based on the first sampling probability in each sorted target set, and the sorted target sets are merged and sorted based on the priority queue to obtain the target list.

8. The dialogue content generation method according to claim 7, characterized in that, The step of generating and outputting corresponding target dialogue content based on the first word element in the target list and the corresponding sampling probability includes: A preset number of the first word elements in the target list are selected as dialogue word elements; Alternatively, select a number of the first word elements in the target list that meet the preset cumulative sum condition as the dialogue word elements; the preset cumulative sum condition is that the cumulative sum of the sampling probabilities of the dialogue word elements is not less than the preset cumulative sum threshold. The target dialogue content is generated based on each of the aforementioned dialogue lexical units.

9. The dialogue content generation method according to claim 8, characterized in that, The step of generating the corresponding target dialogue content based on each of the dialogue lexical units includes: The dialogue words are sorted based on the sampling probability corresponding to each of the dialogue words to obtain sorted dialogue words. The sorted dialogue lexical units are normalized to obtain normalized lexical units, and a corresponding probability interval is constructed based on the sampling probability of each normalized lexical unit. Generate random numbers, and determine the corresponding normalized lexical as the second lexical based on the random numbers and the probability interval; The process then jumps back to the step of generating random numbers and determining the corresponding normalized lexical units as second lexical units based on the random numbers and the probability interval, until the number of second lexical units meets the preset lexical unit quantity condition, and then generates the corresponding target dialogue content based on each second lexical unit.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the dialogue content generation method as described in any one of claims 1 to 9 when executing the computer program.

Images