Data screening, large language model output acceleration method, device, medium and product

Abstract

The application discloses a data screening, large language model output acceleration method, device, medium and product. The data screening method comprises the following steps: selecting a thread bundle and determining a local comparison value of the thread bundle; triggering each thread bundle to execute an operation of comparing the numerical value between each current probability value and the baseline value of itself, and updating the key value of itself using each current probability value greater than the baseline value of itself; comparing the screening threshold of a target screening algorithm with the key value of each thread bundle; if the screening threshold and the key value of each thread bundle do not match, updating the target set and the current probability value set according to the identified first thread bundle and second thread bundle; and if the screening threshold and the key value of the third thread bundle match, updating the target set according to the third thread bundle as a screening result. The technical scheme of the embodiment of the application effectively reduces the calculation complexity of the data screening algorithm in a soft and hard cooperative manner, and maximally utilizes the parallel computing hardware in the artificial intelligence acceleration computing chip.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence (AI) technology, and in particular to a data filtering and large language model output acceleration method, device, medium and product. Background Technology

[0002] In natural language tasks, we typically use a pre-trained large language model to generate output text (such as an answer) from a given input text (such as a question). To generate the output text, we need to let the model progressively predict each word (also called a token) of the desired output until the output termination condition is reached.

[0003] In existing technologies, large language models provide the probability distribution of all candidate words at each prediction step, representing their prediction of the next lexical unit. Then, using a pre-defined data filtering algorithm, such as TopK or TopP, multiple candidate words that meet the filtering criteria are selected from all candidate words. Finally, one of these selected candidate words is chosen as the output lexical unit through probabilistic random selection.

[0004] In the process of realizing this invention, the inventors discovered that regardless of whether the TopK or TopP algorithm is used in data filtering, it is necessary to sort all candidate words. The computational complexity of the filtering algorithm is high, and the entire data filtering process takes a long time to implement, which in turn makes the waiting time for the text output process of the large language model longer. Summary of the Invention

[0005] This invention provides a data filtering method, device, medium, and product for accelerating the output of large language models, which efficiently reduces the implementation complexity of data filtering algorithms through hardware and software collaboration, thereby effectively accelerating the output of large language models.

[0006] According to one aspect of the present invention, a data filtering method is provided, the method comprising:

[0007] In response to the request to use a target screening algorithm to filter each data according to the probability value of the data, multiple thread bundles are selected in the artificial intelligence acceleration computing chip according to the single comparison bit width, and the local comparison value corresponding to each thread bundle is determined.

[0008] The current probability value set is initialized based on the probability values ​​of each data point, and the baseline value of each thread bundle in the current iteration is set based on each local comparison value. The current probability value set contains at least one current probability value, and the data bit width of each baseline value is the same as that of each current probability value.

[0009] The parallel execution of each thread bundle involves comparing the value of each current probability value in the current probability value set with its own baseline value based on its own data parallelism, and using the current probability values ​​that are greater than its own baseline value to update its own key value.

[0010] After the operation is completed, the filtering threshold of the target filtering algorithm is compared with the key value of each thread bundle;

[0011] If the filtering threshold does not match the key value of each thread bundle, then identify the first thread bundle corresponding to the minimum upper bound key value of the filtering threshold and the second thread bundle corresponding to the maximum lower bound key value of the filtering threshold.

[0012] Based on the first and second thread bundles, after identifying the target data whose probability values ​​meet the filtering conditions and adding them to the target set and determining the new current probability value set, the next iteration round is started as the new current iteration round, and the operation of setting the baseline value of each thread bundle in the current iteration round is returned to execute based on each local comparison value.

[0013] If the filtering threshold matches the key value of the third thread bundle, the target data whose probability value meets the filtering conditions is identified by the third thread bundle and added to the target set, and the target set is used as the filtering result.

[0014] According to another aspect of the present invention, a method for accelerating the output of a large language model is also provided, the method comprising:

[0015] Obtain the normalized probability values ​​of all candidate words obtained from the current inference of the large language model;

[0016] According to the target filtering algorithm pre-set in the large language model, the data filtering method as described in any one of the embodiments of the present invention is used to filter out the target set that conforms to the target filtering algorithm from each candidate word;

[0017] After resetting the normalized probability values ​​of each candidate word in the target set, target candidate words are sampled in the target set according to the preset sampling rules, and the target candidate words are used as the output of the model in the current round.

[0018] According to another aspect of the present invention, a data filtering apparatus is also provided, the apparatus comprising:

[0019] The thread bundle parameter determination module is used to respond to the request to filter each data according to the probability value of the data using the target screening algorithm, select multiple thread bundles in the artificial intelligence acceleration computing chip according to the single comparison bit width, and determine the local comparison value corresponding to each thread bundle.

[0020] The baseline value determination module is used to initialize the current probability value set according to the probability values ​​of each data point, and to set the baseline value of each thread bundle in the current iteration round according to each local comparison value. The current probability value set contains at least one current probability value, and the data bit width of each baseline value is the same as that of each current probability value.

[0021] The thread bundle parallel execution module is used to trigger each thread bundle to perform operations in parallel, based on its own data parallelism, comparing the value of each current probability value in the current probability value set with its own baseline value, and using the current probability values ​​that are greater than its own baseline value to update its own key value.

[0022] The key-value comparison module is used to compare the filtering threshold of the target filtering algorithm with the key values ​​of each thread bundle after the operation is completed.

[0023] The adjacent thread bundle identification module is used to identify the first thread bundle corresponding to the minimum upper bound key value of the filtering threshold and the second thread bundle corresponding to the maximum lower bound key value of the filtering threshold if the filtering threshold does not match the key value of each thread bundle.

[0024] The repeated iteration module is used to identify each target data whose probability value meets the screening conditions based on the first thread bundle and the second thread bundle, add them to the target set, determine the new current probability value set, start the next iteration round as the new current iteration round, and return to execute the operation of setting the baseline value of each thread bundle in the current iteration round based on each local comparison value.

[0025] The end-of-iteration module is used to add target data whose probability values ​​meet the filtering conditions to the target set if the filtering threshold matches the key value of the third thread bundle, and use the target set as the filtering result.

[0026] According to another aspect of the present invention, a large language model output acceleration device is also provided, the device comprising:

[0027] The filtering information acquisition module is used to obtain the normalized probability values ​​of all candidate words obtained by the current inference of the large language model;

[0028] The target set filtering module is used to filter out the target set that conforms to the target filtering algorithm from each candidate word according to the target filtering algorithm preset in the large language model and the data filtering method described in any one of the embodiments of the present invention.

[0029] The target candidate word output module is used to reset the normalized probability values ​​of each candidate word in the target set, sample and obtain target candidate words in the target set according to the preset sampling rules, and output the target candidate words as the model's current round output.

[0030] According to another aspect of the present invention, an electronic device is also provided, the electronic device comprising:

[0031] At least one processor; and

[0032] A memory communicatively connected to the at least one processor; wherein,

[0033] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data filtering method according to any embodiment of the present invention, or to perform the large language model output acceleration method according to any embodiment of the present invention.

[0034] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the data filtering method described in any embodiment of the present invention, or to implement the large language model output acceleration method described in any embodiment of the present invention.

[0035] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the data filtering method or the large language model output acceleration method as described in any embodiment of the present invention.

[0036] The technical solution of this invention can fully utilize the abundant thread resources in AI-accelerated computing chips to achieve a data filtering process based on a target selection algorithm. By improving the algorithm through parallel comparison of the probability value of each data point with the baseline value of the thread itself using multiple thread bundles, target data meeting the filtering conditions can be quickly filtered out after one or several iterations. This new hardware-software co-processing data filtering method effectively avoids full sorting of a large amount of data to be filtered, thereby effectively reducing the complexity of existing data filtering algorithms and maximizing the utilization of computing resources in AI-accelerated computing chips. Furthermore, by applying the above-mentioned improved data filtering scheme in the filtering process of large language model output lexical units, the output acceleration of large language models can be efficiently achieved, improving the lexical unit generation efficiency of large language models.

[0037] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of a data filtering method provided according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram illustrating how a dedicated data dispersing instruction is used to disperse and store discrete register data in contiguous shared memory, as applicable to an embodiment of the present invention.

[0041] Figure 3 This is a schematic diagram of the word output process executed by a large language model at each output step in the existing technology;

[0042] Figure 4 This is a flowchart of a method for accelerating the output of a large language model provided in an embodiment of the present invention;

[0043] Figure 5 This is a structural diagram of a data filtering device provided in an embodiment of the present invention;

[0044] Figure 6 This is a structural diagram of a large language model output acceleration device provided in an embodiment of the present invention;

[0045] Figure 7 This is a schematic diagram of the structure of an electronic device that implements the data filtering or large language model output acceleration method of the embodiments of the present invention. Detailed Implementation

[0046] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0047] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0048] First, the solutions in the various embodiments of this invention are mainly used for hardware and software co-optimization of the computational logic of specific data filtering algorithms (hereinafter collectively referred to as target filtering algorithms). Specifically, the target filtering algorithm is either the TopK or TopP algorithm.

[0049] The TopK algorithm sorts all the data to be sorted according to a preset sorting value (e.g., the probability value of the data), and then selects the top K data as the data filtering result in descending order of the sorting value. Similarly, the TopP algorithm sorts all the data to be sorted according to a preset sorting value, and then accumulates the sum of the sorting values ​​of each data in descending order of the sorting value. When the accumulated sum is greater than or equal to the P value, the data that participated in the accumulation is used as the data filtering result.

[0050] To facilitate understanding, the improved schemes of the above-mentioned target data filtering algorithm in each embodiment of the present invention will be briefly described first.

[0051] Taking the sorting value as the probability value and the TopK algorithm as the target selection algorithm as an example, to avoid full sorting, the inventors considered the following algorithm to implement the data selection logic: Assume that the probability value of each data point has X possible values. Then, X baseline values ​​can be set. Next, the probability value of each data point can be compared with the above X baseline values, and the number of probability values ​​greater than each baseline value can be counted. Then, the target baseline value matching the K value can be obtained from these counts. Finally, all data points with probability values ​​greater than the target baseline value can be considered as the selection result satisfying the TopK algorithm selection criteria.

[0052] Figure 1This is a flowchart illustrating a data filtering method provided in an embodiment of the present invention. This embodiment is applicable to situations where the computational complexity of a specific target filtering algorithm is optimized through a combination of hardware and software. The method can be executed by a data filtering device, which can be implemented in hardware and / or software and is generally configured in an electronic device equipped with an artificial intelligence acceleration computing chip, such as a cloud server, edge computing node, or various smart terminal devices. Correspondingly, as... Figure 1 As shown, the method may include:

[0053] S110. In response to a request to use a target filtering algorithm to filter each data according to the probability value of the data, select multiple thread bundles in the artificial intelligence acceleration computing chip according to the single comparison bit width, and determine the local comparison value corresponding to each thread bundle.

[0054] In this embodiment, the request to filter data according to the probability value of each data using a target filtering algorithm can be understood as a data filtering request that selects one or more target data that meet the filtering conditions from the above multiple data based on the probability value of each data to be sorted, using the TopK algorithm or TopP algorithm.

[0055] Each data point's probability value has a preset data bit width, such as 32 bits or 64 bits. This data bit width is generally compatible with the hardware parameters of the AI ​​acceleration computing chip selected for data filtering. This AI acceleration computing chip can be dedicated computing hardware designed for AI computing, such as a GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), or NPU (Neural Processing Unit).

[0056] In this embodiment, the inventors propose an implementation method that compares the probability values ​​of each data point with various possible baseline values ​​in parallel, thereby improving the implementation logic of the target selection algorithm. Based on the above improvements, the rich parallel computing resources of AI acceleration chips, i.e., thread bundles, can be fully utilized, and the complex full sorting operation involved in the execution of existing target selection algorithms can be avoided.

[0057] In AI-accelerated computing chips, a warp is a basic execution unit, referring to a logical group consisting of a fixed number of threads (typically 32). Threads within the same warp can execute synchronously, and similarly, different warps within the same AI-accelerated computing chip can execute synchronously. By utilizing the abundant warp resources within the AI-accelerated computing chip, the numerical comparison process between the probability values ​​of various data and baseline values ​​can be efficiently implemented.

[0058] Furthermore, the inventors discovered that if all selectable baseline values ​​were set directly for the full data width of the probability value, the number of baseline values ​​that needed to be set at once would be too large. For example, for a 32-bit probability value, a total of [number missing] baseline values ​​would need to be set. Only a few baseline values ​​can cover all the selectable probability values, thus requiring the use of... Performing parallel comparisons across multiple thread bundles is clearly difficult to implement. Therefore, the inventors considered performing multiple local comparisons on a complete probability value based on a preset single-comparison bit width. Correspondingly, the single-comparison bit width can be understood as the bit width selected for comparison within a probability value during each local comparison.

[0059] In a specific example, when the single comparison bit width is 4 bits, the comparison can be performed from the most significant bit to the least significant bit, comparing the 4 data bits that match the single comparison bit width with the baseline value based on the probability values. To further simplify the comparison operation, the single comparison bit width can be set while ensuring that the values ​​of each probability value remain unchanged; the baseline value can then be updated accordingly each time a comparison is performed.

[0060] Furthermore, for each single alignment bit width, a local alignment value corresponding to each thread bundle can be set. The selectable values ​​of all local alignment values ​​correspond to all selectable values ​​under the single alignment bit width mentioned above in the probability values.

[0061] Accordingly, in an optional implementation of this embodiment, selecting multiple thread bundles in the AI ​​acceleration computing chip according to the single comparison bit width, and determining the local comparison value corresponding to each thread bundle, may include:

[0062] Based on the single alignment bit width B, and the formula M thread bundles are selected in the artificial intelligence acceleration computing chip;

[0063] Based on the selectable values ​​of the binary string that matches the bit width B of a single alignment, determine the local alignment value corresponding to each thread bundle.

[0064] In this optional implementation, the number of thread bundles can be determined according to the single comparison bit width. For example, if the single comparison bit width is 4 bits, there are a total of 16 possible values ​​(i.e., local comparison values) corresponding to the single comparison bit width. Thus, by setting 16 thread bundles, the numerical comparison of each probability value with each baseline value matched with all local comparison values ​​can be completed simultaneously.

[0065] Therefore, by updating the baseline value of each thread bundle based on the single alignment bit width and the local alignment values ​​of different thread bundles at different iteration rounds, the problem can be solved. For example, assuming the probability value of each data point is 8 bits and the single alignment bit width is B=2, then the number of thread bundles M is 4. Therefore, 4 thread bundles can be selected in the AI ​​acceleration computing chip, and the local alignment values ​​of the above 4 thread bundles can be set to 00, 01, 10 and 11 respectively.

[0066] At this point, for a thread bundle A with a local comparison value of 11, during the first numerical comparison (i.e., in the first iteration), the baseline value of thread bundle A can be set to 11000000. This is equivalent to comparing the baseline value with the 7th and 8th bits of each probability value. Then, during the second numerical comparison, the baseline value of thread bundle A can be updated to 11110000. This is equivalent to continuing to compare the baseline value with the 5th and 6th bits of each probability value, and so on.

[0067] S120. Initialize the current probability value set according to the probability values ​​of each data point, wherein the current probability value set contains at least one current probability value.

[0068] In this embodiment, due to the setting of a single comparison bit width, multiple iterations may be required to finally obtain all the data that meet the data filtering conditions. Therefore, the current probability value set can be initialized based on the probability values ​​of all data first, and after one iteration, the current probability values ​​contained in the current probability value set can be updated before starting a new iteration round, until the entire data filtering process is completed.

[0069] S130. Based on the local comparison values, set the baseline value of each thread bundle in the current iteration round, wherein the data bit width of each baseline value is the same as that of each current probability value.

[0070] In an optional implementation of this embodiment, setting the baseline value of each thread bundle in the current iteration based on each local comparison value may include:

[0071] After shifting the local alignment value of each thread bundle to the left by (Si*B) bits, sum it with the baseline value updated in the previous iteration of each thread bundle to obtain the baseline value of each thread bundle in the current i-th iteration.

[0072] The baseline value of each thread bundle is initialized as a binary string of all zeros, and S is the data bit width of each current probability value.

[0073] As mentioned earlier, after setting the local comparison bit width, data that meets the target selection algorithm can be successively filtered out by comparing from the high bit to the low bit. Furthermore, based on this local comparison value, the baseline values ​​of each thread bundle can be shifted and updated step by step, so that each baseline value matches the local data that each probability value needs to be compared in each iteration.

[0074] In a specific example, the data bit width of each current probability value is 8, the bit width of a single comparison is 4, and the local comparison value of a certain thread bundle is 1101. Then, in the first iteration round, the local comparison value of thread bundle B needs to be shifted left by 4 bits (8-1*4) to get 11010000. Then, this 11010000 is summed with the initialized 8-bit all-zero binary string 00000000 to get the baseline value of thread bundle B in the first iteration round, which is 11010000.

[0075] With the above settings, in the first iteration, the numerical relationship between the first four bits of each probability value and the baseline value of thread bundle B can be compared. Then, in the second iteration, the local comparison value of thread bundle B needs to be shifted left by 0 bits (8-2*4) to obtain 00001101. By summing this 00001101 with the baseline value 11010000 updated in the previous iteration, the baseline value for the second iteration is obtained as 11011101. Since the high four bits of the baseline value have already been compared with each probability value, in the second iteration, it is equivalent to comparing the last four bits of each probability value with the new baseline value of thread bundle B.

[0076] S140. Parallel triggering of each thread bundle to execute operations based on its own data parallelism, comparing the value of each current probability value in the current probability value set with its own baseline value, and using the current probability values ​​that are greater than its own baseline value to update its own key value.

[0077] In this embodiment, since the number of thread bundles matches the number of baseline values, the comparison between each current probability value and each baseline value can be synchronously initiated by triggering each thread bundle to execute simultaneously. Furthermore, since each thread bundle contains multiple threads, these multiple threads represent the data parallelism of each thread bundle itself. For example, 16 or 32. This data parallelism is related to the hardware parameters of the AI ​​acceleration computing chip, and this embodiment does not impose any limitations on it.

[0078] Since multiple data tasks can be executed in parallel within a thread bundle, for N current probability values, the baseline value can be compared with each current probability value through multiple internal iterations based on the thread bundle's own data parallelism M. The number of internal iterations is the rounded-up value of (N / M).

[0079] Based on the above embodiments, the operation of triggering each thread bundle to execute in parallel, comparing the numerical values ​​of each current probability value in the current probability value set with its own baseline value according to its own data parallelism, and updating its own key value using the current probability values ​​greater than its own baseline value, may include:

[0080] Each thread bundle is triggered in parallel, and the following operations are performed respectively:

[0081] S1401. From the current probability value set, sequentially obtain multiple current probability values ​​that match its own data parallelism.

[0082] S1402. Compare the values ​​of each current probability value with its own baseline value in parallel, and update its own key value using each current probability value that is greater than its own baseline value.

[0083] S1403. Determine whether the processing of all current probability values ​​in the current probability value set has been completed. If yes, confirm the end of the operation; otherwise, return to execute S1401.

[0084] In this optional implementation, each thread bundle can be used as the current thread bundle. From the perspective of each current thread bundle, the above implementation process can include: the current thread bundle sequentially obtains multiple current probability values ​​that match its own data parallelism from the current probability value set; the current thread bundle compares the numerical values ​​of each current probability value with its own baseline value in parallel; and updates its own key value using the current probability values ​​that are greater than its own baseline value; the current thread bundle returns to execute the operation of sequentially obtaining multiple current probability values ​​that match its own data parallelism from the current probability value set until the processing of all current probability values ​​in the current probability value set is completed.

[0085] In a specific example, if the total number of current probability values ​​in the current probability value set is 128, and the number of parallel data within each thread bundle is 32, then within each thread bundle, the numerical values ​​of the 32 current probability values ​​can be compared with the thread bundle's own baseline value in each internal iteration. Thus, in a current iteration round, each thread bundle needs to go through 4 internal iterations to complete the numerical comparison between its own baseline value and all current probability values ​​in the current probability value set.

[0086] In this embodiment, by implementing multi-threaded bundle parallelism and multi-data parallelism within a single thread bundle, the rich parallel computing units in the AI ​​acceleration computing chip can be fully utilized to quickly compare the numerical magnitude between each current probability value in the current probability value set and each baseline value under each single comparison bit width.

[0087] After each thread bundle completes the comparison between all current probability values ​​and its own baseline value, it can count the current probability values ​​that are greater than its own baseline value, and update the key value (Value) corresponding to each thread bundle based on each current probability value.

[0088] As mentioned above, the technical solutions of the various embodiments of the present invention can be used to optimize the algorithm logic of the TopK algorithm or the TopP algorithm. For different data filtering algorithms, the method of updating the key value corresponding to each thread bundle based on the current probability value also differs.

[0089] In an optional implementation of this embodiment, the target selection algorithm is a TopK algorithm that selects the top K data points according to their probability values ​​in descending order; where K is the selection threshold in the TopK algorithm.

[0090] Accordingly, the current thread bundle updates its own key value using current probability values ​​greater than its own baseline value, which may include:

[0091] The current thread uses the number of current probability values ​​greater than its own baseline value to cumulatively update its own key value.

[0092] Specifically, since the data filtering goal of the TopK algorithm is to select the top K data in descending order of probability value, each thread can, after comparing its own updated baseline value with each current probability value, count the number of current probability values ​​that are greater than its own baseline value, and update its own key value based on this count.

[0093] In a specific example, when the bit width of a single comparison is 2 and the data bit width of each current probability value is 8, in the first iteration, it is necessary to compare the 7th and 8th bits of each current probability value in the current probability value set with the value of each thread bundle's own baseline. Assuming that thread bundle C finds 10 current probability values ​​greater than its own baseline value in the first iteration, then after the first iteration, the key value of thread bundle C can be updated to 10.

[0094] Furthermore, in the second iteration, it is necessary to compare the 5th and 6th probabilities of each current probability value in the current probability value set with the baseline value of each thread bundle. Assuming that in the second iteration, thread bundle C further counts 8 current probability values ​​greater than its baseline value, then after the second iteration, the key value of thread bundle C can be updated to 10 + 8 = 18.

[0095] In another optional implementation of this embodiment, the target screening algorithm is the TopP algorithm, which sums up the probability values ​​of each data point in descending order of probability value until the summed probability value reaches P, where P is the screening threshold in the TopP algorithm.

[0096] Accordingly, the current thread bundle updates its own key value using current probability values ​​greater than its own baseline value, which may include:

[0097] The current thread bundle sums up all current probability values ​​that are greater than its own baseline value, and uses the summation result to update its own key value.

[0098] Specifically, since the data filtering objective of the TopP algorithm is to filter out one or more data whose cumulative probability value reaches P in descending order of probability value, each thread bundle can calculate the sum of the current probability values ​​that are greater than its own baseline value each time it iterates based on a single comparison bit width, and update its own key value based on the sum of the current probability values.

[0099] In another specific example, when the bit width of a single alignment is 2 and the data bit width of each current probability value is 8, in the first iteration, it is necessary to compare the 7th and 8th bits of each current probability value in the current probability value set with the baseline value of each thread bundle. Assuming that in the first iteration, thread bundle C calculates the sum of 10 current probability values ​​greater than its baseline value to be 0.75, then after the first iteration, the key value of thread bundle C can be updated to 0.75. Further, in the second iteration, it is necessary to compare the 5th and 6th bits of each current probability value in the current probability value set with the baseline value of each thread bundle. Assuming that in the second iteration, thread bundle C calculates the sum of 8 current probability values ​​greater than its baseline value to be 0.12, then after the second iteration, the key value of thread bundle C can be updated to 0.75 + 0.12 = 0.87.

[0100] In various embodiments of the present invention, a method is provided in which each thread bundle obtains a new key value by accumulating and updating the key value generated in the previous iteration in different iteration rounds, and after each accumulation and update, the new key value is compared with the same filtering threshold (the aforementioned K value or P value). In fact, there is another optional implementation method, that is, after completing each iteration round, the key value of each thread bundle is cleared to zero. At the same time, the filtering threshold used by each thread bundle in each iteration is updated to the difference obtained by subtracting the statistical value (quantity or cumulative sum) of each probability value in the target set filtered after the previous iteration from the filtering threshold used in the previous iteration. The same comparison effect can be achieved to obtain various data that meet the TopK or TopP filtering requirements.

[0101] S150. After the operation ends, compare the filtering threshold of the target filtering algorithm with the key value of each thread bundle: if the filtering threshold does not match the key value of each thread bundle, execute S160; if the filtering threshold matches the key value of the third thread bundle, execute S180.

[0102] The criteria for determining the end of the above operation are: each thread bundle completes the entire process of comparing each current probability value in the current probability value set with its own baseline value, and updating its own key value using all current probability values ​​in the current probability value set that are greater than its own probability value.

[0103] After completing the above operations, the filtering threshold of the target filtering algorithm can be compared with the key value of each thread bundle. Specifically, for the TopK algorithm, the filtering threshold is the K value, and for the TopP algorithm, the filtering threshold is the P value.

[0104] In this optional implementation, if the filtering threshold does not match the key value of each thread bundle, it indicates that when comparing the current probability value with the baseline value for a single comparison bit width, the data filtering process cannot be truly completed in the current iteration because the precision of the current comparison data is limited by the single comparison bit width. In this case, by executing steps S160-S170, data that definitely meets the data filtering conditions can be filtered and added to the target set. After redetermining the set of current probability values ​​for the next iteration, a new iteration can be started with higher data precision to compare the current probability values ​​with the updated baseline values.

[0105] Furthermore, if the filtering threshold matches the key value of a specific thread bundle (i.e., the third thread bundle), for example, when K is 10 and the key value of thread bundle D is exactly 10, it indicates that the complete data filtering process has been achieved in the current iteration. Similarly, if when P is 0.9, the difference between the key value of thread bundle D and 0.9 is less than or equal to a preset difference threshold, such as 0.05%, it also indicates that the complete data filtering process has been achieved in the current iteration.

[0106] S160, identify and filter the first thread bundle corresponding to the maximum lower bound key value of the filtering threshold and the second thread bundle corresponding to the minimum upper bound key value of the filtering threshold, and execute S170.

[0107] As mentioned earlier, when the filtering threshold does not match the key values ​​of any thread bundle, it means that the filtering threshold must fall between the key values ​​corresponding to the two thread bundles. It is understandable that the smaller the local comparison value of a thread bundle, the larger the key value corresponding to that thread bundle will be after one iteration.

[0108] For example, assuming the filtering threshold is 0.75, by iterating through the key values ​​of each thread bundle, the local alignment values ​​and matching key values ​​for the four thread bundles are as follows: Thread bundle A: local alignment value 00, key value 0.9; Thread bundle B: local alignment value 01, key value 0.8; Thread bundle C: local alignment value 10, key value 0.7; Thread bundle D: local alignment value 11, key value 0.5.

[0109] By comparing the filtering threshold with the key values ​​of each thread bundle, it was found that the filtering threshold falls between the key value of 0.8 corresponding to thread bundle B and the key value of 0.7 corresponding to thread bundle C. Therefore, the key value of 0.8 can be understood as the minimum upper bound key value of the filtering threshold, while 0.7 can be understood as the maximum lower bound key value of the filtering threshold. That is, thread bundle C is the first thread bundle, and thread B is the second thread bundle.

[0110] Specifically, the maximum lower bound key value can be understood as the maximum value among all key values ​​corresponding to each thread bundle that are less than the filtering threshold. Similarly, the minimum upper bound key value can be understood as the minimum value among all key values ​​corresponding to each thread bundle that are greater than the filtering threshold.

[0111] S170. Based on the first thread bundle and the second thread bundle, identify the target data whose probability values ​​meet the filtering conditions, add them to the target set, and determine the new current probability value set. Then, start the next iteration round as the new current iteration round and return to execute S130.

[0112] In this optional implementation, after identifying the first thread bundle and the second thread bundle, it can be determined that: since the key value of the first thread bundle is the value closest to the filtering threshold among all key values ​​less than the filtering threshold, all current probability values ​​greater than the baseline value of the first thread bundle satisfy the filtering condition matching the target filtering algorithm. Therefore, all current probability values ​​greater than the baseline value of the first thread bundle can be added to the target set.

[0113] Meanwhile, since the key value of the second thread bundle is the value closest to the filtering threshold among all key values ​​greater than the filtering threshold, in subsequent iterations, we only need to refine the filtering among all current probability values ​​that are greater than the baseline value of the second thread bundle and less than the baseline value of the first thread bundle. At this point, we can update all current probability values ​​that are greater than the baseline value of the second thread bundle and less than the baseline value of the first thread bundle to a new set of current probability values, start a new iteration, and redetermine the new baseline value of each thread bundle in the new iteration. Finally, the data filtering process ends when the filtering threshold accurately matches the key value of a specific thread bundle.

[0114] That is, in an optional implementation of this embodiment, identifying target data whose probability values ​​satisfy the filtering conditions and adding them to the target set, and determining a new current probability value set, based on the first thread bundle and the second thread bundle, may include:

[0115] Based on the numerical difference between each current probability value obtained from the first thread bundle comparison and the first baseline value, obtain each first-class probability value that is greater than the first baseline value and add it to the target set;

[0116] Based on the numerical difference between each current probability value obtained from the second thread beam comparison and the second baseline value, obtain each second type probability value that is greater than the second baseline value, and use the result obtained after filtering out each first type probability value from each second type probability value as the new set of current probability values.

[0117] S180. Target data whose probability values ​​meet the filtering conditions are identified by the third thread bundle and added to the target set, and the target set is used as the filtering result.

[0118] As mentioned earlier, the complete data filtering process can be completed based on the third thread bundle. At this time, we can first obtain each current probability value in the current probability value set that is greater than the baseline value of the third thread bundle, and add all the data that match the above current probability values ​​to the target set. Together with the original data in the target set, they will serve as the final data filtering result.

[0119] Accordingly, in an optional implementation of this embodiment, adding target data whose probability values ​​meet the filtering conditions based on the third thread beam identification to the target set may include:

[0120] Based on the numerical difference between each current probability value obtained from the comparison with the third baseline value, obtain each third-class probability value that is greater than the third baseline value and add it to the target set.

[0121] The technical solution of this invention can make full use of the abundant thread bundle resources in the AI ​​acceleration computing chip to realize the data filtering process based on the target filtering algorithm. By improving the algorithm by comparing the probability value of each data with the baseline value of the thread bundle itself in parallel with multiple thread bundles, the target data that meet the data filtering conditions can be quickly filtered out after one or several rounds of iteration. This new data filtering method of hardware and software collaboration effectively avoids the full sorting of a large amount of data to be filtered, thereby effectively reducing the complexity of data filtering and maximizing the use of the computing resources in the AI ​​acceleration computing chip.

[0122] After implementing hardware and software improvements based on the target filtering algorithm, the inventors further discovered that the entire data filtering process requires frequent use of registers for auxiliary execution. Therefore, the inventors creatively constructed dedicated instructions to improve register execution efficiency, thereby maximizing the data filtering efficiency. Accordingly, the register operations involved in the entire data filtering process will be further described below in conjunction with the use of registers.

[0123] Accordingly, after setting the baseline value of each thread bundle in the current iteration based on the local comparison values, it may also include:

[0124] Store the baseline value of each thread bundle in the current iteration into the matching baseline register, where the baseline register is either a scalar register or a vector register;

[0125] Furthermore, the current thread bundle sequentially retrieves multiple current probability values ​​from the current probability value set that match its own data parallelism, which may include:

[0126] The current thread bundle sequentially retrieves multiple current probability values ​​from the current probability value set according to its own data parallelism, and stores each retrieved current probability value into the probability value register, which is a vector register.

[0127] With the above settings, the baseline register and probability value register can be directly manipulated using specially constructed instructions during each iteration, so as to achieve the purpose of rapid data comparison.

[0128] Based on the above embodiments, the current thread bundle compares the numerical values ​​of each current probability value with its own baseline value in parallel, and updates its own key value using each current probability value that is greater than its own baseline value. This can include:

[0129] The current thread acquires the target baseline register used to store its own baseline value;

[0130] The current thread calls a dedicated comparison setting instruction that matches the AI ​​acceleration computing chip, compares the numerical values ​​between each current probability value stored in the parallel probability value register and its own baseline value stored in the target baseline register, and stores the comparison result in the target mask register.

[0131] The target mask register is a scalar register, and when the current probability value of the target is greater than its own baseline value, the storage location in the target mask register that matches the current probability value of the target is set to 1; otherwise, the storage location is set to 0.

[0132] The current thread obtains the current probability value of each target that matches each storage location where the target mask register is set to 1, and updates its own key value based on the current probability value of each target.

[0133] In this embodiment, a dedicated comparison setting instruction is constructed based on the hardware characteristics of the artificial intelligence acceleration computing chip. This instruction can quickly compare multiple current probability values ​​with the baseline value of the same thread bundle at once, thereby effectively improving the algorithm execution efficiency.

[0134] Specifically, the instruction format of this dedicated comparison set instruction can be as shown in Table 1 or Table 2.

[0135] Table 1

[0136]

[0137] Table 2

[0138]

[0139] In Tables 1 and 2, the instruction bits [31:0] in the first row indicate that the dedicated comparison set instruction is a 32-bit instruction. The second row refers to the physical meaning of the data filled in different instruction bits. For example, "0x40" filled in the [6:0] position indicates that the first 7 bits of the dedicated comparison set instruction are the instruction identifier of the dedicated comparison set instruction. SRD[6:0] represents a mask register described using 7 bits of data, which is a scalar register. VRS[7:0] represents a probability value register described using 8 bits of data, which is a vector register. VRT[7:0] represents a baseline register described using 8 bits of data, which is a vector register. SRT[6:0] represents a baseline register described using 7 bits of data, which is a scalar register. NA indicates that the data bits are meaningless.

[0140] Furthermore, based on the numerical difference between each current probability value obtained from the first thread beam comparison and the first baseline value, the first type probability values ​​that are greater than the first baseline value are obtained and added to the target set, which may further include:

[0141] After obtaining the first thread bundle, the first target mask register is updated after comparing the numerical values ​​of each current probability value in the current probability value set with its own baseline value.

[0142] Add the first type probability values ​​matched by each storage location set to 1 in the first target mask register to the target set;

[0143] Accordingly, based on the numerical difference between each current probability value obtained from the second thread beam comparison and the second baseline value, obtaining each second type probability value greater than the second baseline value may include:

[0144] After obtaining the second thread bundle, the second target mask register is updated after comparing the numerical values ​​of each current probability value in the current probability value set with its own baseline value.

[0145] Obtain the second type probability values ​​matched by each storage location that is set to 1 in the second target mask register.

[0146] Furthermore, after obtaining each second-class probability value that is greater than the second baseline value, the result obtained by filtering out each first-class probability value from each second-class probability value can be used as a new set of current probability values.

[0147] In this optional embodiment, the operation of filtering out each first-class probability value from each second-class probability value can be achieved by directly performing a simple XOR operation on the data stored in the first target mask register and the second target mask register. Each current probability value set to 1 after the XOR operation can be used to construct a new set of current probability values. This setting further simplifies the data processing logic and improves data filtering efficiency.

[0148] Based on the above embodiments, after filtering out each first-class probability value from each second-class probability value and using the result as the new current probability value set, it may further include:

[0149] By calling a dedicated data dissemination instruction that matches the AI ​​acceleration computing chip, the results obtained after filtering out each first-class probability value from each second-class probability value are continuously stored in shared memory. This allows for linear loading of the new current probability value set in subsequent iterations, reducing data access latency and accelerating the data filtering process.

[0150] In this optional implementation, considering that the probability values ​​corresponding to all the data initially acquired are stored contiguously in shared memory, the current probability value set used in the first iteration can be loaded into the probability value register in a linear manner within a contiguous address space. However, starting from the second iteration, the current probability values ​​used as the new current probability value set have completed a data filtering process, and therefore the data for the next iteration is no longer stored contiguously in shared memory. Consequently, when retrieving the updated current probability values ​​from the current probability value set, linear loading is no longer possible, leading to significant data access latency.

[0151] To address the aforementioned issues, the inventors further developed a dedicated data dissemination instruction tailored to the hardware characteristics of AI-accelerated computing chips. After determining the new current probability values ​​for each iteration as part of the new current probability value set, this dedicated data dissemination instruction can be invoked to distribute the discrete new current probability values ​​from the register into contiguous shared memory. Subsequently, a linear loading process can be used to load the new current probability value set, consisting of these new current probability values, into the probability value register quickly, thereby effectively reducing memory access latency and accelerating the data filtering process. The instruction format of this dedicated data dissemination instruction is shown in Table 3.

[0152] Table 3

[0153]

[0154] Specifically, the aforementioned dedicated data dissemination instruction is also a 32-bit instruction, with the first 7 bits identified as a dedicated data dissemination instruction using 0x30. Specifically, in Figure 2 The diagram illustrates how discrete register data is distributed and stored in contiguous shared memory based on dedicated data distribution instructions.

[0155] like Figure 2 As shown, this dedicated data dissemination instruction is used to locate the matching current probability values ​​(i.e., T0, T3, and T5) in the vector register VRT[7:0] based on the storage location marked as 1 in the scalar register SRS[6:0]. Based on the shared memory address (which can also be represented as the address offset) specified in the scalar register SRD[6:0], the located current probability values ​​T0, T3, and T5 are stored consecutively at the matching shared memory address.

[0156] Furthermore, the shared memory address specified in SRD[6:0] can be incremented in bytes by calculating the total number of current probability values ​​for each data broadcast, and the calculated address increment is added to the shared memory address used in the most recent data broadcast as the starting address for the next broadcast. Here, INC can be 1 or 0; when INC is 1, it indicates that a new shared memory address needs to be incremented and updated.

[0157] Furthermore, after improving the above data filtering method, the improved data filtering algorithm can be directly applied to the filtering stage of candidate words generated by the large language model. Specifically, in... Figure 3 The diagram shows the lexical output flowchart performed by a large language model in the prior art at each output step.

[0158] like Figure 3 As shown, taking the TopP algorithm as the target selection algorithm as an example, the token output process of the large language model is as follows:

[0159] 1. In one output step, the large language model infers the probability value (also known as logits) of each candidate word.

[0160] 2. The probability value of each candidate word is normalized to be between 0 and 1;

[0161] 3. Sort all normalized probability values ​​in descending order;

[0162] 4. Based on the TopP algorithm, the cumulative value of each probability value is counted from largest to smallest. The value is judged according to the screening threshold P, and each probability value that meets the condition of P is marked with a mask value "1".

[0163] 5. Extract the probability values ​​marked as "1" from the mask values ​​as the data filtering results based on the TopP algorithm;

[0164] 6. Renormalize each probability value in the data filtering results to between 0 and 1;

[0165] 7. Obtain the target probability value by sampling from the renormalized probability values ​​through random selection of probability values;

[0166] 8. The large language model uses the target candidate word corresponding to the target probability value as the output token for model output.

[0167] It is understandable that steps 3-5 of the above operations ( Figure 3 The dashed box in the figure represents the TopP algorithm logic. Furthermore, steps 3-5 above can be optimized and improved using the data filtering methods described in various embodiments of this invention. Since the full sorting operation is completely avoided during the above optimization and improvement process, the algorithm complexity of the TopP algorithm can be reduced from O(n*log(n)) to O(n), thereby achieving efficient acceleration of large language model output.

[0168] Correspondingly, Figure 4 This is a flowchart illustrating a method for accelerating the output of a large language model, provided by an embodiment of the present invention. This embodiment is applicable to situations where the process of outputting tokens from a large language model is efficiently accelerated through a combination of hardware and software. This method can be executed by a large language model output acceleration device, which can be implemented in hardware and / or software and is generally configured in electronic devices equipped with AI acceleration computing chips, such as cloud servers, edge computing nodes, or various smart terminal devices. Correspondingly, as... Figure 4 As shown, the method may include:

[0169] S410. Obtain the normalized probability values ​​of all candidate words obtained by the current inference of the large language model.

[0170] In this context, all candidate words obtained by the large language model in the current inference step constitute the total number of candidate words that can be used as output tokens. At the current output step, the large language model ultimately selects one candidate word from these candidate words as the output token for model output.

[0171] Specifically, the large language model infers a matching probability value for each candidate word. By normalizing these probability values, a normalized probability value for each candidate word can be obtained.

[0172] S420. Based on the target filtering algorithm pre-set in the large language model, the data filtering method described in any one of the embodiments of the present invention is used to filter out the target set that conforms to the target filtering algorithm from each candidate word.

[0173] The normalized probability values ​​of each candidate word are equivalent to the probability values ​​of each data used in the data filtering methods of the aforementioned embodiments. Using the normalized probability values ​​of each candidate word as the original input, candidate words that meet the data filtering conditions of the target filtering algorithm can be filtered out from all candidate words and added to the target set.

[0174] S430. After resetting the normalized probability values ​​of each candidate word in the target set, sample the target candidate words in the target set according to the preset sampling rules, and use the target candidate words as the output of the model in the current round.

[0175] Here, the target candidate word is the word unit output by the large language model in the current output step, that is, the token. In addition, the sampling rule can be a random sampling rule.

[0176] The technical solution of this invention applies a data filtering method improved through hardware and software collaboration during the filtering process of large language model output lexical units, which can efficiently accelerate the output of large language models and effectively improve the lexical unit generation efficiency of large language models.

[0177] It needs to be emphasized again that the focus of the embodiments of this invention is to propose a hardware and software architecture for TopK and TopP generation strategies for large language models, which can be integrated into existing deep learning frameworks. The innovation lies in achieving the desired candidate words that meet the TopK or TopP data selection criteria without requiring a full sorting operation, thereby significantly reducing decoding sampling time and greatly improving efficiency compared to existing accelerator cards. Simultaneously, through hardware and software co-design, it features innovations in both algorithm optimization and instruction design: 1. Utilizing hardware multithreading technology, combined with efficient comparison setting and efficient data propagation, it improves the hardware resource utilization of candidate sampling and reduces the latency of candidate word selection.

[0178] More specifically, the main innovations include:

[0179] 1. The embodiments of this invention propose an accelerated hardware and software architecture for TopK or TopP, utilizing multithreading technology. Each thread bundle is fixedly responsible for a specific baseline value, and one round of iteration can determine the B-bit baseline value, where the size of B is determined by the number of thread bundles participating in the calculation. Iterating sequentially from the high bit to the low bit, the final set of TopP or TopK values ​​that meet the requirements is obtained.

[0180] 2. In addition, each embodiment of the present invention designs an efficient comparison set instruction. Each thread bundle can compare several data points of its own data with its own baseline value in parallel at one time, and write the mask result of the corresponding bit bit into the corresponding scalar register bit bit, which saves storage space. Combined with subsequent instructions, it realizes efficient utilization of hardware efficiency.

[0181] 3. The embodiments of the present invention also construct an efficient data dissemination method, which disseminates the discrete address data of the marked specific mask to the continuous storage space in the shared memory, realizes the continuous storage of the filtered data, ensures the linear loading of the specific data in the next iteration, and reduces the latency.

[0182] Figure 5 This is a schematic diagram of a data filtering device provided in an embodiment of the present invention. Figure 5 As shown, the device includes: a thread bundle parameter determination module 510, a baseline value determination module 520, a thread bundle parallel execution module 530, a key-value comparison module 540, an adjacent thread bundle identification module 550, a repeated iteration module 560, and an iteration termination module 570, wherein:

[0183] The thread bundle parameter determination module 510 is used to select multiple thread bundles in the artificial intelligence acceleration computing chip according to the single comparison bit width in response to a request to filter each data according to the probability value of the data using a target filtering algorithm, and to determine the local comparison value corresponding to each thread bundle.

[0184] The baseline value determination module 520 is used to initialize the current probability value set according to the probability value of each data, and set the baseline value of each thread bundle in the current iteration round according to each local comparison value. The current probability value set contains at least one current probability value, and the data bit width of each baseline value is the same as that of each current probability value.

[0185] The thread bundle parallel execution module 530 is used to trigger each thread bundle to perform operations in parallel, based on its own data parallelism, comparing the value of each current probability value in the current probability value set with its own baseline value, and using the current probability values ​​that are greater than its own baseline value to update its own key value.

[0186] The key-value comparison module 540 is used to compare the filtering threshold of the target filtering algorithm with the key values ​​of each thread bundle after the operation is completed.

[0187] The adjacent thread bundle identification module 550 is used to identify the first thread bundle corresponding to the maximum lower bound key value of the filtering threshold and the second thread bundle corresponding to the minimum upper bound key value of the filtering threshold if the filtering threshold does not match the key value of each thread bundle.

[0188] The repeated iteration module 560 is used to identify each target data whose probability value meets the screening conditions based on the first thread bundle and the second thread bundle, add them to the target set, determine the new current probability value set, start the next iteration round as the new current iteration round, and return to execute the operation of setting the baseline value of each thread bundle in the current iteration round based on each local comparison value.

[0189] The end-of-iteration module 570 is used to add target data whose probability values ​​meet the filtering conditions to the target set if the filtering threshold matches the key value of the third thread bundle, and use the target set as the filtering result.

[0190] The technical solution of this invention can make full use of the abundant thread bundle resources in the AI ​​acceleration computing chip to realize the data filtering process based on the target filtering algorithm. By improving the algorithm by comparing the probability value of each data with the baseline value of the thread bundle itself in parallel with multiple thread bundles, the target data that meet the data filtering conditions can be quickly filtered out after one or several rounds of iteration. This new data filtering method of hardware and software collaboration effectively avoids the full sorting of a large amount of data to be filtered, thereby effectively reducing the complexity of data filtering and maximizing the use of the computing resources in the AI ​​acceleration computing chip.

[0191] Based on the above embodiments, the thread bundle parameter determination module 510 can be specifically used for:

[0192] Based on the single alignment bit width B, and the formula M thread bundles are selected in the artificial intelligence acceleration computing chip;

[0193] Based on the selectable values ​​of the binary string that matches the bit width B of a single alignment, determine the local alignment value corresponding to each thread bundle.

[0194] Based on the above embodiments, the baseline value determination module 520 can be specifically used for:

[0195] After shifting the local alignment value of each thread bundle to the left by (Si*B) bits, sum it with the baseline value updated in the previous iteration of each thread bundle to obtain the baseline value of each thread bundle in the current i-th iteration.

[0196] The baseline value of each thread bundle is initialized as a binary string of all zeros, and S is the data bit width of each current probability value.

[0197] Based on the above embodiments, the iterative repetition module 560 can be specifically used for:

[0198] Based on the numerical difference between each current probability value obtained from the first thread bundle comparison and the first baseline value, obtain each first-class probability value that is greater than the first baseline value and add it to the target set;

[0199] Based on the numerical difference between each current probability value obtained from the second thread bundle comparison and the second baseline value, obtain each second type probability value that is greater than the second baseline value, and use the result after filtering out each first type probability value from each second type probability value as a new set of current probability values;

[0200] Accordingly, the end-of-iteration module 570 can be specifically used for:

[0201] Based on the numerical difference between each current probability value obtained from the comparison with the third baseline value, obtain each third-class probability value that is greater than the third baseline value and add it to the target set.

[0202] Based on the above embodiments, the thread bundle parallel execution module 530 can be specifically used for:

[0203] Each thread bundle is triggered in parallel, and the following operations are performed respectively:

[0204] From the current set of probability values, sequentially obtain multiple current probability values ​​that match its own data parallelism.

[0205] In parallel, compare each current probability value with its own baseline value, and update its own key value using each current probability value that is greater than its own baseline value;

[0206] Return to the previous state and execute the operation of sequentially retrieving multiple current probability values ​​from the current probability value set that match its own data parallelism, until all current probability values ​​in the current probability value set have been processed.

[0207] Based on the above embodiments, the target selection algorithm can be the TopK algorithm, which selects the top K data points according to their probability values ​​from largest to smallest; where K is the selection threshold in the TopK algorithm.

[0208] Accordingly, the thread bundle parallel execution module 530 can be further used for:

[0209] The current thread uses the number of current probability values ​​greater than its own baseline value to cumulatively update its own key value.

[0210] Based on the above embodiments, the target screening algorithm can be the TopP algorithm, which sums up the probability values ​​of each data point in descending order of probability value until the summed probability value reaches P, where P is the screening threshold in the TopP algorithm.

[0211] Accordingly, the thread bundle parallel execution module 530 can be further used for:

[0212] The current thread bundle sums up all current probability values ​​that are greater than its own baseline value, and uses the summation result to update its own key value.

[0213] Based on the above embodiments, a baseline register storage module may also be included, used for:

[0214] After setting the baseline value of each thread bundle in the current iteration based on the local comparison values, the baseline value of each thread bundle in the current iteration is stored in the matching baseline register, where the baseline register is a scalar register or a vector register.

[0215] Accordingly, the thread bundle parallel execution module 530 can be further used for:

[0216] The current thread bundle sequentially retrieves multiple current probability values ​​from the current probability value set according to its own data parallelism, and stores each retrieved current probability value into the probability value register, which is a vector register.

[0217] Based on the above embodiments, the thread bundle parallel execution module 530 can be further used for:

[0218] The current thread acquires the target baseline register used to store its own baseline value;

[0219] The current thread calls a dedicated comparison setting instruction that matches the AI ​​acceleration computing chip, compares the numerical values ​​between each current probability value stored in the parallel probability value register and its own baseline value stored in the target baseline register, and stores the comparison result in the target mask register.

[0220] The target mask register is a scalar register, and when the current probability value of the target is greater than its own baseline value, the storage location in the target mask register that matches the current probability value of the target is set to 1; otherwise, the storage location is set to 0.

[0221] The current thread obtains the current probability value of each target that matches each storage location where the target mask register is set to 1, and updates its own key value based on the current probability value of each target.

[0222] Based on the above embodiments, the iterative repetition module 560 can be further used to:

[0223] After obtaining the first thread bundle, the first target mask register is updated after comparing the numerical values ​​of each current probability value in the current probability value set with its own baseline value.

[0224] Add the first type probability values ​​matched by each storage location set to 1 in the first target mask register to the target set;

[0225] After obtaining the second thread bundle, the second target mask register is updated after comparing the numerical values ​​of each current probability value in the current probability value set with its own baseline value.

[0226] Obtain the second type probability values ​​matched by each storage location that is set to 1 in the second target mask register.

[0227] Based on the above embodiments, a data dissemination module may also be included, which, after filtering out each first-class probability value from each second-class probability value, uses the result as a new current probability value set, calls a dedicated data dissemination instruction matched with the artificial intelligence acceleration computing chip to continuously store the result after filtering out each first-class probability value from each second-class probability value in shared memory, so that in subsequent iterations, the new current probability value set can be loaded with data at consecutive addresses using a linear loading method, reducing data access latency and accelerating the data filtering process.

[0228] The data filtering device provided in the embodiments of the present invention can execute the data filtering method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0229] Figure 6 This is a schematic diagram of a large language model output acceleration device provided in an embodiment of the present invention. Figure 6 As shown, the device includes: a filtering information acquisition module 610, a target set filtering module 620, and a target candidate word output module 630, wherein:

[0230] The filtering information acquisition module 610 is used to acquire the normalized probability values ​​of all candidate words obtained by the current inference of the large language model.

[0231] The target set filtering module 620 is used to filter out the target set that conforms to the target filtering algorithm from each candidate word according to the target filtering algorithm preset in the large language model and the data filtering method described in any one of the embodiments of the present invention.

[0232] The target candidate word output module 630 is used to reset the normalized probability values ​​of each candidate word in the target set, sample and obtain target candidate words in the target set according to the preset sampling rules, and output the target candidate words as the model's current round output.

[0233] The technical solution of this invention applies an improved data filtering algorithm through hardware and software collaboration during the filtering process of large language model output lexical units, which can efficiently accelerate the output of large language models and effectively improve the lexical unit generation efficiency of large language models.

[0234] The large language model output acceleration device provided in the embodiments of the present invention can execute the large language model output acceleration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0235] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0236] Figure 7 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0237] like Figure 7 As shown, the electronic device 10 includes at least one artificial intelligence (AI) accelerator chip 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one AI accelerator chip 11. The memory stores computer programs executable by the at least one AI accelerator chip. The AI ​​accelerator chip 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The AI ​​accelerator chip 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0238] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0239] The AI ​​acceleration computing chip 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the AI ​​acceleration computing chip 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various AI acceleration computing chips running machine learning model algorithms, digital signal processing AI acceleration computing chips (DSPs), and any suitable AI acceleration computing chip, controller, microcontroller, etc. The AI ​​acceleration computing chip 11 performs the various methods and processes described above, such as performing data filtering methods or large language model output acceleration methods as described in the embodiments of the present invention.

[0240] In some embodiments, the data filtering method or large language model output acceleration method as described in the various embodiments of the present invention can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by AI acceleration computing chip 11, one or more steps of the data filtering method or large language model output acceleration method described above as described in the various embodiments of the present invention can be performed. Alternatively, in other embodiments, AI acceleration computing chip 11 can be configured by any other suitable means (e.g., by means of firmware) to perform the data filtering method or large language model output acceleration method as described in the various embodiments of the present invention, wherein:

[0241] The data filtering method includes: in response to a request to filter each data according to the probability value of the data using a target filtering algorithm, selecting multiple thread bundles in the artificial intelligence acceleration computing chip according to the single comparison bit width, and determining the local comparison value corresponding to each thread bundle respectively;

[0242] The current probability value set is initialized based on the probability values ​​of each data point, and the baseline value of each thread bundle in the current iteration is set based on each local comparison value. The current probability value set contains at least one current probability value, and the data bit width of each baseline value is the same as that of each current probability value.

[0243] The parallel execution of each thread bundle involves comparing the value of each current probability value in the current probability value set with its own baseline value based on its own data parallelism, and using the current probability values ​​that are greater than its own baseline value to update its own key value.

[0244] After the operation is completed, the filtering threshold of the target filtering algorithm is compared with the key value of each thread bundle;

[0245] If the filtering threshold does not match the key value of each thread bundle, then identify the first thread bundle corresponding to the maximum lower bound key value of the filtering threshold and the second thread bundle corresponding to the minimum upper bound key value of the filtering threshold.

[0246] Based on the first and second thread bundles, after identifying the target data whose probability values ​​meet the filtering conditions and adding them to the target set and determining the new current probability value set, the next iteration round is started as the new current iteration round, and the operation of setting the baseline value of each thread bundle in the current iteration round is returned to execute based on each local comparison value.

[0247] If the filtering threshold matches the key value of the third thread bundle, the target data whose probability value meets the filtering conditions is identified by the third thread bundle and added to the target set, and the target set is used as the filtering result.

[0248] In addition, the methods to accelerate the output of this large language model include:

[0249] Obtain the normalized probability values ​​of all candidate words obtained from the current inference of the large language model;

[0250] According to the target filtering algorithm pre-set in the large language model, the data filtering method as described in any one of the embodiments of the present invention is used to filter out the target set that conforms to the target filtering algorithm from each candidate word;

[0251] After resetting the normalized probability values ​​of each candidate word in the target set, target candidate words are sampled in the target set according to the preset sampling rules, and the target candidate words are used as the output of the model in the current round.

[0252] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable AI acceleration computing chip, which may be a dedicated or general-purpose programmable AI acceleration computing chip, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0253] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to an AI-accelerated computing chip in a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the AI-accelerated computing chip, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0254] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0255] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0256] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0257] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0258] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0259] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data filtering method, characterized in that, include: In response to the request to use a target screening algorithm to filter each data according to the probability value of the data, multiple thread bundles are selected in the artificial intelligence acceleration computing chip according to the single comparison bit width, and the local comparison value corresponding to each thread bundle is determined. The current probability value set is initialized based on the probability values ​​of each data point, and the baseline value of each thread bundle in the current iteration is set based on each local comparison value. The current probability value set contains at least one current probability value, and the data bit width of each baseline value is the same as that of each current probability value. The parallel execution of each thread bundle involves comparing the value of each current probability value in the current probability value set with its own baseline value based on its own data parallelism, and using the current probability values ​​that are greater than its own baseline value to update its own key value. After the operation is completed, the filtering threshold of the target filtering algorithm is compared with the key value of each thread bundle; If the filtering threshold does not match the key value of each thread bundle, then identify the first thread bundle corresponding to the maximum lower bound key value of the filtering threshold and the second thread bundle corresponding to the minimum upper bound key value of the filtering threshold. Based on the first and second thread bundles, after identifying the target data whose probability values ​​meet the filtering conditions and adding them to the target set and determining the new current probability value set, the next iteration round is started as the new current iteration round, and the operation of setting the baseline value of each thread bundle in the current iteration round is returned to execute based on each local comparison value. If the filtering threshold matches the key value of the third thread bundle, the target data whose probability value meets the filtering conditions is identified by the third thread bundle and added to the target set, and the target set is used as the filtering result.

2. The method according to claim 1, characterized in that, Multiple thread bundles are selected in the AI ​​acceleration computing chip according to the single comparison bit width, and the local comparison value corresponding to each thread bundle is determined, including: Based on the single alignment bit width B, and the formula M thread bundles are selected in the artificial intelligence acceleration computing chip; Based on the selectable values ​​of the binary string that matches the bit width B of a single alignment, determine the local alignment value corresponding to each thread bundle.

3. The method according to claim 2, characterized in that, Based on the local comparison values, set the baseline value for each thread bundle in the current iteration round, including: After shifting the local alignment value of each thread bundle to the left by (Si*B) bits, sum it with the baseline value updated in the previous iteration of each thread bundle to obtain the baseline value of each thread bundle in the current i-th iteration. The baseline value of each thread bundle is initialized as a binary string of all zeros, and S is the data bit width of each current probability value.

4. The method according to claim 1, characterized in that, Based on the first and second thread bundles, target data whose probability values ​​meet the filtering criteria are identified and added to the target set, and a new current probability value set is determined, including: Based on the numerical difference between each current probability value obtained from the first thread bundle comparison and the first baseline value, obtain each first-class probability value that is greater than the first baseline value and add it to the target set; Based on the numerical difference between each current probability value obtained from the second thread bundle comparison and the second baseline value, obtain each second type probability value that is greater than the second baseline value, and use the result after filtering out each first type probability value from each second type probability value as a new set of current probability values; Accordingly, target data whose probability values ​​meet the filtering criteria are identified by the third thread beam are added to the target set, including: Based on the numerical difference between each current probability value obtained from the comparison with the third baseline value, obtain each third-class probability value that is greater than the third baseline value and add it to the target set.

5. The method according to claim 4, characterized in that, Parallel execution of each thread bundle involves comparing the current probability value in the current probability value set with its own baseline value based on its own data parallelism, and updating its own key value using the current probability values ​​greater than its own baseline value. This includes: Each thread bundle is triggered in parallel, and the following operations are performed respectively: From the current set of probability values, sequentially obtain multiple current probability values ​​that match its own data parallelism. In parallel, compare each current probability value with its own baseline value, and update its own key value using each current probability value that is greater than its own baseline value; Return to the previous state and execute the operation of sequentially retrieving multiple current probability values ​​from the current probability value set that match its own data parallelism, until all current probability values ​​in the current probability value set have been processed.

6. The method according to claim 5, characterized in that, The target selection algorithm is the TopK algorithm, which selects the top K data points based on their probability values ​​from largest to smallest; where K is the selection threshold in the TopK algorithm. Accordingly, the current thread bundle updates its own key value using current probability values ​​greater than its own baseline value, including: The current thread uses the number of current probability values ​​greater than its own baseline value to cumulatively update its own key value.

7. The method according to claim 5, characterized in that, The target selection algorithm is the TopP algorithm, which sums up the probability values ​​of each data point in descending order of probability value until the summed probability value reaches P, where P is the selection threshold in the TopP algorithm. Accordingly, the current thread bundle updates its own key value using current probability values ​​greater than its own baseline value, including: The current thread bundle sums up all current probability values ​​that are greater than its own baseline value, and uses the summation result to update its own key value.

8. The method according to claim 5, characterized in that, After setting the baseline value of each thread bundle in the current iteration based on the local comparison values, the following is also included: Store the baseline value of each thread bundle in the current iteration into the matching baseline register, where the baseline register is either a scalar register or a vector register; Accordingly, the current thread bundle sequentially retrieves multiple current probability values ​​from the current probability value set that match its own data parallelism, including: The current thread bundle sequentially retrieves multiple current probability values ​​from the current probability value set according to its own data parallelism, and stores each retrieved current probability value into the probability value register, which is a vector register.

9. The method according to claim 8, characterized in that, The current thread compares each current probability value with its own baseline value in parallel, and updates its own key value using the current probability values ​​that are greater than its own baseline value, including: The current thread acquires the target baseline register used to store its own baseline value; The current thread calls a dedicated comparison set instruction that matches the AI ​​acceleration computing chip, compares the numerical values ​​between each current probability value stored in the parallel probability value register and its own baseline value stored in the target baseline register, and stores the comparison result in the target mask register. The target mask register is a scalar register, and when the current probability value of the target is greater than its own baseline value, the storage location in the target mask register that matches the current probability value of the target is set to 1; otherwise, the storage location is set to 0. The current thread obtains the current probability value of each target that matches each storage location where the target mask register is set to 1, and updates its own key value based on the current probability value of each target.

10. The method according to claim 9, characterized in that, Based on the numerical comparison between each current probability value obtained from the first thread beam comparison and the first baseline value, obtain the first type probability values ​​that are greater than the first baseline value and add them to the target set, including: After obtaining the first thread bundle, the first target mask register is updated after comparing the numerical values ​​of each current probability value in the current probability value set with its own baseline value. Add the first type probability values ​​matched by each storage location set to 1 in the first target mask register to the target set; Accordingly, based on the numerical difference between each current probability value obtained from the second thread beam comparison and the second baseline value, the second type of probability values ​​greater than the second baseline value are obtained, including: After obtaining the second thread bundle, the second target mask register is updated after comparing the numerical values ​​of each current probability value in the current probability value set with its own baseline value. Obtain the second type probability values ​​matched by each storage location that is set to 1 in the second target mask register.

11. The method according to claim 10, characterized in that, After filtering out all first-class probability values ​​from each second-class probability value, the result, as the new current set of probability values, also includes: By calling a dedicated data dissemination instruction that matches the AI ​​acceleration computing chip, the results obtained after filtering out each first-class probability value from each second-class probability value are continuously stored in shared memory. This allows for linear loading of the new current probability value set in subsequent iterations, reducing data access latency and accelerating the data filtering process.

12. A method for accelerating the output of a large language model, characterized in that, include: Obtain the normalized probability values ​​of all candidate words obtained from the current inference of the large language model; Based on the target filtering algorithm pre-defined in the large language model, the data filtering method described in any one of claims 1-11 is used to filter out the target set that conforms to the target filtering algorithm from each candidate word; After resetting the normalized probability values ​​of each candidate word in the target set, target candidate words are sampled in the target set according to the preset sampling rules, and the target candidate words are used as the output of the model in the current round.

13. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data filtering method of any one of claims 1-11, or to perform the large language model output acceleration method of claim 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data filtering method of any one of claims 1-11, or the large language model output acceleration method of claim 12.

15. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data filtering method according to any one of claims 1-11, or the large language model output acceleration method according to claim 12.

Images