Computing method and device of neural network, electronic equipment and storage medium

Abstract

A neural network computing method and device, an electronic device, and a storage medium. The neural network computing method includes: performing sorting processing and deduplication processing on an index sequence corresponding to a word element to obtain a plurality of index segments; dividing each index segment in the plurality of index segments into a plurality of index segment shards according to a first segmentation granularity; assigning a plurality of thread bundles to corresponding index segment shards respectively, calculating a calculation result of each index segment shard in parallel through the plurality of thread bundles, and storing the calculation result of each index segment shard in a tensor core cache of a tensor core; and assigning the plurality of thread bundles to corresponding index segments respectively, accumulating calculation results belonging to the same index segment to obtain a calculation total of each index segment, and writing the calculation total of each index segment into the tensor core cache.

Description

Technical Field

[0001] Embodiments of this disclosure relate to a method and apparatus for computing a neural network, an electronic device, and a storage medium. Background Technology

[0002] During the training of large models, the embedding layer is responsible for mapping discrete symbols to continuous vectors. It aggregates the embedding vectors by index in the forward pass and scatters and accumulates the gradients by index in the backward pass. For example, on a general-purpose graphics processing unit (GPGPU), the embedding layer typically reads gradients sequentially during backpropagation, usually accumulating them to the corresponding positions in the vocabulary weight gradient matrix within high-bandwidth memory through atomic operations. Summary of the Invention

[0003] At least one embodiment of this disclosure provides a computation method for a neural network based on a tensor kernel, comprising: sorting and deduplicating an index sequence corresponding to a word to obtain multiple index segments; dividing each of the multiple index segments into multiple index segment pieces according to a first segmentation granularity; assigning multiple thread bundles to the corresponding index segment pieces respectively, computing the computation results of each index segment piece in parallel through the multiple thread bundles, and storing the computation results of each index segment piece in the tensor kernel cache; and assigning multiple thread bundles to the corresponding index segments respectively, accumulating the computation results belonging to the same index segment to obtain the computation sum of each index segment, and writing the computation sum of each index segment into the tensor kernel cache.

[0004] For example, in a computation method of a neural network based on tensor kernels provided in an embodiment of this disclosure, the sorting and deduplication processing of the index sequence corresponding to the word includes: performing the sorting process on the index sequence corresponding to the word to obtain a sorted index sequence and an index mapping table; and performing deduplication processing on the sorted index sequence.

[0005] For example, in a computation method for a neural network based on tensor kernels provided in an embodiment of this disclosure, the deduplication process for the sorted index sequence includes: performing the deduplication process on the sorted index sequence to obtain an index segment with a unique index sequence.

[0006] For example, in a computation method for a neural network based on tensor kernels provided in an embodiment of this disclosure, the step of dividing each of the plurality of index segments into a plurality of index segment fragments according to a first segmentation granularity includes: determining the number of index segment fragments corresponding to each index segment according to the first segmentation granularity and the index segments of the unique index sequence, and determining the number of each index segment fragment according to the number of index segment fragments corresponding to each index segment, wherein the number of each index segment fragment is used to determine the correspondence between the plurality of thread bundles and each index segment fragment.

[0007] For example, in a computation method for a neural network based on tensor kernels provided in an embodiment of this disclosure, determining the number of index segment fragments corresponding to each index segment includes: determining the number of index segment fragments corresponding to each index segment based on the number of index segments with unique index sequences and a first segmentation granularity.

[0008] For example, in a computation method for a neural network based on tensor kernels provided in an embodiment of this disclosure, the first segmentation granularity is determined based on the number of index segment fragments corresponding to each index segment and the number of thread bundles.

[0009] For example, in a computation method for a neural network based on tensor kernels provided in an embodiment of this disclosure, the step of assigning multiple thread bundles to corresponding index segment slices and computing the computation result of each index segment slice in parallel through multiple thread bundles includes: assigning a corresponding thread bundle to each index segment slice; each thread bundle traversing all words in the corresponding index segment slice and looking up the gradient value corresponding to each word through an index mapping table; and accumulating the gradient values ​​corresponding to each word to obtain the computation result of each index segment slice.

[0010] For example, the computation method of a neural network based on tensor kernels provided in one embodiment of this disclosure further includes: determining a second segmentation granularity and dividing the loop rounds based on the vocabulary size and the capacity of the tensor kernel cache; in each loop round, determining the index segment fragments belonging to the current loop index sequence range; assigning multiple thread bundles to multiple valid index segment fragments in the current loop, with each thread bundle calculating the computation results of all words in its assigned index segment fragment in parallel; after each loop round ends, clearing or marking the computation results of the current loop in the tensor kernel cache as overwriteable, and entering the next loop round until all index segment fragments are processed.

[0011] For example, in a computation method for a neural network based on a tensor kernel provided in one embodiment of this disclosure, the step of assigning multiple thread bundles to corresponding index segments and accumulating the computation results belonging to the same index segment to obtain the computation summation of each index segment includes: assigning a corresponding thread bundle to each index segment; each thread bundle obtaining the computation results of the index segment fragments belonging to each index segment from the tensor kernel cache according to the index sequence of the corresponding index segment; and accumulating the computation results belonging to the same index segment to obtain the computation summation of each index segment.

[0012] For example, the computation method of a neural network based on a tensor kernel provided in one embodiment of this disclosure further includes: determining a second segmentation granularity and dividing the loop rounds based on the vocabulary size and the capacity of the tensor kernel cache; in each loop round, determining the index segments belonging to the current loop index sequence range; assigning multiple thread bundles to the multiple index segments in the current loop respectively, and accumulating the computation results of all index segment fragments corresponding to the assigned index segment by each thread bundle to obtain the computation summation of the index sequence corresponding to each index segment; and after each loop round ends, clearing or marking the computation result of the current loop in the tensor kernel cache as overwriteable, and entering the next loop round until the computation summation of all index segments is completed.

[0013] For example, a computation method for a neural network based on a tensor kernel provided in one embodiment of this disclosure further includes: obtaining the term from a high-bandwidth memory according to the index sequence; reading the computational summation of all index segments from the tensor kernel cache; and writing the computational summation from the tensor kernel cache into the high-bandwidth memory according to the index sequence.

[0014] For example, in one embodiment of the present disclosure, a computation method for a neural network based on tensor kernels is provided, wherein the computation summation is written into the high-bandwidth memory according to the index sequence, comprising: continuously writing the computation summation into the vocabulary storage location in the high-bandwidth memory corresponding to each index sequence according to the order of the index sequence.

[0015] For example, in one embodiment of this disclosure, a computation method for a tensor kernel-based neural network is provided, wherein the computation method for the tensor kernel-based neural network is executed during the backpropagation process of the embedding layer.

[0016] At least one embodiment of this disclosure provides a computing device for a neural network based on a tensor kernel, comprising: a processing module, a segmentation module, a first computing module, and a second computing module. The processing module is configured to sort and deduplicate the index sequences corresponding to tokens to obtain multiple index segments; the segmentation module is configured to segment each of the multiple index segments into multiple index segment fragments according to a first segmentation granularity; the first computing module is configured to allocate multiple thread bundles to the corresponding index segment fragments, compute the calculation results of each index segment fragment in parallel using the multiple thread bundles, and store the calculation results of each index segment fragment in the tensor kernel cache; the second computing module is configured to allocate multiple thread bundles to the corresponding index segments, accumulate the calculation results belonging to the same index segment to obtain a total calculation for each index segment, and write the total calculation for each index segment into the tensor kernel cache.

[0017] At least one embodiment of this disclosure provides an electronic device, including a computing device based on a tensor kernel neural network provided in any embodiment of this disclosure.

[0018] At least one embodiment of this disclosure provides an electronic device, including a processor and a memory. The memory stores at least one computer program, which, when executed by the processor, implements the computation method of a tensor kernel-based neural network provided in any embodiment of this disclosure.

[0019] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium for non-transitory storage of computer-readable instructions, which, when executed by a computer, implement the computation method of a tensor kernel-based neural network provided in any embodiment of this disclosure.

[0020] Through at least one embodiment of the present disclosure as described above, data execution conflicts arising from atomic operations can be overcome, and hardware parallelism can be further fully utilized, significantly improving the computational density and efficiency of the computing unit. In at least one embodiment, by dynamically dividing a large index segment into multiple evenly sized index segment fragments, the computational tasks can be more evenly distributed across a large number of thread bundles, thereby ensuring that each thread bundle is fully utilized, further reducing the idle computing resources caused by uneven load, and improving the overall throughput of the system. Furthermore, all intermediate accumulation results (computation results and computational summation) are completed and temporarily stored in a high-speed tensor kernel cache, greatly reducing the number of accesses to high-latency memory. The computational summation of each index segment is continuously written back to high-bandwidth memory (e.g., HBM) in the processed order, improving the continuous memory access bandwidth storage capacity of memory, thereby significantly improving memory access performance. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below relate to some embodiments of this disclosure, and are not intended to limit this disclosure.

[0022] Figure 1A A schematic diagram of discrete memory access and atomic operations is shown.

[0023] Figure 1B A schematic diagram of a general-purpose graphics processor (GPGPU) provided in at least one embodiment of the present disclosure is shown.

[0024] Figure 2 A flowchart is shown of a computation method for a tensor kernel-based neural network provided in at least one embodiment of this disclosure.

[0025] Figure 3 A schematic diagram of an index sequence preprocessing application provided by at least one embodiment of the present disclosure is shown.

[0026] Figure 4 The illustration shows an application diagram of a computational method based on a tensor kernel neural network provided by at least one embodiment of the present disclosure.

[0027] Figure 5 This illustration shows an application diagram of another computational method based on a tensor kernel neural network provided by at least one embodiment of the present disclosure.

[0028] Figure 6 This illustration shows an application diagram of another computational method based on a tensor kernel neural network provided in at least one embodiment of the present disclosure.

[0029] Figure 7 A schematic block diagram of a computing device based on a tensor kernel neural network provided in at least one embodiment of the present disclosure is shown.

[0030] Figure 8 A schematic block diagram of an electronic device provided in at least one embodiment of the present disclosure is shown.

[0031] Figure 9 A schematic block diagram of another electronic device provided in at least one embodiment of the present disclosure is shown.

[0032] Figure 10 A schematic diagram of a computer-readable storage medium provided in at least one embodiment of the present disclosure is shown. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0034] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising,” “including,” or similar terms mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. The terms “connected,” “linked,” and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “upper,” “lower,” “left,” and “right,” etc., are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0035] With the rapid development of artificial intelligence technology, large-scale language models based on the Transformer architecture have achieved remarkable success in fields such as natural language processing and content generation. In the training process of these large models, the embedding layer plays a crucial role in mapping discrete input symbols (such as tokens) into continuous vector representations.

[0036] A token is a basic unit obtained after input text has been segmented, such as a word, subword, or character. In model training, each token can be regarded as a discrete symbol.

[0037] A vocabulary size is a predefined list containing all possible lexical units. Each lexical unit in the vocabulary is assigned an index, which represents the position of that lexical unit within the vocabulary. Therefore, a vocabulary can be viewed as a mapping from "indexes" to "lexical vector representations".

[0038] In a model processing an input sentence, each word in the sentence is converted into a corresponding index based on its position in the vocabulary. The entire sentence is thus transformed into an index sequence.

[0039] Figure 1A A schematic diagram of discrete memory access and atomic operations is shown.

[0040] During the backpropagation phase of the embedding layer, the gradients of the tokens need to be accumulated and updated to the vocabulary in a high-bandwidth memory (HBM).

[0041] like Figure 1A As shown, an input sequence contains multiple tokens (Token0 to Token7). Each token corresponds to an index, and these indices form an index sequence (e.g., [2,5,2,1,2,3,5,0]), which indicates the position of each token in the vocabulary.

[0042] During backpropagation, each token generates a gradient (grad_token), which is a vector with dimensions matching the model's embedding dimension (embedding dim). These gradients need to be "scattered" back into the vocabulary's gradient matrix (grad_weight). The scattering rule is determined by the index sequence: the gradients of all tokens with the same index need to be accumulated in the same row of the vocabulary's gradient matrix.

[0043] For example, if there are three tokens (token0, token2, token4) in the index sequence that all have an index of 2, then during backpropagation, the gradients of these three tokens need to be accumulated and then used to update the row with index 2 in the vocabulary gradient matrix.

[0044] The forward propagation process of the embedding layer is to gather the corresponding embedding vectors from the vocabulary weight matrix according to the input index sequence, while its backward propagation (bwd) process is to scatter and reduce the gradients calculated upstream back to the corresponding positions in the vocabulary weight matrix according to the same index sequence.

[0045] Specifically, the gradients of all lexical units originating from the same index need to be summed in the row of that index in the vocabulary weight matrix.

[0046] For example, when implementing embedding backpropagation (embedding_bwd) on high-performance computing hardware for AI processing, such as general-purpose graphics processing units (GPGPUs), a straightforward approach is typically adopted. This involves sequentially reading the gradient data corresponding to each input lexical unit and then, based on its index, accumulating the gradient value to the corresponding position in the vocabulary weight gradient matrix in a high-bandwidth memory (such as HBM) through atomic operations, such as atomic_add.

[0047] The inventors of this disclosure have noted that the above-mentioned technical solutions have at least the following defects or technical bottlenecks.

[0048] For example, there is a technical problem of low memory access efficiency. Since the distribution of indices in the input sequence is usually random and disordered, the memory access pattern of writing back to high-bandwidth memory (such as HBM) via atomic operations exhibits a high degree of discontinuity. This random memory access cannot effectively utilize the continuous memory access bandwidth advantage of high-bandwidth memory (such as HBM), resulting in severe memory access latency and becoming the performance bottleneck of the entire training process, namely the "memory wall" problem.

[0049] For example, there is a serious technical problem of atomic operation conflicts. In large-scale model training, the batch size and sequence length are extremely large, leading to a situation where many different terms point to the same vocabulary index. When multiple computation threads simultaneously attempt to atomically accumulate gradient values ​​at the same location in high-bandwidth memory (such as HBM), large-scale access conflicts occur. These conflicts force a large number of threads into a serial waiting state, severely reducing the parallel computing efficiency of massively parallel processors such as GPGPUs.

[0050] For example, there is also the technical problem of poor system scalability. Referring to the memory access efficiency and atomic conflict issues mentioned above, these problems become more and more significant as the vocabulary size increases, the model embedding dimension increases, and the training batch size grows. This restricts further improvement in training speed and increases the time and economic cost of training large models.

[0051] Therefore, there is an urgent need for an optimization algorithm that can effectively avoid atomic operation conflicts, optimize memory access patterns, and thus significantly improve the computational efficiency of backpropagation in the embedding layer.

[0052] Figure 1B A schematic diagram of a general-purpose graphics processor (GPGPU) provided in at least one embodiment of the present disclosure is shown.

[0053] like Figure 1BAs shown, a general-purpose graphics processor is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), such as including... Figure 1B The diagram shows streaming processor clusters 1, ..., M, where M is a positive integer greater than 1. In a general-purpose graphics processor, one streaming processor cluster handles one computational task, or multiple streaming processor clusters handle one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0054] like Figure 1B As shown, taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, such as... Figure 1B The system is structured as Computation Unit 1, Computation Unit 2, ..., Computation Unit N, where N is a positive integer. Each Computation Unit (CU) performs arithmetic and logical operations, such as accumulation, reduction, and standard addition, subtraction, multiplication, and division. A Computation Unit includes multiple cores (also called computational kernels), each of which includes an Arithmetic Logic Unit (ALU), a floating-point unit, etc. These cores are used to execute specific computational tasks. Furthermore, the Computation Unit also includes registers (e.g., ...). Figure 1B The register file and shared memory in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared memory in a computing unit is used to share data between the cores of that computing unit.

[0055] like Figure 1B As shown, each computing unit also provides a tensor core for performing tensor-related computations, such as tensor shrinking operations. Tensor cores can accelerate tensor operations such as matrix multiplication. Tensor cores in multiple computing units can be scheduled and controlled uniformly.

[0056] like Figure 1B As shown, each streaming processor cluster also provides a buffer for caching data from the N computational units in the streaming processor cluster, such as a tensor buffer.

[0057] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1B(Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block can be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply thread bundles, warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0058] In each computing unit, the thread beam scheduling / distribution module ( Figure 1B (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can execute concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory-executed instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or global memory (e.g., [example cache]). Figure 1B High-bandwidth memory (HBM) is used for read and write operations, etc.

[0059] At least one embodiment of this disclosure provides a computation method and apparatus, electronic device, and non-transient computer-readable storage medium for a tensor kernel-based neural network. The computation method for the tensor kernel-based neural network includes: sorting and deduplicating an index sequence corresponding to a term to obtain multiple index segments; dividing each index segment into multiple index segment pieces according to a first segmentation granularity; assigning multiple thread bundles to the corresponding index segment pieces, computing the computation results of each index segment piece in parallel using the multiple thread bundles, and storing the computation results of each index segment piece in a tensor kernel cache; and assigning multiple thread bundles to the corresponding index segments, accumulating the computation results belonging to the same index segment to obtain a computational sum for each index segment, and writing the computational sum for each index segment into the tensor kernel cache. This computation method for the tensor kernel-based neural network can significantly reduce the backpropagation time of the embedding layer, thereby accelerating the training process of the entire large model.

[0060] The computation method of the tensor kernel-based neural network provided in at least one embodiment of this disclosure can be executed, for example, during the backpropagation process of the embedding layer. For example, in forward propagation, the embedding layer gathers the corresponding vectors from the vocabulary according to the index; while in backpropagation, the gradients need to be scattered and accumulated back to the corresponding positions in the vocabulary.

[0061] It should be noted that although the embodiments of this disclosure take the embedding layer as an example, the technical concept of this disclosure can also be applied to other scenarios that require similar scatter-reduce operations, such as node feature updates in graph neural networks and sparse matrix operations.

[0062] Figure 2 A flowchart is shown of a computation method for a tensor kernel-based neural network provided in at least one embodiment of this disclosure.

[0063] like Figure 2 As shown, the computation method of the tensor kernel-based neural network includes steps S100 to S130.

[0064] Step S100: Sort and deduplicate the index sequences corresponding to the tokens to obtain multiple index segments.

[0065] Step S110: Based on the first granularity of slicing, each index segment in the multiple index segments is sliced ​​into multiple index segment fragments.

[0066] Step S120: Assign multiple thread bundles to the corresponding index segment slices respectively, calculate the calculation result of each index segment slice in parallel through multiple thread bundles, and store the calculation result of each index segment slice in the tensor kernel cache of the tensor kernel.

[0067] Step S130: Assign multiple thread bundles to their respective index segments, sum the calculation results of the same index segment to obtain the total calculation of each index segment, and write the total calculation of each index segment to the tensor kernel cache.

[0068] Step S100 is the data preprocessing and reorganization stage, which can transform the original disordered index sequence, which may contain a large number of duplicates, into an ordered and logically grouped structure, laying the foundation for subsequent conflict-free parallel accumulation calculation.

[0069] For example, the preprocessed data may include text data, image data, audio data, or video data, etc., and the embodiments disclosed herein do not limit this.

[0070] For example, in a training batch, a sequence of 8 tokens (e.g., 8 words in a sentence) is transformed into an index sequence [2,5,2,1,2,3,5,0] by querying the vocabulary. This sequence indicates that the tokens at positions 0, 2, and 4 all correspond to the same token with index 2 in the vocabulary. Therefore, during backpropagation, the gradients of these three tokens need to be accumulated and used to update the row with index 2 in the vocabulary gradient matrix.

[0071] First, the index sequence can be sorted. For example, a sorting algorithm suitable for parallel computing (such as radix sort) can be used to obtain the sorted index sequence: [0,1,2,2,2,3,5,5]. This process ensures that all identical indices are arranged contiguously in physical memory.

[0072] At the same time, an index map can be generated, which can be used to record the position in the original input sequence corresponding to each position in the sorted sequence, so that the original gradient data corresponding to each index can be found correctly later.

[0073] Subsequently, the sorted index sequence can be deduplicated. This process identifies all contiguous regions consisting of the same index and defines each such region as an index segment. Each index segment is associated with a unique index value. For example, after deduplicating the sequence [0,1,2,2,2,3,5,5], five index segments are obtained, each corresponding to a unique index [0,1,2,3,5]. The segments corresponding to indices 2 and 5 contain multiple terms. For instance, another mapping table can be used to record information such as the number of each index before deduplication, allowing the sequence to be restored before deduplication when needed.

[0074] For step S110, even if the index sequence has been grouped, some index segments may contain far more terms than other index segments. Directly having a single thread handle an excessively large index segment could lead to underutilization of computational resources and latency.

[0075] For example, the first segmentation granularity can be a configurable parameter used to control the granularity at which an index segment is segmented into smaller task units. For example, the first segmentation granularity can be set to 2 or 3, etc., and the embodiments of this disclosure are not limited thereto.

[0076] For example, based on this first segmentation granularity, each index segment can be further segmented into multiple index segment partitions. Each index segment partition can be a subset of the original index segment, including a continuous set of terms within its corresponding index segment.

[0077] For example, suppose the index segment corresponding to index 2 contains 3 words. Using the first segmentation granularity 2, this index segment is divided into two fragments: the first fragment contains 2 words, and the second fragment contains 1 word. In this way, a potentially heavy task (processing 3 words) can be split into two smaller, more balanced tasks (processing 2 and 1 words respectively), facilitating subsequent parallel scheduling.

[0078] Step S120 is the first stage of parallel computing. For example, a thread bundle is used as the basic unit of execution for an AI processor such as a GPGPU. For example, multiple available thread bundles can be dynamically allocated to multiple index segment slices generated in step S110. A thread bundle can be configured to be responsible for processing a corresponding index segment slice.

[0079] For example, each assigned thread can perform the following operations: traverse all tokens within its corresponding index segment, use the index mapping table to find the position of each token in the original input sequence, and thus obtain the corresponding gradient vector from the upstream gradient tensor (grad_output); sum the gradient vectors of all tokens within the index segment, and this summation result is the computation result of that index segment. Since all tokens within an index segment belong to the same unique index, this summation is completed within the thread, thus overcoming the technical bottleneck caused by atomic operations.

[0080] For example, the computation results calculated by all threads can be uniformly stored in the tensor kernel cache. The tensor kernel cache is a high-speed memory located on the GPGPU chip, with access latency far lower than high-bandwidth memory.

[0081] Step S130 is the second stage of parallel computing. For example, thread bundles can be reassigned as tasks, where the task unit is an index segment (i.e., a unique index). For example, each index segment can be assigned a thread bundle.

[0082] For example, each assigned thread bundle performs the following operation: Based on the index segment it is responsible for, it collects the computation results of all index segment fragments belonging to that index segment from the tensor kernel cache. For example, the thread bundle responsible for index 2 can collect the two computation results calculated in step S120 by the two index segment fragments corresponding to that index. For example, these computation results from different fragments are finally summed, and this final summation is the computation summation of that index segment (i.e., the unique index).

[0083] For example, the summed value can be used to represent the total gradient that ultimately needs to be updated for that vocabulary position.

[0084] For example, each thread bundle sums the computed calculations and writes them back to a specific area in the tensor kernel cache.

[0085] Through at least one embodiment of the present disclosure as described above, data execution conflicts arising from atomic operations can be overcome, and hardware parallelism can be further fully utilized, significantly improving the computational density and efficiency of the computing unit. In at least one embodiment, by dynamically dividing a large index segment into multiple evenly sized index segment fragments, the computational tasks can be more evenly distributed across a large number of thread bundles, thereby ensuring that each thread bundle is fully utilized, further reducing the idle computing resources caused by uneven load, and improving the overall throughput of the system. Furthermore, all intermediate accumulation results (computation results and computational summation) are completed and temporarily stored in a high-speed tensor kernel cache, greatly reducing the number of accesses to high-latency memory. The computational summation of each index segment is continuously written back to high-bandwidth memory (e.g., HBM) in the processed order, improving the continuous memory access bandwidth storage capacity of memory, thereby significantly improving memory access performance.

[0086] In some embodiments of this disclosure, the method of sorting and deduplicating the index sequence corresponding to the word element in step S100 may further include sub-steps S101 to S102.

[0087] Step S101: Sort the index sequence corresponding to the word element to obtain the sorted index sequence and index mapping table.

[0088] Step S102: Perform deduplication on the sorted index sequence.

[0089] For step S101, the original unordered index sequence can be converted into an ordered sequence, laying the foundation for subsequent efficient grouping processing.

[0090] For the received raw index sequence from the upstream computing unit, for example, an input containing 8 tokens, the raw index sequence is [2,5,2,1,2,3,5,0].

[0091] For example, the sequence can be sorted using a sorting algorithm suitable for massively parallel processing, such as radix sort, which is well-suited for AI processors like GPGPUs. For instance, the sorted sequence would be [0,1,2,2,2,3,5,5], thus ensuring that identical indices are clustered together contiguously in physical memory.

[0092] For example, during the sorting process, an index map is generated synchronously. This index map can be used to record the position of each position in the sorted sequence and its corresponding position in the original sequence.

[0093] For example, for the sorted sequence [0,1,2,2,2,3,5,5], its index mapping table can be represented as [7,3,0,2,4,5,1,6]. For instance, the index 0 of the first position after sorting corresponds to the word at the 7th position in the original sequence; the index 1 of the second position after sorting corresponds to the word at the 3rd position in the original sequence, and so on.

[0094] For example, this index mapping table can be used to accurately find the gradient data corresponding to each lexical in subsequent steps.

[0095] For step S102, based on the sorting, consecutive regions with the same index can be determined. For example, the sorted index sequence [0,1,2,2,2,3,5,5] can be traversed to determine multiple consecutive intervals consisting of the same index. These intervals can be used to define "index segments" later. For example, one interval can be determined for index 0, one interval for index 1, one consecutive interval for index 2 (including 3 positions), one interval for index 3, and one consecutive interval for index 5 (including 2 positions).

[0096] In at least one embodiment of this disclosure, sorting the indexes transforms subsequent access to gradient data from completely random access to partially ordered access. When reading gradients from the index mapping table, the corresponding gradient data are more likely to be located adjacently in memory due to the proximity of the indices, thereby improving cache hit rate and significantly reducing latency when reading data from high-bandwidth memory (e.g., HBM).

[0097] In some embodiments of this disclosure, step S102 may further include sub-step S1021.

[0098] Step S1021: Perform deduplication on the sorted index sequence to obtain an index segment with a unique index sequence.

[0099] For example, after determining the continuous intervals with the same index in step S102, step S1021 can further abstract these intervals into index segments. Each index segment can represent a unique index value and includes all positional information of that index in the sorted sequence.

[0100] Following the example above, after deduplication, a set of index segments [0,1,2,3,5] is obtained. For example, each segment is bound to a unique index: the index segment corresponding to index 0, the index segment corresponding to index 1, the index segment corresponding to index 2 (for example, covering the positions of all three 2s in the sorted sequence), the index segment corresponding to index 3, and the index segment corresponding to index 5 (for example, covering the positions of all two 5s in the sorted sequence).

[0101] These index segments can form objects for subsequent granular segmentation. For example, the segment corresponding to index 2 (which includes 3 words) can be further segmented into multiple index segment fragments (e.g., segmented into 2 parts with a granularity of 2).

[0102] In at least one embodiment of this disclosure, by obtaining unique index segments through deduplication, multiple accumulation tasks originally scattered and targeting the same index can be merged into a unified computation task, thereby overcoming the technical problem of atomic operations. Furthermore, the index segment clearly defines the computational and data range of each unique index, providing precise input for the next step of index segment partitioning (e.g., dividing into index segment fragments), ensuring that the computation task neither misses any data that needs to be accumulated nor involves duplicate computation, thus providing the efficiency of thread-based parallel computation.

[0103] In some embodiments of this disclosure, step S110 may further include sub-steps S111 to S112.

[0104] Step S111: Determine the number of index segment fragments (partials_per_segment) corresponding to each index segment based on the first segmentation granularity and the index segments of the unique index sequence.

[0105] Step S112: Determine the number of shards for each index segment based on the number of shards for each index segment.

[0106] For example, the number of partitions per index segment (num_per_partials) can be used to determine the correspondence between multiple thread bundles and each partition of the index segment.

[0107] For example, the first segmentation granularity can also be dynamically determined.

[0108] For example, the first segmentation granularity can be determined based on the number of index segment fragments corresponding to each index segment and the number of thread bundles. This allows for the distribution of computational tasks (i.e., index segments with the largest number of tokens) across various thread bundles, improving computational efficiency.

[0109] For example, the formula for calculating the first segmentation granularity can be expressed as:

[0110] First segment granularity = CEILING(num_max / max_warps).

[0111] For example, the index segment with the most tokens (i.e., the index sequence with the most repetitions) can be found from the index segments obtained in step S100. For example, among all index segments, the index segment corresponding to index sequence 2 has the most tokens, let's assume num_max=8. Then, the maximum number of warps (max_warps) available for computation on the current GPGPU can be determined. This number of warps can be determined by the hardware specifications and kernel startup configuration.

[0112] For example, assuming num_max=8 and max_warps=5, then:

[0113] The first segment size = CEILING(8 / 5) = CEILING(1.6) = 2.

[0114] For example, for each index segment, the number of index segment fragments (partials_per_segment) is calculated based on its own number of tokens (count).

[0115] For example, the formula for calculating the number of index segment fragments can be expressed as:

[0116] partials_per_segment=CEILING(number of tokens in the i-th index segment / first segmentation granularity).

[0117] For example, determining the number of index segment fragments (partials_per_segment) corresponding to each index segment in step S111 may further include determining the number of index segment fragments corresponding to each index segment based on the number of index segments with unique index sequences and the first segmentation granularity.

[0118] For example, after removing duplicates from the index sequence [0,1,2,2,2,3,5,5], the number of tokens obtained are [1,1,3,1,2], respectively.

[0119] For example, let's take a first segmentation size of 2 as an example for calculation:

[0120] Index segment 0: count=1, CEILING(1 / 2)=1;

[0121] Index segment 1: count=1, CEILING(1 / 2)=1;

[0122] Index segment 2: count=3, CEILING(3 / 2)=2. Here, 3 words require more than 1 segment (each segment has a maximum of 2 words) to process.

[0123] Index segment 3: count=1, CEILING(1 / 2)=1;

[0124] Index segment 5: count=2, CEILING(2 / 2)=1;

[0125] Therefore, the resulting partials_per_segment array is [1,1,2,1,1].

[0126] For step S112, for example, each index segment and its corresponding partials_per_segment value can be traversed. For each index segment, the tokens it includes can be distributed as evenly as possible into a specified number of segments according to their order in the sorted sequence, and each segment can record the actual number of tokens it corresponds to.

[0127] Figure 3 A schematic diagram of an index sequence preprocessing application provided by at least one embodiment of the present disclosure is shown.

[0128] like Figure 3 As shown, for example, taking the partials_per_segment array as [1,1,2,1,1] and the first segmentation granularity as 2:

[0129] Index segment 0: partials_per_segment=1, count=1, can be divided into 1 index segment fragment, which corresponds to 1 word, and num_per_partials records [1];

[0130] Index segment 1: partials_per_segment=1, count=1, can be divided into 1 index segment fragment, which corresponds to 1 word, and num_per_partials is appended [1];

[0131] Index segment 2: partials_per_segment=2, count=3, can be divided into 2 index segment shards. To balance the load, the first index segment shard (Part0) corresponds to 2 words, and the second shard (Part1) corresponds to 1 word, and num_per_partials is appended with [2,1].

[0132] Index segment 3: partials_per_segment=1, count=1, can be divided into 1 index segment fragment, which corresponds to 1 word, and num_per_partials is appended[1];

[0133] Index segment 5: partials_per_segment=1, count=2, can be divided into 1 index segment fragment, which corresponds to 2 words, and num_per_partials is appended [2];

[0134] Finally, a global, flattened list of the number of partitions for each index segment is generated: num_per_partials=[1,1,2,1,1,2].

[0135] For example, num_per_partials sequentially describes the number of tokens that should be processed for each index segment fragment to be created.

[0136] For example, num_per_partials can determine the total number of thread bundles that need to be started and is used to determine the correspondence between multiple thread bundles and each index segment fragment.

[0137] In at least one embodiment of this disclosure, precise task partitioning based on the actual data distribution is achieved through two-level calculations of `partials_per_segment` and `num_per_partials`. This allows the computational tasks to be finely and rationally decomposed. For index segments with large data volumes, the load on a single thread bundle is reduced by increasing the number of index segment fragments; for index segments with small data volumes, fewer index segment fragments are maintained to reduce scheduling overhead. This fine-grained management makes the computation time of all thread bundles tend to be similar, achieving a high degree of load balancing and maximizing the utilization of parallel computing resources. Furthermore, the local accumulation operation performed by each thread bundle when processing its corresponding index segment fragment can be completed within the thread bundle itself, without the need for synchronization or atomic operations with thread bundles processing other index segment fragments, further improving the efficiency of parallel computing.

[0138] In some embodiments of this disclosure, in step S120, the method of allocating multiple thread bundles to the corresponding index segment fragments and calculating the calculation results of each index segment fragment in parallel by multiple thread bundles may further include sub-steps: S1201~S1203.

[0139] Step S1201: Assign a corresponding thread bundle to each index segment shard.

[0140] Step S1202: Each thread bundle traverses all tokens in the corresponding index segment and looks up the gradient value corresponding to each token through the index mapping table.

[0141] Step S1203: Accumulate the gradient values ​​corresponding to each word to obtain the calculation results of each index segment fragment.

[0142] For step S1201, task scheduling can be performed, for example, based on the list of index segment fragments (i.e., all partials) determined in step S110. This list includes the total number of all index segment fragments that need to be processed (e.g., the length of the num_per_partials list).

[0143] For example, a warp may include 32 threads that can execute the same instructions in a lock-step manner, which is not a limitation of the embodiments disclosed herein.

[0144] For example, a one-to-one mapping relationship can be established between thread bundles and index segment shards. For instance, the first thread bundle (Warp 0) is assigned to the first index segment shard (Partial 0), the second thread bundle (Warp 1) is assigned to the second index segment shard (Partial 1), and so on, until all shards are assigned.

[0145] For example, thread bundle allocation can also be performed based on the list of index segment fragment counts (num_per_partials) determined in step S112. Each element in the num_per_partials list represents a specific index segment fragment and the number of tokens it needs to process.

[0146] For example, a corresponding thread bundle can be assigned to each index segment fragment listed in the num_per_partials list, in the order listed.

[0147] For example, consider num_per_partials=[1,1,2,1,1,2]:

[0148] Warp 0 can be assigned to the first segment that processes one word;

[0149] Warp 1 can be assigned to the second slice that processes one word;

[0150] Warp 2 can be assigned to the third slice that processes 2 tokens;

[0151] Warp 3 can be allocated to the fourth slice that processes one lexical unit;

[0152] Warp 4 can be allocated to the fifth segment that processes one lexical unit;

[0153] Warp 5 can be allocated to the sixth segment, which processes two tokens. This thread-based allocation method ensures that each index segment has corresponding computing resources and can be started and executed in parallel.

[0154] For step S1202, for example, each activated thread bundle first determines the data range covered by its corresponding index segment piece (i.e., the continuous or segmented continuous word positions and word counts corresponding to the index segment piece in the sorted index), and the threads within the thread bundle cooperate to traverse each word position within the index segment piece. For example, Warp 2 corresponding to num_per_partials[2]=2 can traverse 2 word positions.

[0155] For example, for each lexical position traversed, the thread bundle uses the index mapping table in step S100 to look up the corresponding position in the original input sequence. Based on this original position, the thread bundle can read the gradient value corresponding to the lexical (e.g., a vector of length embedding_dim) from the global memory (e.g., a high-bandwidth memory (e.g., HBM)) that stores the upstream gradients.

[0156] For step S1203, for example, each thread bundle can maintain a local accumulator (e.g., it can be implemented using the registers or shared memory of the threads within the thread bundle). For example, the thread bundle can sequentially accumulate the gradient vectors of all words belonging to its index segment segment read in S1202 into this local accumulator.

[0157] For example, once the gradients of all words within an index segment have been accumulated, the result obtained is the calculation result of that index segment.

[0158] For example, each thread writes its computation results to a pre-allocated location in the tensor buffer of the tensor kernel that uniquely corresponds to its index segment fragment. This tensor buffer is located in high-speed memory on the chip, providing fast access speed for subsequent stages.

[0159] In at least one embodiment of this disclosure, by arranging thread bundles on index segment fragments, the serialization waiting problem caused by atomic operations is solved, enabling multiple threads to achieve high-efficiency parallel computing, reducing thread bundle resource idleness and task queuing, and greatly improving the utilization efficiency of computing units.

[0160] In some embodiments of this disclosure, step S120 may further include sub-steps: S1204~S1207.

[0161] Step S1204: Based on the vocabulary size and the capacity of the tensor kernel cache, determine the second segmentation granularity and divide the loop rounds.

[0162] Step S1205: In each iteration, determine the index segment fragments that belong to the current loop index sequence range.

[0163] Step S1206: Assign multiple thread bundles to multiple valid index segment slices in the current loop, and have each thread bundle calculate the calculation results of all words in the index segment slice to which it is assigned in parallel.

[0164] Step S1207: After each loop, clear or mark the calculation result of the current loop in the tensor kernel cache as overwriteable, and enter the next loop until all index segment fragments are processed.

[0165] For example, the second segmentation granularity can be used to represent the number of vocabulary indices that can be calculated and output in each loop.

[0166] For example, for the second splitting granularity, i.e., the total number of bytes of data blocks output in each loop, it needs to be less than or equal to the capacity of the tensor kernel cache.

[0167] For example, the calculation method for the second segmentation granularity can be expressed as:

[0168] The second segmentation granularity = C_buffer / (D * sizeof(gradient data type)).

[0169] Here, C_buffer can represent the capacity of the tensor kernel buffer, and D can represent the embedding dimension.

[0170] For example, taking a gradient data type tensor of type BF16, C_buffer of 4MB, and D=512 as an example, that is, C_buffer=4MB=4*1024*1024 bytes, sizeof(BF16)=2 bytes, then:

[0171] The second segmentation particle size = 4 * 1024 * 1024 / (512 * 2) = 4096.

[0172] For example, for the division of cycle rounds, the entire vocabulary can be divided into multiple processing blocks according to the second segmentation granularity.

[0173] For example, the calculation method for the number of cycles can be expressed as:

[0174] Cycle number = CEILING(V / second segmentation granularity).

[0175] Here, V can represent the vocabulary size, which is the total number of rows in the gradient matrix that needs to be updated.

[0176] For example, assuming V=12768, then the number of cycles = CEILING(12768 / 4096) = 4 cycles.

[0177] For step S1205, the data filtering stage identifies the set of subtasks that need to be processed in each loop.

[0178] For example, for the k-th iteration (e.g., k starts from 0), the current index range being processed can be: [k * second partition granularity, (k+1) * second partition granularity - 1].

[0179] For example, you can scan all index segments and select those index segments whose unique index values ​​fall within the current circular index range as valid index segments for the current round.

[0180] For example, in the first loop (k=0), the index range to be processed can be represented as [0, 4095]. Assuming that the unique indices of the index segment shards are 1024, 2048, and 5000, then the shards with indices 1024 and 2048 will be processed in this round, while the shard with index 5000 will be skipped.

[0181] For step S1206, for example, a thread bundle can be allocated to each valid index segment fragment. Each thread bundle traverses the tokens in its fragment, reads the gradient through the index mapping table, and accumulates the results within the thread bundle to obtain the calculation results.

[0182] For example, the computation results of all valid slices are stored in a designated area of ​​the tensor kernel cache.

[0183] For example, the specific implementation method and beneficial effects of step S1206 can be referred to the relevant descriptions of steps S1201 to S1203 in the foregoing embodiments, which will not be repeated here.

[0184] For example, in step S1207, after all valid slices in a loop have been computed, the corresponding computed result region in the tensor kernel cache can be marked as overwriteable, or it can be cleared directly. Then, the loop counter can be incremented to enter the next loop and process the slices for the next index range. This process can be repeated until all vocabulary index ranges have been processed.

[0185] In at least one embodiment of this disclosure, a block-based loop mechanism can process vocabulary gradient matrices much larger than the tensor kernel cache capacity, thereby overcoming hardware cache capacity limitations and supporting large-scale vocabulary training. Furthermore, each loop maximizes the utilization of the limited tensor kernel cache, reducing resource waste or the risk of overflow, and improving overall processing efficiency.

[0186] Figure 4 The illustration shows an application diagram of a computational method based on a tensor kernel neural network provided by at least one embodiment of the present disclosure.

[0187] like Figure 4 As shown, multiple thread bundles can be assigned to their respective index segment shards.

[0188] For example, partial 0 (the part of the word with index=0) corresponds to warp 0, partial 1 (the other part of the word with index=0) corresponds to warp 1, partial 2 (the word with index=2) corresponds to warp 2, partial 3 (the other words in the index sequence between 0 and 2047) corresponds to warp 3, ... partial 256 (the word with index=2067) corresponds to warp 256 (not shown in the figure).

[0189] For example, taking 2048 as the single-round processing granularity, each round only processes "valid index segment fragments whose index sequence falls within the current 2048 range". Thus, in the first round of the loop, the index sequence range of 0~2047 is processed, that is, thread bundles (warps) are allocated to index segment fragments whose index sequence is within the range of 0~2047, gradient accumulation is completed and temporarily stored in the tensor kernel cache.

[0190] For example, a warp can traverse all tokens within its corresponding index segment (due to pre-sorting for deduplication, the index sequences within the same index segment are identical), find the position of each token in the original sequence through the index map, read the upstream gradient value (grad_token) of the token from high-bandwidth memory (e.g., HBM), and accumulate the gradient values ​​of all tokens within the index segment within the warp to obtain the "partial_sum" of that index segment.

[0191] For example, warp 0 processes partial 0: it accumulates the gradients of token4 and token30, resulting in "partial 0 token4 + token30";

[0192] For example, warp 1 processes partial 1: it accumulates the gradient of token100, resulting in "partial 1token100";

[0193] For example, warp 2 processes partial 2: it accumulates the gradient of token6, resulting in "partial 2 token6".

[0194] For example, the partial_sum of all thread bundles can be uniformly stored in the tensor kernel cache (i.e., tensor buffer for partial_sum) to reserve data for subsequent global accumulation operations of index segments.

[0195] In some embodiments of this disclosure, in step S130, multiple thread bundles are assigned to corresponding index segments respectively, and the calculation results belonging to the same index segment are accumulated to obtain the calculation summation of each index segment. This method may further include steps S1301 to S1303.

[0196] Step S1301: Assign a corresponding thread bundle to each index segment.

[0197] Step S1302: Each thread bundle retrieves the calculation results of the index segment fragments belonging to each index segment from the tensor kernel cache according to the index sequence of the corresponding index segment.

[0198] Step S1303: Accumulate the calculation results belonging to the same index segment to obtain the total calculation of each index segment.

[0199] For step S1301, task scheduling can be performed, for example, based on the list of index segments obtained in step S100 (e.g., index segments with unique indexes obtained after deduplication).

[0200] For example, a thread bundle equal to the number of index segments can be allocated.

[0201] For example, a mapping relationship between index segments and thread bundles can be established.

[0202] For example, the first thread bundle (Warp 0) can be assigned to the first index segment (Segment 0), the second thread bundle (Warp 1) can be assigned to the second index segment (Segment 1), and so on.

[0203] For step S1302, for example, each activated thread bundle can determine the unique index value of its corresponding index segment. The thread bundle can then search for and collect the calculation results of all index segment fragments belonging to that index segment from the tensor kernel cache based on the unique index value.

[0204] For example, suppose a thread bundle is assigned to an index segment with a unique index of 2. In the previous steps, this index segment was divided into multiple index segment fragments (e.g., 2 index segment fragments), and the calculation result of each index segment fragment has been stored in the tensor kernel cache. Then the thread bundle can read the calculation results of these two index segment fragments from the cache.

[0205] For step S1303, for example, each thread bundle sums the calculation results of all index segment fragments belonging to the same index segment obtained in the aforementioned step S1302. This summation result is the calculation summation of that index segment, which can be used to represent the total gradient that needs to be updated for that unique index (vocabulary position) throughout the entire training batch. This summation operation does not require any atomic operations, maintaining efficient lock-free parallelism.

[0206] For example, each thread bundle can sum up its computations and output them to the location in the tensor kernel cache corresponding to that unique index.

[0207] In at least one embodiment of this disclosure, by employing a thread bundle allocation strategy on the index segment, the accumulation tasks of all slices of the same index are consolidated into a single execution unit, and the final accumulation is completed within the thread bundle, forming a closed loop of the lock-free computation pipeline and improving the efficiency of gradient aggregation. Furthermore, the thread bundle allocation in this stage is independent yet well-connected to the aforementioned S120 stage, ensuring a balanced allocation of computational tasks at different granularities. This allows GPGPU computing resources to be fully utilized throughout the backpropagation process, reducing and avoiding idle computing resources caused by uneven task allocation and improving the overall system throughput.

[0208] In some embodiments of this disclosure, step S130 may further include steps S1304 to S1307.

[0209] Step S1304: Based on the vocabulary size and the capacity of the tensor kernel cache, determine the second segmentation granularity and divide the loop rounds.

[0210] Step S1305: In each iteration, determine the index segments that belong to the current loop index sequence range.

[0211] Step S1306: Assign multiple thread bundles to multiple index segments within the current loop. Each thread bundle accumulates the calculation results of all index segment fragments corresponding to its assigned index segment to obtain the summation of the index sequences corresponding to each index segment.

[0212] Step S1307: After each loop, clear or mark the calculation results of the current loop in the tensor kernel cache as overwriteable, and enter the next loop until the calculation of all index segments is completed.

[0213] For step S1304, the specific method for determining the second segmentation granularity and dividing the cycle can be referred to, for example, the relevant description of step S1204 in the aforementioned embodiment, and will not be repeated here.

[0214] For step S1305, taking the second segmentation granularity = 4096 as an example, the cycle number is CEILING(V / 4096).

[0215] For example, for the k-th iteration, the current word index range is: [k * second segmentation granularity, (k+1) * second segmentation granularity - 1]. For example, iteration 0 (index 0~4095), iteration 1 (index 4096~8191), iteration 2 (index 8192~12287), iteration 3 (index 12288~12767), and so on.

[0216] For example, you can scan all the index segments and select those index segments whose unique index values ​​fall within the current circular index range as the valid index segments for the current round.

[0217] For step S1306, the corresponding thread bundle can be assigned only to each index segment that is determined to be "valid" in the current loop.

[0218] For example, each thread bundle can be configured to perform the following operations: based on the unique index value of its assigned index segment, collect the computation results of all index segment fragments belonging to that index segment from the tensor kernel cache, and finally sum these fragment results to obtain the computation summation of that index segment.

[0219] For example, the summation of all valid index segments can be stored in the tensor kernel cache in the storage area reserved for this loop.

[0220] For example, the specific implementation method and beneficial effects of step S1306 can be referred to the relevant descriptions of S1301 to S1303 in the foregoing embodiments, which will not be repeated here.

[0221] For step S1307, for example, after the merging calculation of all valid index segments in a loop is completed, the summation of the calculations in this round stored in the tensor kernel cache can be further processed (e.g., preparing to write back to high-bandwidth memory (e.g., HBM) in batches).

[0222] For example, regions in the cache related to the current loop can be cleared or marked as overwhelmable to free up space for the next loop. The loop counter can be incremented to process the next vocabulary index range, and the process from S1305 to S1307 can be repeated until all vocabulary index ranges have been processed.

[0223] In at least one embodiment of this disclosure, by decomposing the entire merging process into multiple loops, it is possible to process vocabularies of arbitrary size, without being limited by on-chip cache capacity. This echoes the block-splitting strategy provided in the preceding embodiments, forming a large-scale data processing pipeline from local accumulation to global merging. Loop control based on the second segmentation granularity ensures that even in the final merging output stage, tensor kernel cache overflow will not occur due to excessive data volume. Simultaneously, cyclical cache clearing provides a good memory environment for the next round of computation. In each loop, the pattern of corresponding index segments and thread bundles is maintained, and each thread bundle independently and without conflict completes the final accumulation of its corresponding index segment, making the entire merging process efficient and correct. Thus, it is possible to achieve smooth and efficient execution in a block-based manner under limited hardware resources, realizing full-process optimization for training ultra-large-scale embedding layers.

[0224] Figure 5 This illustration shows an application diagram of another computational method based on a tensor kernel neural network provided by at least one embodiment of the present disclosure.

[0225] like Figure 5 As shown, the thread bundles are arranged on the index segments, and each thread bundle is responsible for processing the corresponding index segment (e.g., warp 0→segment 0, warp 1→segment 1, warp 2→segment 2, ..., warp 64→segment64).

[0226] For example, each index segment corresponds to a unique index sequence in the vocabulary (e.g., segment 0 corresponds to index 0, segment 2 corresponds to index 2, and segment 64 corresponds to index 2067).

[0227] For example, each index segment includes all index segment fragments that were split during the preprocessing stage (e.g., segment 0 includes partials 0 and 1, both of which belong to the index 0 fragment), enabling the gradient merging of multiple index sequences to be executed in parallel (e.g., warp 0 merges the partial_sum of index 0, and warp 1 merges the partial_sum of index 1).

[0228] For example, in the sum and scatter phase, that is, during the global merging of partial_sums of the same index sequence, each thread bundle reads all partial_sums included in its corresponding index segment from the tensor kernel cache.

[0229] For example, warp 0 reads partial 0 (token4+token30) and partial 1 (token100) of segment 0. Both belong to index 0, so they need to be merged.

[0230] For example, warp 2 reads partial 2 (token6) from segment 2, which belongs to index 2, and there is only one partial_sum that does not need to be merged;

[0231] For example, the thread that processes index 2067 reads partial 256 (token1+token5) of segment 64, which belongs to index 2067, and directly uses it as the merged result of this index sequence.

[0232] For example, since all partial values ​​within the same index segment belong to the same index sequence, the thread bundle can directly accumulate within the thread bundle to obtain the final gradient corresponding to the index sequence.

[0233] For example, the final gradient of index 0 is the sum of partial 0 and the sum of partial 1; for example, the final gradient of index 2 is the sum of partial 2 (single partial gradients do not need to be accumulated).

[0234] For example, a thread bundle can "scatter" the merged final gradient to the corresponding index sequence position in the on-chip global memory buffer (grad weight for gmb).

[0235] For example, the gradient of index 0 is written to position 0 of grad weight for gmb; for example, the gradient of index 2 is written to position 2; for example, the gradient of index 2067 is written to the corresponding position (position 2067) in the range of 2048~4096.

[0236] For example, taking 2048 as the single-round processing granularity, the merged gradient is first temporarily stored in the on-chip global memory buffer (grad weight for gmb). This buffer is divided into 2048 granularities to store the index sequence gradients in the ranges of 0~2047 and 2048~4096 respectively.

[0237] For example, after all the gradients of the index sequence within a certain range (e.g., 0~2047) have been merged, the gradients of that range are continuously and batch-written into the grad_weight array (corresponding to the gradient storage area of ​​the vocabulary) in a high-bandwidth memory (e.g., HBM).

[0238] For example, gradients in the range of 0 to 2047 can be written to positions 0 to 2047 of grad_weight all at once, and gradients in the range of 2048 to 4096 can be written to positions 2048 to 4095 all at once.

[0239] In some embodiments of this disclosure, the computation method of the tensor kernel-based neural network may further include steps S140 to S160.

[0240] Step S140: Obtain the tokens from the high-bandwidth memory according to the index sequence.

[0241] Step S150: Read the computational summation of all index segments from the tensor kernel cache.

[0242] Step S160: Based on the index sequence, write the calculated sums from the tensor kernel cache into the high-bandwidth memory respectively.

[0243] In the backpropagation of the embedding layer, this step S140 can correspond to obtaining the gradient value corresponding to each lexical position, which will be accumulated back to the corresponding position in the vocabulary according to the index sequence.

[0244] For example, at the start of the backpropagation process, the required input data can be read from a high-bandwidth memory (such as HBM).

[0245] For example, the data read may include gradient tensors (grad_output) propagated back from upstream or the original index sequence (index), etc. The embodiments of this disclosure are not limited in this regard. This read data can be loaded into the memory of the AI ​​processor to provide input for subsequent operations such as sorting, splitting and accumulation calculation.

[0246] For step S150, for example, after step S130 is completed, the summation of all index segments (i.e., the final gradient value of each unique index) has been stored in the tensor kernel cache. These calculation results can be read from the tensor kernel cache to prepare for the final output operation. For example, these calculation results are usually arranged consecutively in the cache according to the processed index order, which facilitates batch write-back to high-bandwidth memory (e.g., HBM).

[0247] For step S160, for example, the computational sums in the tensor kernel cache can be written to the corresponding position in the vocabulary gradient matrix (grad_weight) in memory. For example, the writing position can be determined by the unique index value corresponding to each computational sum, thereby realizing the transformation of the gradient from the temporary computation result to the stored result.

[0248] In some embodiments of this disclosure, step S160 may further include writing the calculated sums sequentially into the high-bandwidth memory at the vocabulary storage locations corresponding to each index sequence, according to the order of the index sequence.

[0249] Since the index sequence was sorted in the aforementioned step S100, the vocabulary indexes that need to be updated are arranged continuously or locally continuously in the tensor kernel cache. This characteristic can be used to write the calculation summation into a high-bandwidth memory (e.g., HBM) in batches according to the order of the index values ​​in a continuous memory access mode.

[0250] For example, it can be determined that multiple computations stored contiguously in the tensor kernel cache can be summed and then written in batches to corresponding contiguous regions in a high-bandwidth memory (such as HBM) via a single direct memory access (DMA) operation or a series of store instructions. This contiguous write mode can fully utilize the high bandwidth of HBM compared to random scatter writes.

[0251] In at least one embodiment of this disclosure, for example, input is read from a high-bandwidth memory (e.g., HBM), efficient computation is performed in a tensor kernel cache, and finally, the data is written back to the high-bandwidth memory (e.g., HBM) in an optimized manner, forming a complete data closed loop. This end-to-end optimization, from data loading, preprocessing, parallel computation to final output, enables efficient execution of the entire embedding layer backpropagation process. By transforming random scattering writes into continuous batch writes, the performance bottleneck of extremely low bandwidth utilization due to a large number of random write operations to high-bandwidth memory (e.g., HBM) in traditional methods is solved.

[0252] Figure 6 This illustration shows an application diagram of another computational method based on a tensor kernel neural network provided in at least one embodiment of the present disclosure.

[0253] like Figure 6 As shown, the computational method of this tensor kernel-based neural network can include the following five stages.

[0254] For example, the first stage is the input and index sequence preprocessing stage. For example, for the input data, Figure 6 The original index sequence [2,5,2,1,2,3,5,0] (each token corresponds to a word index, such as token0 corresponds to index2 and token1 corresponds to index5), and the embedding vector of each token (used for gradient calculation during backpropagation).

[0255] For example, for preprocessed output, unique index per segment can be used to indicate that the index sequence is deduplicated to obtain a unique set of indices (such as 0, 1, 2, 3, 5), with each index corresponding to an "index segment";

[0256] `index_nums_per_segment` can be used to represent the number of terms included in each index segment (unique index sequence) (e.g., ...). Figure 6 As shown, for example, index2 corresponds to token0, token2, and token4, a total of 3 tokens).

[0257] index_map can be used to represent the positional mapping between the sorted index sequence of records and the original word elements (e.g., ... Figure 6 As shown, for example, index2 corresponds to the original positions of token0, token2, and token4 after sorting, which is used for subsequent gradient lookup.

[0258] For example, the second stage is the index segment partitioning stage.

[0259] For example, for sharding logic, to adapt to the load balancing of warps, each index segment can be split into multiple index segment shards.

[0260] like Figure 6 As shown, for example, the index segment of index2 includes 3 words, which are split into 2 index segment fragments according to the splitting granularity of 2 (the first 2 words are partial0, and the 3rd word is partial1); the index segments of other index sequences are also split according to similar rules.

[0261] For example, for split output, `partials_per_segment` can be used to represent the number of index segment fragments included in each index segment (e.g., ...). Figure 6 As shown, for example, the index segment of index2 includes two index segment fragments.

[0262] index_nums_per_partials can be used to represent the number of tokens within each index segment partition;

[0263] index_id_per_partials can be used to represent the index sequence corresponding to each index segment shard.

[0264] For example, the Phase 3 thread bundle is placed on the index segment shard.

[0265] For example, independent thread bundles can be allocated to each index segment shard to perform gradient accumulation in parallel.

[0266] For example, token0 and token2 belong to partial0 of index2 and are processed by the same thread bundle. The gradients of the two are accumulated to obtain the partial gradient sum of the index segment of “token0 + token2” (partial_sum).

[0267] token4 belongs to partial1 of index2 and is processed by another thread. Its gradient is accumulated to obtain the partial gradient sum of the index segment of "token4" (partial_sum).

[0268] Other index segment shards (such as index0 and index1 index segment shards) are also processed in parallel by their respective thread bundles.

[0269] For example, all partial_sums can be temporarily stored in the tensor kernel cache to reserve data for subsequent merge operations.

[0270] For example, the stage 4 thread bundle is placed on the index segment.

[0271] For example, the index segments of each unique index sequence are processed by the corresponding thread bundle, which reads and accumulates the partial gradient sum (partial_sum) of all corresponding index segments.

[0272] like Figure 6 As shown, for example, the index segment of index2 includes two partial_sums ("token0+token2" and "token4"), which are accumulated by a thread bundle to obtain the final gradient of "token0+token2+token4";

[0273] The index segment of index5 consists of two index segment slice gradients (partial_sum) ("token1" and "token6"), which are accumulated by a thread bundle to obtain the final gradient of "token1 + token6".

[0274] For example, the same index segment can be processed by a single thread bundle, and the accumulated result continues to be temporarily stored in the tensor kernel cache.

[0275] For example, stage five outputs to a high-bandwidth memory (e.g., HBM) stage.

[0276] For example, once the gradients of all index segments have been merged, the final gradients in the tensor kernel cache can be written continuously to the vocabulary gradient storage area of ​​a high-bandwidth memory (such as HBM) all at once.

[0277] For example, the gradients of index0, index1, index2 (token0+token2+token4), index3, index5 (token1+token6), etc., are written back in batches.

[0278] Figure 7 A schematic block diagram of a computing device based on a tensor kernel neural network provided in at least one embodiment of the present disclosure is shown.

[0279] like Figure 7 As shown, the computing device 200 based on tensor kernel neural network includes a processing module 210, a segmentation module 220, a first computing module 230, and a second computing module 240.

[0280] The processing module 210 is configured to sort and deduplicate the index sequences corresponding to the tokens to obtain multiple index segments. For example, the processing module 210 can also be configured as step S100 in the foregoing embodiments.

[0281] The segmentation module 220 is configured to segment each of the multiple index segments into multiple index segment fragments according to a first segmentation granularity. For example, the segmentation module 220 can also be configured as step S110 in the foregoing embodiments.

[0282] The first calculation module 230 is configured to allocate multiple thread bundles to corresponding index segment slices, calculate the calculation result of each index segment slice in parallel through multiple thread bundles, and store the calculation result of each index segment slice in the tensor kernel cache. For example, the first calculation module 230 can also be configured as step S120 in the foregoing embodiment.

[0283] The second calculation module 240 is configured to allocate multiple thread bundles to their respective index segments, accumulate the calculation results belonging to the same index segment to obtain the calculation sum for each index segment, and write the calculation sum for each index segment to the tensor kernel cache. For example, the second calculation module 240 can also be configured as step S130 in the aforementioned embodiment.

[0284] In some embodiments of this disclosure, the processing module 210 may further be configured to sort the index sequence corresponding to the word element to obtain a sorted index sequence and an index mapping table; and to perform deduplication processing on the sorted index sequence. For example, the processing module 210 may also be configured to perform steps S101 to S102 in the foregoing embodiments.

[0285] In some embodiments of this disclosure, the processing module 210 may be further configured to perform deduplication processing on the sorted index sequence to obtain an index segment with a unique index sequence. For example, the processing module 210 may also be configured as step S1021 in the foregoing embodiments.

[0286] In some embodiments of this disclosure, the segmentation module 220 may be further configured to determine the number of index segment fragments corresponding to each index segment based on the first segmentation granularity and the index segments of the unique index sequence, and to determine the number of fragments for each index segment based on the number of fragments for each index segment, wherein the number of fragments for each index segment is used to determine the correspondence between multiple thread bundles and each index segment fragment. For example, the segmentation module 220 may also be configured as steps S111~S112 in the foregoing embodiments.

[0287] In some embodiments of this disclosure, the segmentation module 220 may be further configured to determine the number of index segment fragments corresponding to each index segment based on the number of index segments with unique index sequences and the first segmentation granularity.

[0288] In some embodiments of this disclosure, the first segmentation granularity is determined based on the number of index segment fragments corresponding to each index segment and the number of thread bundles.

[0289] In some embodiments of this disclosure, the first calculation module 230 may be further configured to allocate a corresponding thread bundle to each index segment; each thread bundle traverses all tokens in its corresponding index segment, looks up the gradient value corresponding to each token through the index mapping table; and accumulates the gradient values ​​corresponding to each token to obtain the calculation result for each index segment. For example, the first calculation module 230 may also be configured as steps S1201 to S1203 in the foregoing embodiments.

[0290] In some embodiments of this disclosure, the first calculation module 230 may be further configured to determine a second segmentation granularity and divide the loop rounds based on the vocabulary size and the capacity of the tensor kernel cache; in each loop round, identify the index segment fragments belonging to the current loop index sequence range; allocate multiple thread bundles to multiple valid index segment fragments in the current loop, and have each thread bundle calculate the calculation results of all words in its assigned index segment fragment in parallel; after each loop round, clear or mark the calculation results of the current loop in the tensor kernel cache as overwriteable, and enter the next loop round until all index segment fragments are processed. For example, the first calculation module 230 may also be configured as steps S1204 to S1207 in the foregoing embodiments.

[0291] In some embodiments of this disclosure, the second calculation module 240 may be further configured to allocate a corresponding thread bundle to each index segment; each thread bundle retrieves the calculation results of the index segment fragments belonging to each index segment from the tensor kernel cache according to the index sequence of the corresponding index segment; and accumulates the calculation results belonging to the same index segment to obtain the calculation summation of each index segment. For example, the second calculation module 240 may also be configured as steps S1301 to S1303 in the foregoing embodiments.

[0292] In some embodiments of this disclosure, the second calculation module 240 may be further configured to determine the second segmentation granularity and divide the loop rounds based on the vocabulary size and the capacity of the tensor kernel cache; in each loop round, the index segments belonging to the current loop index sequence range are determined; multiple thread bundles are respectively assigned to multiple index segments in the current loop, and the calculation results of all index segment fragments corresponding to the assigned index segment are accumulated by each thread bundle to obtain the calculation sum of the index sequence corresponding to each index segment; after each loop round, the calculation results of the current loop in the tensor kernel cache are cleared or marked as overwriteable, and the next loop round is entered until the calculation sum of all index segments is completed. For example, the second calculation module 240 may also be configured as steps S1304 to S1307 in the foregoing embodiments.

[0293] In some embodiments of this disclosure, the processing module 210 may be further configured to retrieve tokens from the high-bandwidth memory according to the index sequence; read the calculated sum of all index segments from the tensor kernel cache; and write the calculated sum from the tensor kernel cache into the high-bandwidth memory according to the index sequence. For example, the processing module 210 may also be configured to perform steps S140 to S160 in the foregoing embodiments.

[0294] In some embodiments of this disclosure, the processing module 210 may be further configured to continuously write the calculated sums into the vocabulary storage location corresponding to each index sequence in the high-bandwidth memory according to the order of the index sequence.

[0295] In some embodiments of this disclosure, the tensor kernel-based neural network computing device 200 can execute the tensor kernel-based neural network computing method provided in any embodiment of this disclosure during the backpropagation process of the embedded layer.

[0296] For example, in embodiments of this disclosure, the processing module 210, the segmentation module 220, the first computing module 230, and the second computing module 240 can be hardware, software, firmware, or any feasible combination thereof. For example, the processing module 210, the segmentation module 220, the first computing module 230, and the second computing module 240 can be dedicated or general-purpose circuits, chips, or devices, or they can be a combination of a processor and a memory. The embodiments of this disclosure do not limit the specific implementation of the above modules.

[0297] It should be noted that, in the embodiments of this disclosure, each module of the computing device 200 based on a tensor kernel-based neural network can correspond to each step of the computing method based on a tensor kernel-based neural network provided in this disclosure. Figure 7 The components and structures of the tensor kernel-based neural network computing device 200 shown are merely exemplary and not limiting. The tensor kernel-based neural network computing device 200 may also include other components and structures as needed.

[0298] It should be noted that, for details regarding the functions or beneficial effects of the various modules of the aforementioned tensor kernel-based neural network computing device 200, please refer to the descriptions of the computational methods of the tensor kernel-based neural network in the foregoing embodiments; these details will not be repeated here.

[0299] Figure 8 A schematic block diagram of an electronic device provided in at least one embodiment of the present disclosure is shown.

[0300] At least some embodiments of this disclosure also provide an electronic device, such as Figure 8 As shown, the first electronic device 300 includes a computing device based on a tensor kernel-based neural network provided in any embodiment of this disclosure. For example, the computing device based on a tensor kernel-based neural network can be implemented as a sender and / or receiver of instructions / data, and can be used, for example, in memory or external storage.

[0301] For example, the first processor 301 may be a central processing unit (CPU), an AI processor (such as a general-purpose graphics processing unit (GPGPU)), or other form of processing unit with data processing capabilities and / or program execution capabilities; for example, the central processing unit (CPU) may be a RISC, x86, or ARM architecture, etc.

[0302] The first electronic device 300 in the embodiments of this disclosure may include mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), and any device such as digital TVs, desktop computers, servers, etc. It may also be any combination of data processing devices and hardware, and the embodiments of this disclosure do not limit this.

[0303] Figure 8 The first electronic device 300 shown is an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0304] For example, such as Figure 8 As shown, in some examples, the first processor 301 can perform various appropriate actions and processes based on a program stored in the read-only memory (ROM) 302 or a program loaded from the storage device 308 into the random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the computer system. The first processor 301, ROM 302, and RAM 303 are connected via a communication channel 304. An input / output (I / O) interface 305 is also connected to the communication channel 304. For example, the memory controller for the RAM 303 includes the aforementioned computing device based on a tensor kernel neural network.

[0305] For example, the following components can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc., whose storage controllers include the aforementioned computing devices based on tensor kernel neural networks; and communication devices 309, such as LAN cards, modems, etc. Communication device 309 allows the first electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data and perform communication processing via networks such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage device 308 as needed. Although Figure 8 A first electronic device 300 including various devices is shown; however, it should be understood that implementation or inclusion of all shown devices is not required. More or fewer devices may be implemented or included alternatively.

[0306] For example, the first electronic device 300 may further include a peripheral interface (not shown in the figure). This peripheral interface can be various types of interfaces, such as a USB interface, a Lightning interface, etc. The communication device 309 can communicate wirelessly with a network and other devices, such as the Internet, an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN). Wireless communication can use any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0307] It should be noted that, in the embodiments of this disclosure, the specific functions and technical effects of the first electronic device 300 can be referred to, for example, the relevant descriptions of the calculation method and apparatus based on tensor kernel neural networks in the embodiments of this disclosure, and will not be repeated here.

[0308] Figure 9 A schematic block diagram of another electronic device provided in at least one embodiment of the present disclosure is shown.

[0309] At least one embodiment of this disclosure also provides an electronic device, such as Figure 9 As shown, the second electronic device 400 includes at least one second processor 410 and at least one memory 420.

[0310] For example, memory 420 can be used to non-transitory store computer-readable instructions (e.g., one or more computer program modules). Second processor 410 can be used to execute these computer-readable instructions, which, when executed by second processor 410, can perform one or more steps of the computational method based on tensor kernel neural networks described above. Memory 420 and second processor 410 can be interconnected via a bus or link, using wired, wireless, and / or other forms of communication media, as embodiments of this disclosure do not limit.

[0311] For example, the second processor 410 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), or other form of processing unit with data processing capabilities and / or program execution capabilities. For example, the central processing unit (CPU) may be based on RISC, x86, or ARM architectures. The second processor 410 may be a general-purpose processor or a dedicated processor, capable of controlling other components in the second electronic device 400 to perform desired functions.

[0312] For example, memory 420 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the second processor 410 may run one or more computer program modules to implement various functions of the second electronic device 400. Various application programs and various data, as well as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.

[0313] At least one embodiment of this disclosure also provides a non-transitory computer-readable storage medium for non-transitory storage of computer-readable instructions that, when executed by a computer, can implement the computation method of the tensor kernel-based neural network provided in any of the above embodiments.

[0314] Figure 10 This is a schematic diagram of a computer-readable storage medium provided for some embodiments of this disclosure.

[0315] like Figure 10 As shown, the computer-readable storage medium 500 is used to store computer-readable instructions 510. For example, when the computer-readable instructions 510 are executed by a computer, one or more steps in the computation method based on a tensor kernel-based neural network described above can be performed.

[0316] For example, the computer-readable storage medium 500 can be used in the first electronic device 300 or the second electronic device 400 described above. For example, further details regarding the non-volatile computer-readable storage medium 500 can be found in [reference needed]. Figure 8 The storage device 308 in the first electronic device 300 shown and Figure 9The corresponding description of the memory 420 in the second electronic device 400 shown will not be repeated here.

[0317] It should be noted that, in the embodiments of this disclosure, the specific functions and technical effects of the computer-readable storage medium 500 can be referred to, for example, the relevant descriptions of the calculation method and apparatus based on tensor kernel neural networks above, and will not be repeated here.

[0318] The following points need to be explained:

[0319] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0320] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0321] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

Claims

1. A method of computing a neural network based on a tensor core, characterized by, The computing method of the tensor core-based neural network comprises: performing sorting processing and deduplication processing on the index sequence corresponding to the word units to obtain a plurality of index segments; according to a first splitting granularity, splitting each of the plurality of index segments into a plurality of index segment shards; allocating a plurality of thread bundles to the corresponding index segment shards respectively, calculating the calculation results of each index segment shard in parallel by the plurality of thread bundles, and storing the calculation results of each index segment shard in a tensor core cache of the tensor core; and allocating a plurality of thread bundles to the corresponding index segments respectively, accumulating the calculation results belonging to the same index segment to obtain the calculation total of each index segment, and writing the calculation total of each index segment into the tensor core cache.

2. The method of claim 1, wherein, The sorting processing and deduplication processing on the index sequence corresponding to the word units comprise: performing the sorting processing on the index sequence corresponding to the word units to obtain a sorted index sequence and an index mapping table; and performing deduplication processing on the sorted index sequence.

3. The method of claim 2, wherein, The deduplication processing on the sorted index sequence comprises: performing the deduplication processing on the sorted index sequence to obtain an index segment with a unique index sequence.

4. The method of claim 3, wherein, According to the first splitting granularity and the index segment with a unique index sequence, the number of index segment shards corresponding to each index segment is determined, and according to the number of index segment shards corresponding to each index segment, the number of index segment shards is determined, wherein the number of index segment shards is used to determine the correspondence between the plurality of thread bundles and the index segment shards. According to the number of index segments with a unique index sequence and the first splitting granularity, the number of index segment shards corresponding to each index segment is determined.

5. The method of claim 4, wherein, The first splitting granularity is determined based on the number of index segment shards corresponding to each index segment and the number of thread bundles. The allocation of the plurality of thread bundles to the corresponding index segment shards respectively, the calculation of the calculation results of each index segment shard in parallel by the plurality of thread bundles comprises:

6. The method of claim 4, wherein, allocating a corresponding thread bundle to each index segment shard; 7. The method of claim 1, wherein, each thread bundle traverses all word units in the corresponding index segment shard, finds the gradient value corresponding to each word unit through the index mapping table; and accumulating the gradient value corresponding to each word unit to obtain the calculation result of each index segment shard. The computing method of the tensor core-based neural network further comprises: determining a second splitting granularity and dividing a cycle round based on the vocabulary size and the capacity of the tensor core cache; 8. The method of claim 7, wherein, in each cycle, determining the index segment shards belonging to the index sequence range of the current cycle; allocating a plurality of thread bundles to a plurality of effective index segment shards in the current cycle respectively, calculating the calculation results of all word units in the index segment shard allocated by each thread bundle in parallel; and after each cycle, clearing or marking the calculation results of the current cycle in the tensor core cache as coverable, entering the next cycle, until all index segment shards are processed. ​ ​ 9. The method of claim 1, wherein, The method comprises the following steps: allocating a corresponding thread bundle to each index segment; each thread bundle obtains the calculation results of the index segment fragments belonging to the corresponding index segment from the tensor core cache according to the index sequence of the corresponding index segment; and accumulating the calculation results belonging to the same index segment to obtain the calculation summary of each index segment.

10. The method of claim 9, wherein, The method further comprises the following steps: determining a second split granularity and dividing the loop rounds based on the vocabulary size and the capacity of the tensor core cache; in each loop round, determining the index segments belonging to the current loop index sequence range; allocating a plurality of thread bundles to the plurality of index segments in the current loop, and accumulating the calculation results of all index segment fragments corresponding to the index segments allocated by each thread bundle to obtain the calculation summary of the index sequence corresponding to each index segment; and after each loop round, clearing or marking the calculation results of the current loop in the tensor core cache as overwritable, and entering the next loop round until the calculation summary of all index segments is completed.

11. The method of claim 1, wherein, The method further comprises the following steps: obtaining the vocabulary items from the high-width-band memory according to the index sequence; reading the calculation summaries of all index segments from the tensor core cache; and writing the calculation summaries from the tensor core cache into the high-width-band memory according to the index sequence.

12. The method of claim 11, wherein, The method of writing the calculation summaries into the high-width-band memory according to the index sequence comprises the following steps: writing the calculation summaries into the high-width-band memory in the order of the index sequence and at the vocabulary storage positions corresponding to the index sequence.

13. The method of claim 1-12, wherein, The method is performed in the back propagation process of the embedding layer.

14. A computing device of a neural network based on a tensor core, characterized by, The device comprises: a processing module configured to sort and de-duplicate the index sequences corresponding to the vocabulary items to obtain a plurality of index segments; a split module configured to split each index segment in the plurality of index segments into a plurality of index segment fragments according to a first split granularity; a first calculation module configured to allocate a plurality of thread bundles to the corresponding index segment fragments, calculate the calculation results of each index segment fragment in parallel through the plurality of thread bundles, and store the calculation results of each index segment fragment in the tensor core cache of the tensor core; and a second calculation module configured to allocate a plurality of thread bundles to the corresponding index segments, accumulate the calculation results belonging to the same index segment to obtain the calculation summary of each index segment, and write the calculation summary of each index segment into the tensor core cache.

15. An electronic device, comprising: The electronic device comprises the device for calculating the neural network based on the tensor core according to claim 14.

16. An electronic device, comprising: The electronic device comprises: a processor; and a memory, wherein the memory stores at least one computer program which, when executed by the processor, implements the method for calculating the neural network based on the tensor core according to any one of claims 1-13.

17. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium is configured to non-transiently store computer readable instructions, when the computer readable instructions are executed by a computer, a computing method of the tensor core based neural network in any one of claims 1-13 is implemented.

Images