Data processing method and device, equipment, storage medium and program product

Abstract

The application provides a data processing method and device, equipment, a storage medium and a program product. The method comprises: obtaining a first logical view of an input first tensor, wherein a first logical dimension arrangement of the first logical view comprises a batch dimension, a sequence dimension and a hidden state dimension in sequence; performing first rearrangement of the first logical view based on a second logical dimension arrangement to obtain a second logical view of the first tensor, wherein the second logical dimension arrangement comprises the sequence dimension and a merged dimension in sequence, and the merged dimension is a product of the batch dimension and the hidden state dimension; and performing set communication on the first tensor in the sequence dimension based on the second logical view. The application can reduce the memory usage of data processing in a computer cluster, improve the speed of data processing, and improve the performance of the computer cluster.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] In distributed training of large language models, data processing systems widely employ sequence parallelism to support extremely long texts and accelerate computation. This requires ensemble communication data processing, such as All-Gather or Reduce-Scatter, along the sequence dimension of the input tensor across multiple computing devices to collect or distribute segmented data, thereby ensuring the correctness of training and the efficient collaboration of cluster computing power.

[0003] In related technologies, AI chips often use BSH tensor arrangement to optimize matrix multiplication. Since mainstream communication libraries only support the 0 axis by default, the tensor needs to be transposed before communication, swapping the 0 and 1 axes (BSH to SBH). This transposition causes non-contiguous memory access, leading to increased bandwidth consumption, higher communication latency, and insufficient utilization of GPU memory, thus reducing the performance of the computing cluster. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, storage medium, and program product that can reduce memory usage in data processing within a computer cluster, increase data processing speed, and improve the performance of the computing cluster.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a data processing method, the method comprising:

[0007] Obtain a first logical view of the first input tensor, wherein the first logical dimensions of the first logical view are arranged in sequence as batch dimension, sequence dimension and hidden state dimension;

[0008] The first logical view is rearranged based on the second logical dimension to obtain the second logical view of the first tensor. The second logical dimension arrangement includes the sequence dimension and the merge dimension in sequence. The merge dimension is the product of the batch dimension and the hidden state dimension.

[0009] Based on the second logical view, set communication is performed on the first tensor in the sequence dimension.

[0010] This application provides a data processing apparatus, including:

[0011] The acquisition module is used to acquire the first logical view of the input first tensor, wherein the first logical dimension arrangement of the first logical view includes the batch dimension, the sequence dimension and the hidden state dimension in sequence.

[0012] The rearrangement module is used to perform a first rearrangement based on the second logical dimension on the first logical view to obtain a second logical view of the first tensor, wherein the second logical dimension arrangement includes a sequence dimension and a merge dimension in sequence, and the merge dimension is the product of the batch dimension and the hidden state dimension.

[0013] A communication module is used to perform set communication on the first tensor in the sequence dimension based on the second logical view.

[0014] This application provides an electronic device, the electronic device comprising:

[0015] Memory is used to store executable instructions or computer programs.

[0016] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data processing method provided in the embodiments of this application.

[0017] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data processing method provided in this application when executed by a processor.

[0018] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the data processing method provided in this application.

[0019] The embodiments of this application have the following beneficial effects:

[0020] By acquiring a first logical view arranged sequentially as batch, sequence, and hidden state dimensions, and rearranging it into a second logical view that sequentially includes a sequence dimension and a merged dimension (the product of the batch and hidden state dimensions), the sequence dimension to be communicated is placed in the first dimension (first axis) at the logical level. This rearrangement at the logical level eliminates the process of explicitly transposing the three-dimensional tensor from BSH to SBH in actual memory to meet the first axis communication requirements in related technologies, thereby reducing the problems of non-contiguous memory access, high-frequency cache misses, and insufficient video memory utilization caused by three-dimensional transposition. In addition, based on the second logical view, set communication is performed on the first tensor in the sequence dimension. Since the rearranged view conforms to the operation rules of the first axis in mainstream communication libraries, and the merged dimension ensures the continuity and alignment of data blocks, no additional data padding is required. The data processing method of this application reduces the memory bandwidth overhead and end-to-end latency of data reorganization, and achieves highly efficient set communication with extremely low loss in sequence parallel scenarios without destroying the underlying data scale. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the data processing system architecture provided in the embodiments of this application;

[0022] Figure 2 This is a schematic diagram of the structure of the AI ​​accelerator provided in the embodiments of this application;

[0023] Figure 3 This is a first flowchart illustrating the data processing method provided in an embodiment of this application;

[0024] Figure 4 This is a second flowchart illustrating the data processing method provided in the embodiments of this application;

[0025] Figure 5 This is a schematic diagram of the third process of the data processing method provided in the embodiments of this application;

[0026] Figure 6 This is a schematic diagram of the fourth process of the data processing method provided in the embodiments of this application;

[0027] Figure 7 This is a schematic diagram of the fifth process of the data processing method provided in the embodiments of this application;

[0028] Figure 8 This is a sixth flowchart illustrating the data processing method provided in the embodiments of this application;

[0029] Figure 9 This is a schematic diagram of the seventh process of the data processing method provided in the embodiments of this application;

[0030] Figure 10This is the eighth flowchart of the data processing method provided in the embodiments of this application;

[0031] Figure 11 This is a ninth flowchart illustrating the data processing method provided in the embodiments of this application;

[0032] Figure 12 This is a schematic diagram of the tenth process of the data processing method provided in the embodiments of this application;

[0033] Figure 13 This is a schematic diagram of the physical arrangement structure of the original tensors provided in the embodiments of this application;

[0034] Figure 14 This is a schematic diagram of the memory layout of the first logical view provided in the embodiments of this application;

[0035] Figure 15 This is a schematic diagram of the memory layout of the second logical view provided in the embodiments of this application.

[0036] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0040] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0041] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0042] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0043] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0044] 1) Memory (Global Memory / High Bandwidth Memory) refers to the physical storage component in a computer or AI accelerator used to store computational data, program instructions and intermediate model results. In this application, it usually refers to the global video memory (such as HBM) on the AI ​​chip board.

[0045] 2) Non-contiguous memory access refers to the situation where the physical memory addresses accessed when reading or writing multidimensional tensor data are not sequentially connected in space. This access mode usually breaks the hardware's memory access rules and seriously reduces the bandwidth utilization of the memory bus.

[0046] 3) Cache miss refers to the phenomenon where the processor or computing core attempts to read the required data from the on-chip cache, but finds that the target data does not reside in the cache, and thus has to load the data from the slower main memory (global memory).

[0047] 4) Operator / Kernel refers to pre-compiled program code or basic parallel computing unit in a deep learning framework or underlying computing device used to perform specific mathematical operations or data processing logic (such as matrix multiplication, tensor rearrangement).

[0048] 5) Logical View (or Tensor View) refers to the abstract interpretation and description of multidimensional data (tensors) stored in physical contiguous memory at the level of logical structure such as dimension, shape, and step size. It determines how the program parses the multidimensional coordinates of the data without changing the actual physical storage location of the data in memory.

[0049] 6) Temporary Communication Buffer refers to a physically contiguous storage area specifically allocated in global video memory. It is mainly used to temporarily store data blocks that have been organized according to specific communication protocols and arrangement rules before performing multi-node data interaction operations.

[0050] 7) Collective Communication refers to the global data interaction and synchronization operations that multiple computing nodes or processes in a distributed computing cluster participate in according to a specific communication protocol, such as All-Gather or Reduce-Scatter commonly used in large model training.

[0051] 8) Stride refers to the number of elements or address span required to move along a specific dimension to the next adjacent element in a multidimensional tensor data structure.

[0052] 9) Sequence Parallelism refers to a parallel computing technique used in distributed deep learning model training to divide extremely long input text or data along the sequence dimension and distribute them to multiple computing devices in the cluster for parallel processing, thereby reducing the memory usage of a single device and overcoming the bottleneck of long text training.

[0053] 10) Memory offset refers to the one-dimensional linear distance or index difference between the precise storage location of a data element in a multidimensional tensor and the starting address (base address) of the target memory region when it is mapped, rearranged, or copied to a target contiguous physical memory block.

[0054] 11) Memory Coalescing refers to the underlying hardware memory controller's optimization mechanism in high-concurrency computing architectures such as GPUs or NPUs, where multiple parallel threads simultaneously request access to contiguous physical memory addresses. This mechanism packages and merges these discrete read and write requests into a single or very few second-wide bus memory transactions, thereby improving the utilization of video memory bandwidth.

[0055] 12) Index Mapping Relationship refers to the data location correspondence rule established between the source logical dimension arrangement and the target logical dimension arrangement of a tensor. It is essentially a set of mathematical addressing formulas used to guide the underlying operators to accurately "project" or write back the source elements to the target physical memory address without generating a complete intermediate transpose tensor.

[0056] In related technologies, in distributed deep learning (especially large language model) training scenarios, to optimize matrix multiplication (GEMM) performance and ensure efficient contiguous memory access, hardware devices typically use a BSH (Batch-Sequence-Hidden) format to organize input tensors, which sequentially includes batch, sequence, and hidden state dimensions. When sequence parallelism is enabled to process long texts, all-gather or reduce-scatter operations need to be performed along the sequence dimension of the tensor. Since the underlying set communication libraries usually only support communication operations on the 0th axis (first dimension) of the tensor by default, a common data processing method is to explicitly perform a full transpose operation on the three-dimensional tensor to be communicated. That is, the physical positions of the 0th axis (batch dimension) and the 1st axis (sequence dimension) of the tensor are swapped in memory to generate a complete intermediate transposed tensor with SBH (Sequence-Batch-Hidden) arrangement, which is then stored in memory. Subsequently, this intermediate transposed tensor in memory is called for set communication.

[0057] However, the explicit transposition of the 3D tensor described above has the following problems: First, generating and storing the complete intermediate transposed tensor requires additional contiguous memory allocation, leading to insufficient utilization of the data processing system's GPU memory. Second, the complete transposition operation between BSH and SBH disrupts the physical continuity of the original data, resulting in a large number of non-contiguous memory accesses and cache misses, thus increasing memory bandwidth consumption. Especially when the batch size is large, the kernel execution time and memory copying amount caused by this transposition operation increase linearly, leading to increased end-to-end communication latency. This not only reduces the communication efficiency of sequence parallelism but also negates the computing power advantage of the underlying AI accelerator under the BSH deployment.

[0058] As an example, the operation of generating and saving a complete intermediate transposed tensor in related technologies can be performed as follows: First, obtain the original tensor according to the physical arrangement of BSH (corresponding to the first tensor in the embodiment of this application). It should be noted that in the code, the tensor appears to be three-dimensional (Batch, Sequence, Hidden), hereinafter referred to as BSH, but in physical memory (such as the HBM memory of the NPU), the storage space is a one-dimensional straight line. Therefore, the elements with adjacent physical addresses of the original tensor are [b0_s0_h0], [b0_s0_h1], ..., [b0_s1_h0]..., while the physical arrangement of the intermediate transposed tensor to be generated is (Sequence, Batch, Hidden), hereinafter referred to as SBH, that is, the elements with adjacent physical addresses of the intermediate transposed tensor are [s0_b0_h0], [s0_b0_h1], ..., [s0_b1_h0].... Normally, a complete intermediate transposed tensor can be generated in another region of physical memory by calling the command `tensor.transpose(0, 1).contiguous()`. However, in order to write the original tensor in the contiguous format of SBH, the program needs to read the original contiguous BSH memory in the logical order of SBH. For example, first read s0_b0_h0 (offset 0 in BSH), then read s0_b1_h0 (offset increased by S*H positions in BSH), and then read s0_b2_h0 (offset increased by S*H positions again). However, hardware memory is not read "individual elements" but rather in "cache lines" (usually 64 bytes or 128 bytes) into the cache. When the program reads s0_b0_h0, the hardware memory automatically moves the adjacent elements s0_b0_h1 to s0_b0_hn into the cache line. However, according to the transpose operation order, the program needs s0_b1_h0 next, which is not included in the cache line just loaded. Therefore, the hardware needs to clear the data already read into the cache line and search for s0_b1_h0 again from the slow global memory. When the batch or sequence dimension is large, this kind of jump read with a large span is frequently triggered, resulting in a decrease in cache hit rate and a large amount of memory bandwidth being consumed in moving useless data. At the same time, an extra space must be allocated in global memory to write down the complete SBH tensor, creating dual pressure on video memory read and write.

[0059] Based on the problems existing in related technologies, embodiments of this application provide a data processing method, apparatus, device, storage medium, and program product, which can reduce memory usage in data processing within a computer cluster, improve data processing speed, and enhance the performance of the computing cluster. The following describes exemplary applications of the data processing device provided in this application embodiment. The electronic device provided in this application embodiment can be implemented as a computing cluster, a graphics processor, a tensor processor, etc. Exemplary applications when the device is implemented as a computing cluster will be described below.

[0060] See Figure 1 , Figure 1 This is a schematic diagram of the node architecture of a computing cluster provided in an embodiment of this application. In order to support a data processing application, a data processing application can be provided, such as a data processing application running on a supernode in the computing cluster.

[0061] See Figure 1 The computing cluster may include multiple computing nodes 100, and each computing node 100 may include multiple AI accelerators 200. Each AI accelerator 200 is connected to each other via a high-speed serial bus (for simplicity, Figure 1 Only the connection structure of one AI accelerator 200 is shown; in reality, each AI accelerator 200 is connected to other AI accelerators 200. A high-speed serial bus refers to a point-to-point serial communication link with a transmission rate of 32GT / s or higher (such as the PCIe 5.0 standard). Inside the AI ​​accelerator 200, a scheduler is also included. Figure 1 (Not shown in the image), the scheduler is used to load pre-stored operators (pre-compiled code) and send the operators to the instruction cache or dedicated storage area of ​​the AI ​​accelerator to drive the computing core of the AI ​​accelerator 200 to perform read, memory offset calculation and output operations.

[0062] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the AI ​​accelerator 200 provided in the embodiments of this application. Figure 2 The AI ​​accelerator 200 shown includes at least one processor 210, a memory 250, and at least one network interface 220. The various components in the AI ​​accelerator 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0063] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0064] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices physically located away from the processor 210.

[0065] The memory 250 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 250 described in this application embodiment is intended to include any suitable type of memory.

[0066] In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0067] Operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0068] The input processing module 254 is used to detect user input or interaction and translate the detected input or interaction.

[0069] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A data processing device 255 stored in memory 250 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: an acquisition module 2551, a rearrangement module 2552, and a communication module 2553. These modules are logically linked and can therefore be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0070] See Figure 3 , Figure 3 This is a flowchart illustrating the data processing method provided in the embodiments of this application, which will be combined with... Figure 3 The steps shown are explained below. Figure 3The main body of the steps is the nodes. For example... Figure 3 As shown, the data processing method includes the following steps 101 to 103.

[0071] In step 101, the first logical view of the input first tensor is obtained.

[0072] The first logical dimension of the first logical view includes, in sequence, the batch dimension, the sequence dimension, and the hidden state dimension.

[0073] Here, a logical view refers to an abstract structure that interprets tensor data stored in physical memory in terms of dimensions. It defines the logical organization of multidimensional data (e.g., organization at the memory index level, organization at the operator processing level, etc.) without changing the actual storage order of the data in physical memory. In the embodiments of this application, this view is used to describe the arrangement characteristics of input data in the GPU memory of an AI accelerator in a Large Language Model (LLM) training scenario.

[0074] Here, the first tensor refers to the high-dimensional data object that needs to be transferred or processed between different computing nodes during the training of a deep learning model. It typically carries the activation values ​​or gradient information of intermediate layers in the neural network. The first logical dimension can be arranged in a BSH (Browser-Side Array) configuration. ,in, Indicates batch dimension, Indicates the sequence dimension. Represents the hidden state dimension.

[0075] In some embodiments, the first tensor can be obtained through the underlying memory manager of a deep learning framework such as PyTorch or MindSpore. This tensor is typically a contiguous block of memory data optimized for General Matrix Multiplication (GEMM) as the output of a preceding operator (such as an Embedding layer or Transformer layer). Its physical memory layout follows a contiguous flattening principle, meaning that each Hidden State vector is stored contiguously in memory address order.

[0076] In some embodiments, obtaining a first logical view of the input first tensor can be achieved by: reading the tensor's metadata information and determining its shape parameters. And the corresponding memory step size. Assume the batch size is... The sequence length is The hidden state dimension is Then any element in physical memory storage address offset It can be done through formula The calculations determine that 'b' represents the batch dimension index, 's' represents the sequence dimension index, and 'h' represents the hidden state dimension index. This arrangement ensures high memory access locality and computational unit utilization during subsequent linear projections or matrix operations.

[0077] In step 102, the first logical view is rearranged based on the second logical dimension to obtain the second logical view of the first tensor.

[0078] The second logical dimension includes the sequence dimension and the merge dimension, which is the product of the batch dimension and the hidden state dimension.

[0079] Here, the first arrangement refers to the process of transforming tensors into a layout suitable for a specific communication mode by changing the data indexing method or physical storage location without changing the total amount of physical data storage; the second logical view is specifically... The dimensional form is designed to place the sequence dimension in the highest dimension of the tensor (axis=0).

[0080] In some embodiments, see Figure 4 The first logical view in step 102 is rearranged based on the second logical dimension to obtain the second logical view of the first tensor, which can be achieved by executing the following steps 1021 to 1022.

[0081] In step 1021, a first index mapping relationship is constructed based on the first logical dimension arrangement and the second logical dimension arrangement of the first logical view.

[0082] Here, the first index mapping relationship refers to the mathematical model describing the logical correspondence between the positions of tensor elements in the original three-dimensional logical space (B, S, H) and the target two-dimensional logical space (S, M, where M is the merged dimension). It establishes the transformation rules for each data point in the memory address space from the "source offset" to the "target offset" when data undergoes dimensional transformation.

[0083] In some embodiments, the first and second logical dimension arrangements can be obtained by reading tensor metadata in the current deep learning computation graph, including batch size (B), sequence length (S), and hidden layer dimension (H). Based on these dimensional parameters, the data processing system pre-calculates the cross-dimensional stride information, providing parameter support for constructing the mapping relationship.

[0084] In some embodiments, constructing the first index mapping relationship can be achieved by: establishing a set of linear address calculation formulas, and mapping the three-dimensional index coordinates in the first logical view. A flattened physical index mapped to the second logical view. For example, suppose the original element position is... The logical position of the object in the target view is calculated through the mapping relationship. Here, b represents the index of the batch dimension, s represents the index of the sequence dimension, and h represents the index of the hidden state dimension. This mapping ensures that the internal continuity of the hidden state dimension (H) is preserved during the rearrangement process, thereby optimizing memory access performance.

[0085] In some embodiments, see Figure 5 The construction of the first index mapping relationship based on the first logical dimension arrangement and the second logical dimension arrangement of the first logical view in step 1021 can be achieved by executing the following steps 10211 to 10212.

[0086] In step 10211, based on the number of batches, the number of sequences, and the number of hidden states in the first logical dimension arrangement, the first memory offset for rearranging the first logical view into a second logical dimension arrangement is determined.

[0087] Here, the first memory offset refers to the one-dimensional linear distance or index value of a multidimensional logical element in a tensor relative to the starting address of the target contiguous physical memory (such as a send buffer for set communication) when the element is rearranged to that memory region.

[0088] In some embodiments, the number corresponding to the batch dimension (denoted as ) ), the number of sequence dimensions (denoted as ) and the number corresponding to the hidden state dimension (denoted as The shape attribute of the tensor passed to the deep learning framework can be obtained in the following ways: during the initialization phase of the communication operator, the tensor shape attribute can be parsed, or metadata can be extracted from the tensor descriptor output by the upstream operator.

[0089] In some embodiments, determining the first memory offset can be done based on a target logical view. This is achieved by arithmetic calculation of the dimensional span (Stride). For example, for coordinates in the original logical view... For any element, directly use the formula Calculate its first memory offset, where This process merges dimensions. It avoids generating complete intermediate transposed tensors (such as SBH arrangements) and ensures the hidden state dimensions are preserved after rearrangement by calculating the first memory offset and performing logical view rearrangement. The continuity is maintained. Furthermore, in this embodiment, data is directly mapped to the communication buffer via a first memory offset, thus preserving... The continuity of dimensions helps the hardware's memory coalescing mechanism copy data from memory, reducing memory bandwidth overhead and improving copying efficiency.

[0090] In step 10212, the first memory offset of each element in the first logical view is used as the first index mapping relationship.

[0091] Here, the first index mapping relationship refers to the global addressing rules that describe the physical arrangement of the entire input tensor from the source end (memory) to the communication arrangement of the destination end (temporary communication buffer).

[0092] In some embodiments, the first memory offset of each element in the first logical view is generated on-the-fly during the operation of the fusion operator, and the node does not need to allocate additional storage space in the video memory to maintain or store a static physical mapping table (second logical view).

[0093] In some embodiments, using the first memory offset as a mapping relationship can be achieved by directly embedding the offset calculation logic into the addressing instructions of the NPU multi-threaded kernel. For example, in a three-dimensional threaded grid, each computation thread derives the corresponding [database] based on its own global index. Then, the first memory offset is calculated by directly substituting it into the formula, and this is used as the unique target address pointer for writing data, guiding the element to complete a one-time cross-memory copy.

[0094] This application's embodiments directly use the memory offsets of each element as indexes for mapping relationships, i.e., implementing subsequent communication through on-the-fly arithmetic addressing rules. This method eliminates the need to allocate additional space in video memory to create, store, and maintain a large and redundant static physical mapping table (Look-Up Table), thereby reducing the video memory burden during the mapping relationship construction process.

[0095] In step 1022, the first logical view is rearranged based on the second logical dimension according to the first index mapping relationship to obtain the second logical view of the first tensor.

[0096] Here, the first rearrangement refers to the physical process of rearranging tensor elements stored in physical memory from their original linear order to an order that conforms to the target logical structure, according to a preset index mapping rule. This process is essentially a preparatory operation for subsequent efficient memory copying, aiming to achieve single-step communication to obtain physically contiguous data blocks in subsequent aggregate communication operations. This is related to the generation of complete... Unlike the intermediate transpose tensor method, the first arrangement in this step directly "projects" the data from the original arrangement format into a contiguous memory buffer specifically prepared for communication.

[0097] In some embodiments, the first rearrangement of the first logical view can be achieved by: utilizing the multi-threaded parallel computing capabilities of the NPU to launch a three-dimensional thread grid, whose grid dimensions correspond to... , and Each thread is responsible for processing a single element and performing a copy according to the mapping relationship constructed in step 1021. For example, if Then the coordinates are The element (original offset 4) will be passed through the first partial memory offset. The calculated target offset is 2. This multi-threaded concurrent copying method can trigger the underlying hardware memory request merging, thereby minimizing the performance loss of step access and achieving almost lossless pre-rearrangement.

[0098] This embodiment of the application directly maps data to the location of a temporary communication buffer based on a pre-built index mapping relationship, eliminating the step of pre-generating a complete transposed intermediate tensor. This direct mapping strategy avoids the additional allocation and reading / writing of complete intermediate data blocks in slow global video memory, reducing one complete video memory read / write cycle and thus alleviating video memory bandwidth pressure. Furthermore, even with an increased batch size, the time overhead of the rearrangement process remains low because it relies solely on address mapping without generating additional redundant copies. When constructing the first index mapping relationship, this embodiment specifically maps the batch dimension and hidden state dimension to a merged dimension, rather than simply swapping dimension positions. This mapping logic strictly preserves the continuity of elements within each hidden state vector at the physical level. When the hardware loads data, it ensures that consecutive data blocks in the cache line are fully utilized, improving the hit rate and access efficiency of the local cache.

[0099] In step 103, set communication is performed on the first tensor in the sequence dimension based on the second logical view.

[0100] Here, group communication refers to data exchange operations between a group of processes in distributed computing. In the embodiments of this application, group communication typically includes all-gather or reduce-scatter operations designed for sequence parallelism, used to synchronize sequence-dimensional partitioned data between different computing nodes or computing cores.

[0101] In some embodiments, the prerequisite for performing set communication is that the logical view rearrangement in step 102 has been completed. This is because the rearranged second logical view places the sequence dimension on axis 0 and merges dimensions. Since the memory is contiguous, the rearranged memory starting address, total data volume, and communication domain parameters can be directly passed to the underlying communication operator, directly meeting the alignment requirements of the underlying high-performance communication library for the data input format, without the need for additional data splicing or conversion.

[0102] In some embodiments, performing set communication at the sequence dimension can be achieved by calling communication primitives from high-performance communication libraries such as HCCL and NCCL.

[0103] This embodiment of the application rearranges the logical dimension views of the first tensor and performs set communication on the first tensor based on the rearranged second logical view. This eliminates the step of generating a complete intermediate transposed tensor, reduces video memory usage, and decreases the number of read / write operations to global video memory, thus reducing memory bandwidth pressure and improving processing speed. Furthermore, since the hidden state dimension of the original input is stored contiguously, merging the batch dimension and the hidden state dimension preserves the locality of the original data of the first tensor. In the case of multi-threaded concurrent copying, this arrangement of merged dimensions facilitates hardware-level memory merged access, allowing the copy (write) process from (B, S, H) to the temporary communication buffer to fully utilize the cache lines of AI accelerators (such as NPU / GPU), reducing cache misses caused by non-contiguous memory access. Further, performing set communication through the second logical view allows the sequence dimension to be placed on the 0th axis and the data blocks to be regularized. The contiguous memory segments allow this scheme to directly call standard, efficient collection communication interfaces (such as All-Gather / Reduce-Scatter) without requiring additional data padding or splitting. This ensures efficient utilization of communication bandwidth, enabling near-theoretical peak communication efficiency even in sequential parallel scenarios.

[0104] In some embodiments, see Figure 6 The set communication performed on the first tensor in the sequence dimension based on the second logical view in step 103 can be achieved by executing steps 1031 to 1034.

[0105] In step 1031, the original storage address of the first tensor under the first logical view is obtained, and the target storage address corresponding to the original storage address in the second logical view is obtained.

[0106] Here, the target storage address refers to the physical location of each data element of the first tensor within a pre-defined communication buffer, which is typically located in another memory location. Because the first logical view (BSH) and the second logical view (S, B×H) have different dimensional arrangements, the linear offset of the same data element differs between the two views. The target storage address determines the order in which the elements are arranged after the cross-memory copy.

[0107] In some embodiments, the original memory address can be calculated by obtaining the base pointer of the first tensor in video memory and combining it with the stride of the first logical view. The target memory address is a pointer position pre-calculated by the operator logic based on the contiguous memory layout required for communication before the fusion operator starts.

[0108] In some embodiments, obtaining the target storage address can be achieved by: based on the first index mapping relationship constructed in step 10212, assigning a logical coordinate to each logical coordinate... The global linear offset of each element in the second logical view is calculated. By adding this offset to the starting address of the communication buffer, the target storage address of each element to be moved can be locked in real time, thus providing an addressing basis for subsequent precise memory rewriting.

[0109] In step 1032, the element to be communicated is read from memory based on the original storage address, and the element to be communicated is written to a temporary communication buffer based on the target storage address.

[0110] Here, the element to be communicated refers to the smallest data unit that constitutes a tensor (such as a single-precision floating-point number Float32 or a half-precision floating-point number Float16). A temporary communication buffer is a separate, contiguous memory area in video memory used to store rearranged data for direct reading by the underlying communication library (such as HCCL, NCCL).

[0111] In some embodiments, read and write operations are performed as a single memory access instruction within a single kernel iteration. This non-in-place cross-memory copy method avoids logical conflicts that may result from directly modifying data in the original tensor storage area, while ensuring the integrity of the original data.

[0112] In some embodiments, cross-memory writes of elements to be communicated can be achieved by leveraging the NPU's memory bus concurrency capabilities to initiate multi-threaded parallel copy tasks. Each thread is responsible for reading one or more consecutive elements from the original memory address into a register or cache (L1 / L2 cache), and then writing them to a temporary communication buffer according to the target memory address. For example, by using the memory controller's (MC) Coalesced Writing technique, fragmented write requests can be combined into consecutive block writes, improving bandwidth utilization for cross-memory transfers.

[0113] In some embodiments, see Figure 7 The step 1032, which involves writing the element to be communicated into the temporary communication buffer based on the target storage address, can be achieved by executing the following steps 10321 to 10323.

[0114] In step 10321, based on the number of batches, the number of sequences, and the number of hidden states in the first logical dimension arrangement, the first memory offset for rearranging the first logical view into a second logical dimension arrangement is determined.

[0115] In some embodiments, determining the first memory offset can be achieved by the operator logic layer first parsing the dimension parameters of the input tensor. , and Then, using the stride calculation rule, each logical coordinate is... The elements are used to build a target location model. For example, through calculation formulas. The sequence dimension, which was originally in the middle dimension in the first logical view, is calculated as the highest dimension offset in the second logical view.

[0116] In step 10322, based on the first memory offset, the second physical address of the temporary communication buffer to be written is determined, corresponding to the first physical address where each element in the first tensor is located.

[0117] Here, the first physical address refers to the actual pointer in memory where the original tensor is stored, and the second physical address refers to the pointer in the temporary communication buffer where the element should be placed after being copied across memory.

[0118] In some embodiments, the second physical address can be determined by obtaining the base pointer of a pre-allocated temporary communication buffer and performing a pointer addition operation with the first memory offset obtained in step 10321. This process achieves the final conversion from "logical coordinates" to "physical addressing".

[0119] In some embodiments, the process of determining the second physical address actually establishes a set of "source-destination" address pairs. For contiguous or non-contiguous elements in the first physical address space, this address pair mapping precisely guides the data flow to a specific contiguous segment in the temporary communication buffer. This determination method ensures that even under complex dimensional transformations, data write operations strictly adhere to the physical layout protocol required by the collection communication library.

[0120] In step 10323, the elements in the first tensor are read in the order of the first physical address, and written to the second physical address in one step.

[0121] Here, single-step write refers to the process where data elements are read from the original memory region and directly written to the target physical address via registers or cache, without any intermediate global memory write-back operation involving transpose tensors. This is a typical execution mechanism of "one-time memory access to complete rearrangement".

[0122] In some embodiments, reading according to the order of the first physical address maximizes the utilization of the coalesced access characteristic of the video memory. Because the original tensor is in Physically contiguous in dimension, sequential reading ensures that the memory controller fills cache lines with the highest bandwidth efficiency, thereby reducing memory access latency.

[0123] In some embodiments, single-step writing to the second physical address can be implemented as follows: During NPU kernel execution, each concurrent thread loads a data block from its corresponding first physical address into an on-chip buffer and immediately executes a write-back (Store) instruction based on the calculated second physical address. In this way, the two global memory round trips that originally required "transposing the data into intermediate memory and then reading it from the intermediate memory for communication" are simplified to a single "read-write" process. This not only eliminates the GPU memory requirements of intermediate tensors but also reduces the overall memory access pressure, hiding the overhead of rearrangement operations within the inherent latency of data movement.

[0124] This application embodiment accurately derives the first memory offset based on batch, sequence, and hidden state dimensions, and constructs a direct mapping relationship from the source physical address to the target physical address. This enables the underlying hardware to efficiently read the original tensor elements in strict physical sequential order and directly place them into the temporary communication buffer in a "single-step write" manner. This progressive execution logic not only makes full use of the continuous memory access advantage brought by sequential reading (improving memory bus bandwidth utilization and cache hit rate), but also skips the redundant steps of generating complete intermediate transposed tensors in related technologies, simplifying the complex dimension reshaping process into a single read and write operation, thereby saving global video memory space overhead and reducing video memory bandwidth usage.

[0125] In some embodiments, step 10323 can be achieved by performing the following steps:

[0126] First, select multiple starting elements from the elements of the first tensor.

[0127] Here, the starting element refers to the initial task anchor point assigned to each independent thread or thread block for data processing in a parallel computing architecture (such as an NPU or GPU). By determining multiple starting elements, a large tensor data transfer task can be broken down into multiple subtasks that can be executed simultaneously, thereby fully activating the hardware's concurrent throughput capabilities.

[0128] In some embodiments, selecting multiple starting elements can be achieved by: based on the shape parameters of the input tensor ( , , A three-dimensional thread grid is constructed, and the global index corresponding to each thread in the grid is used as the basis for locating the starting element. For example, each thread is located based on its position in the grid. , or The offset in the dimension locks a specific starting position in the physical memory of the first tensor.

[0129] In some embodiments, the starting element can be selected as the hidden state dimension. The first element or the element at a specific step size. Since the embodiments of this application use BSH arrangement, selecting the starting position of the hidden state dimension as the starting element can ensure that each parallel unit can start execution from a relatively independent storage block, thereby avoiding logical overlap or redundant calculations of multiple threads during reading.

[0130] Then, multiple starting elements are used as the first elements for parallel reading. According to the order of the first physical address, multiple elements in the first tensor are read in parallel, and the second physical address corresponding to multiple elements is written step by step.

[0131] Here, parallel reading refers to a large number of threads simultaneously loading data from different physical locations in the video memory, while single-step writing refers to data being transferred to registers within the thread and then directly written to the communication buffer according to the target mapping address.

[0132] In some embodiments, parallel readings are performed according to the order of the first physical address, aiming to trigger the underlying hardware memory coalescing mechanism. This is because data after the starting element is in the hidden state dimension. The cache lines are physically contiguous, and when a thread reads, it loads an entire cache line at once. This sequential read mode can improve the utilization of the video memory bus and reduce the average memory access latency.

[0133] In some embodiments, single-step writing to the second physical address can be implemented as follows: each computation thread, after obtaining the set of elements it is responsible for (e.g., a set of elements of length 1), writes the second physical address in one step. After vector fragmentation, the write (Store) instruction is executed directly using a determined second physical address. Since the write target for each thread is unique and deterministic in the communication buffer, the entire parallel process does not require locking or atomic operations. This mechanism of "multi-point concurrent reading and directional single-step placement" enables dimensional transposition and rearrangement to be completed simultaneously with data transport, eliminating the performance bottlenecks caused by serial processing or multiple memory round trips in related technologies, and ensuring efficient execution of sequential parallel communication under the BSH layout.

[0134] This application embodiment selects multiple starting elements and initiates parallel read and single-step write operations based on these elements, distributing the data transfer task to the underlying multi-threaded concurrency mechanism (such as a three-dimensional thread grid). This decentralized design enables multiple computing threads to independently complete constant-level (O(1)) data reading and single-step placement without dependence or blocking, eliminating cross-thread synchronization overhead. At the same time, the concurrent read and write requests initiated by multiple threads based on physical address order can trigger the memory coalescing mechanism of the hardware memory controller, which packages the discrete memory access requests originally caused by dimensional transformation at the underlying level, thereby effectively masking the latency penalty caused by non-continuous access and ensuring that the entire cross-memory copy process is extremely lightweight and efficient.

[0135] In step 1033, a data block to be interacted that conforms to the second logical view is generated in a temporary communication buffer.

[0136] Here, the data block to be interacted with refers to data that is strictly contiguous in physical memory and logically consistent. A set of arranged data. It is the smallest protocol unit that the All-Gather / Reduce-Scatter communication operator can directly process, and its physical arrangement order directly determines the semantic correctness of the data after network transmission.

[0137] In some embodiments, the generation of the data block to be interacted with is automatic as all thread handling tasks in step 1032 are completed. Since each element is precisely positioned according to the target address of the second logical view, the flattened data in the buffer has logically and automatically evolved into an arrangement with the sequence dimension as the first axis.

[0138] In some embodiments, the process of generating data blocks to be interacted with further includes executing memory synchronization primitives (MemoryBarrier / Sync). For example, after multi-threaded copying is complete, the synchronization instruction inside the operator is called to ensure that all elements to be communicated have been flushed from the original storage area to the temporary communication buffer. This ensures that when the underlying communication library reads the data blocks, it obtains a complete, consistent rearrangement result that conforms to the definition of the second logical view, avoiding data gaps or old data residues.

[0139] In step 1034, set communication is performed on the data blocks to be interacted.

[0140] Here, performing aggregate communication refers to using a temporary communication buffer as the source (for All-Gather) or destination (for Reduce-Scatter) to synchronize data between multiple computing nodes or processes via a high-speed interconnect network (such as HCCS, NVLink).

[0141] In some embodiments, the prerequisite for performing set communication is that the data blocks to be exchanged have been aligned based on the sequence dimension within a temporary communication buffer. Since the data is now physically continuous and the sequence dimension is at the highest dimension, the communication operator does not need to perform additional memory copying or padding.

[0142] In some embodiments, aggregate communication can be implemented by calling the communication interface of the underlying communication library (such as hcclAllGather or hcclReduceScatter) and passing the starting address of the temporary communication buffer as an input parameter to the interface. For example, in a serial parallelism scenario, each process executes the communication primitive in parallel, sending its assigned data block to other processes and receiving data blocks returned by other processes. Since the communication takes place on rearranged contiguous memory, it maximizes the utilization of network hardware throughput and eliminates communication performance bottlenecks caused by non-contiguous memory access.

[0143] This application embodiment directly "projects" elements originally scattered under the first logical view to a temporary communication buffer through precise target address mapping copying. This process merges the independent "transpose" and "data transport" in related technologies into a single memory read / write task, eliminating the additional video memory allocation and global bandwidth occupation required to generate a complete intermediate transpose tensor from the perspective of reasoning logic. In addition, the physically strict continuous data blocks to be interacted generated in real time in the temporary communication buffer can meet the hardware requirements of the underlying set communication operator for the continuity of the first axis, thereby ensuring that when performing set communication, the network hardware can be directly driven to transmit data at peak bandwidth, reducing the performance penalty problem when performing parallel communication on non-continuous sequence dimensions.

[0144] In some embodiments, see Figure 8 After step 103, steps 104 to 105 can also be executed.

[0145] In step 104, the third logical view of the second tensor is obtained, and the third logical view is rearranged in a second arrangement based on the first logical dimension to obtain the fourth logical view of the second tensor.

[0146] The second tensor is obtained by performing set communication on the first tensor in the sequence dimension based on the second logical view.

[0147] Here, the second tensor refers to the original data result stored in the temporary communication buffer (Receive Buffer) after the set communication operator (such as All-Gather or Reduce-Scatter) has been executed.

[0148] In some embodiments, the third logical view refers to the immediate logical shape of the second tensor after communication is completed. If an All-Gather operation is performed, its shape is... If the Reduce-Scatter operation is executed, its shape is as follows: To adapt to the BSH layout preferences of subsequent deep learning operators (such as the GEMM operator in the Transformer layer), a second arrangement is needed to restore its logical view.

[0149] In some embodiments, the second rearrangement is implemented through reverse memory address mapping. This rearrangement process does not involve immediate physical memory movement, but rather reconstructs a set of addressing logic based on the structural characteristics of the first logical dimension arrangement (i.e., BSH arrangement). This is achieved by flattening the dimensions in the third logical view. Re-split into independent batch dimensions and hidden state dimension and adjust sequence dimensions ( The logical position of (the new length after communication) is used to obtain the conformance. The fourth logical view as defined.

[0150] In some embodiments, see Figure 9 The step 104, which involves arranging the third logical view in a second arrangement based on the first logical dimension to obtain the fourth logical view of the second tensor, can be achieved by executing steps 1041 to 1042.

[0151] In step 1041, when the set communication is a full collection, a second index mapping relationship is constructed based on the second logical dimension arrangement of the third logical view.

[0152] Here, the second index mapping relationship refers to the mathematical transformation rule that reinterprets and repositions linear data stored in a specific order in the underlying contiguous memory as coordinates of each dimension of the target three-dimensional tensor (i.e., BSH structure).

[0153] In some embodiments, the third logical view and its second logical dimension arrangement can be obtained by retrieving metadata from the receive buffer after the All-Gather operation is completed in the underlying communication library (such as NCCL or HCCL). Since the All-Gather operation concatenates the data of all participating processes along the first dimension, this arrangement typically appears as a two-dimensional flattened shape after the sequence dimension has increased (e.g., ...). ,in (This is the total length of the concatenated sequence).

[0154] In some embodiments, constructing the second index mapping relationship can be achieved by parsing the scales of each dimension of the tensor after communication (i.e., batch size). Total sequence length Hidden state dimension This involves using the stride calculation rule for multidimensional arrays to derive the absolute address of each element. For example, the addressing formula can be used... The coordinates in the unfolded view are dynamically calculated. The accurate offset of the element in the target three-dimensional contiguous memory is used to establish the underlying mapping.

[0155] In some embodiments, see Figure 10 The second logical dimension arrangement based on the third logical view in step 1041, which constructs the second index mapping relationship, can be achieved by executing the following steps 10411 to 10412.

[0156] In step 10411, based on the number of batches, the number of sequences, and the number of hidden states in the second logical dimension arrangement of the third logical view, the second memory offset for rearranging the third logical view into the first logical dimension arrangement is determined.

[0157] Here, the second memory offset refers to the relative position or linear index of a data element in the target contiguous physical memory block, used to precisely indicate the exact address where a certain element in the receive buffer should be stored after being restored to the BSH three-dimensional arrangement.

[0158] In some embodiments, the number of batch dimensions (B), the number of sequence dimensions (S'), and the number of hidden state dimensions (H) can be obtained by extracting the metadata of the tensor descriptors returned by parsing the network layers after the all-gather operation is completed. Since the all-gather operation concatenates the sequence fragments from each computation node, the number of sequence dimensions here... Typically, it is the total global length involved in the communication (e.g., ...). ).

[0159] In some embodiments, determining the second memory offset for rearranging the third logical view into a first logical dimension arrangement can be achieved by employing a linearized address translation algorithm for multidimensional tensors, combined with multiply-accumulate calculations based on the stride weights of each dimension. For example, for a third logical view (such as one with shape...) The coordinates in the middle are The elements to be processed can be obtained using formulas. The unique linear offset corresponding to it in the target BSH physical memory is dynamically calculated.

[0160] In step 10412, the second memory offset of each element in the third logical view is used as the second index mapping relationship.

[0161] Here, the second index mapping relationship refers to the one-to-one correspondence between the physical address of the communication result tensor and the physical address of the target computation tensor. It is the "addressing guide rule" that guides subsequent multi-threaded operators to accurately deliver data during the memory write-back phase.

[0162] In some embodiments, the second memory offset of each element in the third logical view can be obtained in the following way: it is derived in real time by the computing unit of the underlying hardware device based on the local index allocated by each thread during runtime, without the need to pre-allocate space in global memory to store the complete offset array.

[0163] In some embodiments, using the second memory offset of each element as the second index mapping relationship can be achieved by directly embedding the mathematical logic for calculating the second memory offset into addressing instructions in the ParallelKernel. For example, when constructing the fusion operator, the offset calculation formula is converted into register-level operation code by the hardware-level compiler, allowing a large number of concurrently executing threads to directly map and "single-step" to the unique target memory location after reading the coordinates of the element they are responsible for. The mapping formula itself then serves as the carrier of the index relationship.

[0164] This application's embodiments directly and accurately derive the second memory offset based on the dimensional scale (batch, sequence, hidden state) after full collection, and directly solidify its mathematical calculation logic into the second index mapping relationship, transforming the complex cross-dimensional tensor rearrangement into a constant-level addressing process. This eliminates the additional GPU memory overhead and bandwidth consumption caused by building and storing a large explicit index table in related technologies, and provides a simplified and efficient underlying physical navigation for subsequent multi-threaded kernel execution of "zero intermediate tensor" concurrent write-back operations. This fundamentally ensures that the results of sequential parallel communication can be seamlessly restored to the continuous BSH layout required for computation with minimal latency.

[0165] In step 1042, the third logical view is rearranged based on the first logical dimension according to the second index mapping relationship to obtain the fourth logical view of the second tensor.

[0166] Here, the second arrangement refers to the process of restoring the tensor logic structure according to the preset index rules. That is, at the logical level, the data structure "flattened for communication adaptation" is restored to a three-dimensional BSH arrangement structure "designed for computation adaptation".

[0167] In some embodiments, the arrangement requirements of the first logical dimension can be obtained by reading a preset model parameter configuration file to ensure that the rearranged fourth logical view can meet the access requirements of the downstream matrix multiplication (GEMM) operator for physical memory continuity.

[0168] In some embodiments, a second rearrangement of the third logical view based on the first logical dimension can be achieved by applying the constructed second index mapping relationship to refresh the description information of the second tensor. Specifically, this action can employ a "View Reshape" algorithm, which does not change the actual storage location of the data in physical memory, but rather modifies the tensor's stride metadata, enabling the data processing system to perform logical addressing according to... The hierarchical order is parsed to output a shape of... The fourth logical view.

[0169] This application embodiment accurately constructs a second index mapping relationship and executes a second logical arrangement in a full collection scenario. It uses a logical view transformation mechanism to replace the high-latency physical transpose operation in related technologies. It achieves accurate regression of the communication flattened view to the three-dimensional BSH calculation layout with extremely low time complexity. This not only eliminates the need for additional large tensor storage space in the post-processing stage, but also ensures that the sequence data after full collection can immediately adapt to the high-efficiency memory access requirements of the NPU's underlying matrix multiplication with constant-level mapping efficiency, thereby improving the execution efficiency of the end-to-end sequence parallel training task.

[0170] In some embodiments, see Figure 11 The step 104, which involves arranging the third logical view in a second arrangement based on the first logical dimension to obtain the fourth logical view of the second tensor, can also be achieved by executing the following steps 1043 to 1044.

[0171] In step 1043, when the set communication is a reduction distribution, a fifth logical view is selected from the third logical view, and a third index mapping relationship is constructed based on the second logical dimension arrangement of the fifth logical view.

[0172] Here, the fifth logical view refers to a local data slice view that belongs specifically to the current computing process (or device) and is segmented from the communication results containing global information after the reduce-scatter operation of the collection communication is completed.

[0173] In some embodiments, selecting the fifth logical view from the third logical view can be achieved by: selecting the fifth logical view from the flattened third logical view based on the global process ID (Rank ID) of the currently participating distributed training device and the total number of devices participating in the communication (WorldSize). In this process, the data is segmented and extracted along the sequence dimension, selecting consecutive data segments belonging to the current process. For example, if the total sequence length is... Then the length of the local sequence allocated to the current process is . .

[0174] In some embodiments, constructing a third index mapping relationship based on the second logical dimension arrangement of the fifth logical view can be achieved by resolving the batch dimension count in the local data slice view. Number of local sequence dimensions and the number of hidden state dimensions This leads to the derivation of addressing rules for the transformation of local elements into the target 3D layout (BSH). For example, a coordinate mapping algorithm can be used to transform the flattened coordinates in the fifth logical view into... The linear indices of the elements are converted into the output tensor to be written to the target in the future. The memory offset rules at that time establish the reverse mapping relationship unique to that process.

[0175] In some embodiments, see Figure 12 The second logical dimension arrangement based on the fifth logical view in step 1043, which constructs the third index mapping relationship, can be achieved by executing the following steps 10431 to 10432.

[0176] In step 10431, based on the number of batches, the number of sequences, and the number of hidden states in the second logical dimension arrangement of the fifth logical view, a third memory offset is determined to rearrange the fifth logical view into a first logical dimension arrangement.

[0177] Here, the third memory offset refers to the absolute linear address or position offset of an element in the local data slice allocated to the current computing node within the final contiguous target 3D tensor (BSH arrangement). It serves as the specific addressing coordinates that guide the precise placement of local discrete data into global physical memory.

[0178] In some embodiments, the quantity (B) corresponding to the batch dimension and the quantity (B) corresponding to the sequence dimension of the fifth logical view are... The number of hidden state dimensions (H) and the corresponding number of hidden state dimensions can be obtained as follows: after the underlying communication operator completes the reduce-scatter operation, it is extracted from the metadata of the current device's receive buffer. Since the reduce-scatter operation splits the sequence dimensions, the number of sequence dimensions at this point... The total global sequence length is divided by the total number of devices participating in the communication (i.e., ).

[0179] In some embodiments, determining the third memory offset for rearranging the fifth logical view into a first logical dimension arrangement can be achieved by using a stride mapping algorithm for multidimensional arrays in underlying contiguous memory to perform reverse address deduction for local slice data. For example, for the fifth logical view (e.g., shape... The coordinates in the middle are For data elements, operators can utilize addressing formulas. Calculate the element in the target output tensor The only linear offset in, This represents the number of sequence dimensions during reduction distribution.

[0180] In step 10432, the third memory offset of each element in the fifth logical view is used as the third index mapping relationship.

[0181] Here, the third index mapping relationship refers to the mapping rule established between the local communication result set of the current process and the physical storage space required for the final computation. It is essentially a set of mathematical addressing rules that avoids the physical intermediate state transposition.

[0182] In some embodiments, the third memory offset of each element in the fifth logical view can be obtained by a parallel thread of the target hardware (such as the CUDA Core of a GPU or the Vector Core of an NPU) in real time during runtime. The operator scheduler allocates an independent thread for each local element to be processed, and the thread calculates the offset on the fly based on the local coordinates of the source address it is responsible for, without having to allocate a huge amount of space in global memory (HBM) to pre-store these offset coordinates.

[0183] In some embodiments, using the third memory offset of each element as a third index mapping relationship can be achieved by directly inlining the algebraic formula for calculating the third memory offset into the multi-threaded parallel kernel instructions responsible for write-back operations. For example, in the kernel code, register-level mathematical operation instructions replace the memory lookup instructions in related technologies, allowing tens of thousands of concurrent threads to read their allocated local coordinates. At this time, the mapping relationship can be established by executing the above formula step by step, thereby realizing the transformation of logical arrangement with high computational efficiency. The index for the sequence dimension during reduction distribution.

[0184] This application's embodiments accurately deduce the third memory offset based on the local dimensional scale (batch, local sequence, hidden state) after reduction distribution, and directly solidify its mathematical calculation logic into the third index mapping relationship. This application's embodiments transform the complex multidimensional rearrangement of local data slices into a constant-level addressing process. This method saves the additional memory occupation and bandwidth consumption caused by building and reading a large explicit index table in global memory, and provides a simplified underlying physical navigation for subsequent concurrent write-back of multi-threaded kernels. It ensures that after each distributed node obtains network communication slices, it can seamlessly align to the three-dimensional BSH continuous arrangement required for downstream computation with the efficiency of "zero intermediate memory allocation", thereby improving the end-to-end throughput performance of parallel training of large model sequences.

[0185] In step 1044, the fifth logical view is rearranged in a third manner based on the first logical dimension according to the third index mapping relationship, to obtain the fourth logical view of the second tensor.

[0186] Here, the third arrangement refers to the process of reshaping the logical dimension of the fragmented data belonging to the current process using the constructed local index rules. The aim is to restore the two-dimensional local fragments that are continuously stored for network reduction communication to a standard three-dimensional physical layout (i.e., the first logical dimension arrangement, BSH structure) that is compatible with the subsequent computing engine.

[0187] In some embodiments, the structural specification of the first logical dimension arrangement of the target can be obtained by reading the shape configuration file of the corresponding network layer (such as the self-attention mechanism layer or the feedforward neural network layer) for the input slice tensor in the current large language model (such as the Transformer architecture) to ensure that the distributed slice data can meet the strict alignment requirements of the underlying matrix multiplication engine for continuous memory access after logical reconstruction.

[0188] In some embodiments, the third rearrangement of the fifth logical view according to the third index mapping can be achieved by directly binding the stride and offset parameters representing the third index mapping to the data slice pointer of the current process in the logical control flow of operator execution. For example, by performing a "view reshape" algorithm at the tensor descriptor level, no physical memory data is copied; it is only removed from the addressing metadata. The two-dimensional form is switched to The fourth logical view eliminates the dimensional flattening and misalignment caused by reduction distribution at the logical level.

[0189] This application embodiment precisely extracts a local data view specific to the current process for the reduction and distribution scenario and constructs a third index mapping relationship. This application embodiment applies a "zero-copy" view reshaping mechanism (third arrangement) to replace the expensive physical transposition process of video memory. This design not only eliminates the intermediate tensor video memory occupation and video memory bandwidth consumption when restoring distributed slice data to the 3D BSH layout, but also ensures that after completing network reduction, each computing node can independently and correctly seamlessly connect the local sequence slices to the downstream matrix calculation engine with extremely low addressing latency, thereby improving the end-to-end throughput and distributed execution efficiency of parallel training of large model sequences.

[0190] In step 105, a write-back operation is performed on the second tensor based on the fourth logical view.

[0191] Here, the write-back operation refers to moving the elements in the receive buffer that are in the communication layout state to the final output tensor memory area, and ensuring that they conform to the BSH contiguous layout in physical storage.

[0192] In some embodiments, the write-back operation is implemented using a multi-threaded parallel kernel. Each thread calculates the target location of its assigned element according to the addressing rules of the fourth logical view. For example, using the reverse offset calculation formula. Threads can directly retrieve elements from the receive buffer. The position is read and the correct physical address of the output tensor is written step by step.

[0193] In some embodiments, writing back based on the fourth logical view has the following technical advantages: First, since the write-back process directly uses reverse address mapping, the entire operation does not generate intermediate transposed tensors, saving GPU memory space; second, this operation ensures the consistency of the output tensor and the input tensor in physical memory layout. This means that after completing sequential parallel communication, subsequent matrix multiplication (GEMM) operations can be started directly on contiguous memory without additional transpose or reshaping operations, thereby achieving efficient connection of the end-to-end training process.

[0194] This embodiment of the application achieves seamless restoration of the communication result to the original BSH layout at the inference level by performing reverse logic rearrangement on the second tensor after communication and writing it back directly, replacing the explicit transpose operator with reverse address mapping logic. This mechanism ensures that the output tensor can immediately adapt to the continuous memory access requirements of the NPU's underlying GEMM computation, while eliminating the memory overhead of generating temporary large tensors in the post-processing stage. This allows the end-to-end execution latency to be reduced while maintaining high-performance memory access.

[0195] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0196] In related technologies, the sequential parallel communication scheme for distributed training of large language models has failed to fully resolve the contradiction between the underlying matrix calculation engine's dependence on continuous physical arrangement (such as BSH layout) and the underlying communication library's requirement for first-axis communication (such as SBH layout). This restricts the development of computing clusters in core computing characteristics such as ultra-long text processing, memory utilization efficiency, communication bandwidth maximization, and end-to-end training throughput. Dimensional reorganization of multidimensional tensors remains at the level of inefficient physical transport by explicitly creating complete intermediate transpose tensors, making it difficult to utilize the bandwidth of the underlying cache and memory bus, and also failing to fully leverage the computing potential of modern AI accelerators in ultra-large-scale distributed training.

[0197] To address the problems of frequent non-contiguous memory access, increased communication latency, and insufficient global memory utilization caused by the lack of an efficient, intermediate-state-free tensor reshaping mechanism in sequence parallel scenarios, this application provides a data processing method. This method is an efficient set communication data flow method based on multi-dimensional logical view reshaping and real-time memory mapping. This method improves the overall communication performance and hardware execution efficiency of computing nodes in distributed collaboration by constructing a low-level addressing logic that integrates dimensional step-by-step computation and continuous memory access characteristics. This application uses an operator-based multi-threading mechanism to dynamically deduce the first memory offset (corresponding to the first index mapping relationship in the above embodiment) based on the first logical dimension arrangement of the input tensor, and optimizes the data reading pattern by combining a continuous access strategy that merges batch dimensions and hidden state dimensions. Based on the deduced offset vector, multi-dimensional data in memory is written directly from the original physical address to a temporary communication buffer in a single step, and then written back after communication.

[0198] This application's embodiments enable computing nodes to precisely bypass the physical generation process of intermediate transposed tensors, providing data alignment support with zero additional GPU memory overhead for ensemble communication. This application's embodiments solve the problem of redundant transposed data occupying valuable global GPU memory space in large-scale distributed training, ensuring memory access efficiency and scalability of hardware devices under high load, enabling computing nodes to handle large model training tasks with longer sequence spans. This application's embodiments enable computing clusters to improve the utilization depth and cache hit accuracy of the underlying GPU memory bus bandwidth through continuous reading and single-step directional writing in continuous distributed data interaction. This application's embodiments reduce the dependence on inefficient physical memory copying in the ensemble communication preprocessing stage, enhancing the underlying operators' ability to instantly reshape complex multidimensional data structures, enabling seamless integration with mainstream high-performance communication libraries with extremely low latency. This application's embodiments improve the end-to-end communication throughput and memory access overhead hiding capabilities of large model training systems, overcoming the "bandwidth wall" defect caused by explicit three-dimensional transposition in related technologies, making distributed computing power collaboration more natural and smooth. The embodiments of this application provide a key underlying data flow mechanism foundation for the evolution of large language model training systems from an architecture limited by communication and memory access bottlenecks to a new generation of computing clusters with high memory efficiency and extremely low communication latency.

[0199] The following describes the specific implementation process of the data processing method provided in the embodiments of this application.

[0200] Step 1: Perform tensor pre-rearrangement within the communication operator.

[0201] Inside the communication operator, a logical view (first logical view) is pre-stored according to the physical arrangement order of the original tensor (first tensor). That is, copying the original tensor according to the first logical view will result in a tensor to be communicated with the same arrangement as the first tensor in the temporary communication buffer.

[0202] See Figure 13 The original tensor X has the shape (batch_size, sequence_length, hidden_dim), meaning it follows a BSH (Batch-Synchronous Hidden State) layout. Each hidden state vector (of length H) is stored sequentially according to its memory address, forming a basic data block. The storage format is [h1, h2, ..., h_H] (where h_i represents the i-th hidden state value). In both batch 0 and batch 1, the tensor is divided into different segments according to sequence 0, sequence 1, and sequence 2, and the hidden state vectors in each segment are ordered sequentially. The S hidden state vectors (hidden states) within the same sequence are concatenated sequentially to form a contiguous memory segment. Finally, the memory segments of the B sequences (hidden states) are further concatenated sequentially to form a complete tensor memory layout: [batch1_seq1_h1, ..., batch1_seq_S_h_H, batch2_seq1_h1, ..., batch_B_seq_S_h_H]. In the original tensor, all data is stored strictly contiguously at physical addresses without gaps, ensuring efficient memory access.

[0203] Inside the communication operator (for the first logical view), the rearrangement is automatically calculated based on the target communication dimension to obtain a rearranged logical view (the second logical view) with a shape of (sequence_length, batch_size*hidden_dim). That is, the rearranged logical view is arranged according to SB×H.

[0204] To ensure the communication axis is located in dimension 0 (axis=0), the fusion operator internally performs a logical view transformation, converting the logical shape of the input tensor from (B, S, H) to (S, B×H) without generating a complete intermediate transpose tensor. Unlike related technologies that explicitly perform transpose to generate an intermediate (S, B, H) tensor, this embodiment uses direct memory address mapping combined with multi-threaded parallel copying.

[0205] In the logical view after arrangement, the position of element X[b, s, h] is [s, b×H+h].

[0206] The corresponding formula for calculating the target memory offset is: offset_out = s × (B × H) + b × H + h;

[0207] The fusion operator starts a three-dimensional thread grid (grid / block dimensions correspond to sequence, batch, and hidden, respectively). Each thread is responsible for one element: calculate (b, s, h) based on the global thread index, read the element directly from the input offset_in, and write it to the pre-allocated communication buffer at the corresponding offset_out position.

[0208] This process involves only one necessary element copy, without generating additional complete tensors, and the communication buffer memory is contiguous after the copy, which facilitates efficient transmission by the underlying communication library.

[0209] A three-dimensional thread block layout is used, corresponding to the sequence, batch, and hidden dimensions respectively. Each thread processes one element, directly copying it from the input position to the output position.

[0210] See Figure 14 , Figure 14 The diagram illustrates the first logical view of the original tensor in physical memory (i.e., a typical BSH layout). As can be seen, the data is first divided by batch dimension at the outermost layer (macro level) (e.g., batch 0 and batch 1 in the diagram); within each batch, the data is arranged sequentially by sequence dimension (e.g., sequence 0, sequence 1, sequence 2); and within each sequence block, the innermost layer is a physically contiguous hidden state dimension (e.g., hidden state vectors 0 and 1, marked with different shading, within sequence 0 in the diagram). In this layout, data belonging to the same sequence position (e.g., sequence 0 in batch 0 and sequence 0 in batch 1) are separated by strides in physical memory. Direct communication based on this state at the sequence dimension would inevitably lead to large-scale, non-contiguous memory accesses.

[0211] See Figure 15 , Figure 15 After rearranging the data according to the first index mapping, the second logical view (i.e., the arrangement with S as the first axis) is written into the temporary communication buffer. As can be seen from the diagram, the outermost organizational dimension of the data has undergone a fundamental change; the sequence dimension has been extracted to the first axis position (i.e., the outermost independent sequences 0, 1, and 2 in the diagram). This is achieved through multi-threaded concurrent memory offset mapping. Figure 14 The same sequence fragments, originally scattered across different batches (e.g., sequence 0 in batch 0 and sequence 0 in batch 1), were precisely "extracted" and continuously "assembled" into... Figure 15 Within the same sequence 0 block. Compare Figure 14 and Figure 15As can be seen, the data rearrangement process in this application is essentially a memory projection that "transforms discrete data into continuous data." It breaks the original batch boundaries, forcibly clustering and aligning the sequence segments (Seq) to be communicated in physical space. Through this mapping process, Figure 15 Each Seq data block becomes a physically contiguous independent unit. This arrangement aligns with the hardware requirement that underlying set communication operators (such as All-Gather or Reduce-Scatter) can only perform continuous readings on the 0th axis (first axis), thus enabling the system to directly... Figure 15 The corresponding memory address is submitted to the communication library, eliminating the need to generate a complete intermediate transpose tensor, thus achieving efficient data transfer with zero additional padding and zero communication bandwidth waste.

[0212] Taking an original tensor with dimensions B=2, S=2, and H=2 as an example, the positions of the elements in the logical view of the original tensor are shown in Table 1:

[0213] Table 1: Memory layout order of elements in the logical view of the original tensor BSH=(2,2,2) (offset 0~7):

[0214]

[0215] The positions of elements in the logical view after pre-rearrangement are shown in Table 2:

[0216] Table 2: Expected memory layout order of elements in the rearranged logical view (S, B*H) = (2, 4) (contiguous memory order, offset 0~7):

[0217]

[0218] Referring to Tables 1 and 2, taking the original tensor size of B=2, S=2, H=2 as an example, this embodiment of the application intuitively demonstrates the physical storage changes of the first tensor before and after rearrangement and the underlying mapping logic through specific numerical values. Observing the original BSH logical view shown in Table 1, it can be found that a total of 8 elements are arranged in batch, sequence, and hidden state dimensions. Although the innermost hidden state data (such as b0s0h0 and b0s0h1 at offsets 0 and 1) are continuous in memory, for the same sequence dimension that needs to perform set communication ( For example, in the case of s0, data belonging to batch b0 and data belonging to batch b1 (b1s0h0 and b1s0h1 at offsets 4 and 5) are completely blocked in physical memory by data in sequence s1 (b0s1h0 and b0s1h1 at offsets 2 and 3). This staggered distribution makes it impossible to obtain a complete sequence fragment through a single continuous read during communication. To address this discreteness issue, Table 2 shows the expected S(B*H) logical view after pre-rearrangement. The core analysis process lies in using single-step address mapping logic (i.e., the target offset equals...). This method precisely groups the originally scattered data. For example, in Table 1, the element b1s0h0, whose original offset was 4, becomes 2 when its target offset is calculated by substituting it into the formula. This perfectly matches the closely arranged b0s0h0, b0s0h1, b1s0h0, and b1s0h1 at offsets 0 to 3 in Table 2. Based on the above table reading and analysis, it can be seen that the same sequence segments that were originally separated by steps in physical memory have been successfully spliced ​​into physically continuous independent units. This makes the tensor exhibit a large block feature with the sequence dimension S as the first axis and the batch and hidden state dimensions merged and continuous. This "converting discrete into continuous" underlying mapping projection eliminates the step of generating a complete SBH intermediate transpose tensor. It not only avoids the cache miss penalty caused by non-contiguous memory access, but also makes the reorganized continuous memory layout fit the hardware requirements of the underlying collection communication library for continuous reading of the first axis, thereby realizing efficient collection communication data flow with zero bandwidth waste.

[0219] Step 2: Perform set communication on the original tensors according to the pre-rearranged logical view.

[0220] The memory starting address of the rearranged logical view is passed to the collection communication library. Following the rearranged logical view, the original tensors in memory are copied to a temporary communication buffer. The data blocks to be interacted following the temporary communication buffer are arranged in the same logical dimensions as the rearranged logical view, i.e., arranged according to SB×H. Then, All-Gather or Reduce-Scatter is executed on the data blocks to be interacted in the sequence dimension.

[0221] Step 3: Write the data blocks after the set communication back to the original memory.

[0222] Perform an All-Gather or Reduce-Scatter operation on the data in the temporary communication buffer.

[0223] If it is an All-Gather operation, after communication each process receives a receive buffer of shape (S×world_size, B×H);

[0224] If it is a Reduce-Scatter operation, after communication each process receives a receive buffer of shape (S / world_size, B×H) (world_size is the number of processes participating in the sequence parallelism).

[0225] The call interface for writing the original tensor in memory to the temporary communication buffer and writing the collection communication result (i.e., the second tensor in the above embodiment) in the temporary communication buffer back to memory is as follows:

[0226]

[0227] After the set communication is completed, the logical shape of the receive buffer is (S', B×H) (S' is the sequence dimension length after communication). The fusion operator performs the reverse memory address mapping, which is the opposite of step 2:

[0228] The element is mapped from the receive buffer position [s', b×H+h] back to the output tensor position [b, s', h];

[0229] The reverse offset calculation formula is: offset_final=b×(S'×H)+s'×H+h. It is also implemented using a multi-threaded parallel kernel and directly copied to the final output tensor.

[0230] Once completed, the logical shape and memory layout of the output tensor are completely consistent with those of the input tensor, i.e., it is still a continuous BSH layout of (B, S', H), which can be directly used for GEMM calculations in subsequent Transformer layers without additional adjustments.

[0231] The following description continues to illustrate the exemplary structure of the data processing apparatus 255 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the data processing device 255 of the memory 250 may include:

[0232] The acquisition module 2551 is used to acquire the first logical view of the input first tensor, wherein the first logical dimension arrangement of the first logical view includes the batch dimension, the sequence dimension and the hidden state dimension in sequence.

[0233] The rearrangement module 2552 is used to perform a first rearrangement based on the second logical dimension of the first logical view to obtain the second logical view of the first tensor. The second logical dimension arrangement includes the sequence dimension and the merge dimension in sequence. The merge dimension is the product of the batch dimension and the hidden state dimension.

[0234] Communication module 2553 is used to perform set communication on the first tensor in the sequence dimension based on the second logical view.

[0235] In some embodiments, the rearrangement module 2552 is further configured to construct a first index mapping relationship based on the first logical dimension arrangement and the second logical dimension arrangement of the first logical view;

[0236] The first logical view is rearranged based on the second logical dimension according to the first index mapping relationship to obtain the second logical view of the first tensor.

[0237] In some embodiments, the rearrangement module 2552 is further configured to determine a first memory offset for rearranging the first logical view into a second logical dimension arrangement based on the number of batches, the number of sequences, and the number of hidden states in the first logical dimension arrangement.

[0238] Use the first memory offset of each element in the first logical view as the first index mapping relationship.

[0239] In some embodiments, the communication module 2553 is further configured to obtain the original storage address of the first tensor under the first logical view, and obtain the target storage address corresponding to the original storage address in the second logical view.

[0240] The element to be communicated is read from memory based on the original storage address, and written to a temporary communication buffer based on the target storage address.

[0241] Generate a data block to be interacted that conforms to the second logical view within a temporary communication buffer;

[0242] Perform collection communication on the data blocks to be interacted with.

[0243] In some embodiments, the communication module 2553 is further configured to determine a first memory offset for rearranging the first logical view into a second logical dimension arrangement based on the number of batches, the number of sequences, and the number of hidden states in the first logical dimension arrangement.

[0244] Based on the first memory offset, determine the second physical address of the temporary communication buffer to be written to, corresponding to the first physical address where each element in the first tensor is located.

[0245] Read each element in the first tensor in the order of the first physical address, and write it to the second physical address step by step.

[0246] In some embodiments, the communication module 2553 is further configured to select a plurality of start elements from the elements of the first tensor;

[0247] Multiple starting elements are used as the first elements for parallel reading. Multiple elements in the first tensor are read in parallel according to the order of the first physical address, and the second physical address corresponding to the multiple elements is written step by step.

[0248] In some embodiments, the communication module 2553 is further configured to obtain a third logical view of the second tensor, and to perform a second rearrangement of the third logical view based on the first logical dimension to obtain a fourth logical view of the second tensor.

[0249] The second tensor is obtained by performing set communication on the first tensor in the sequence dimension based on the second logical view;

[0250] The second tensor is written back based on the fourth logical view.

[0251] In some embodiments, the communication module 2553 is further configured to construct a second index mapping relationship based on the second logical dimension arrangement of the third logical view when the collection communication is a full collection.

[0252] The third logical view is rearranged based on the first logical dimension according to the second index mapping relationship, resulting in the fourth logical view of the second tensor.

[0253] In some embodiments, the communication module 2553 is further configured to determine a second memory offset for rearranging the third logical view into a first logical dimension arrangement based on the number of batches, the number of sequences, and the number of hidden states in the second logical dimension arrangement of the third logical view.

[0254] Use the second memory offset of each element in the third logical view as the second index mapping relationship.

[0255] In some embodiments, the communication module 2553 is further configured to select a fifth logical view from the third logical view and construct a third index mapping relationship based on the second logical dimension arrangement of the fifth logical view when the set communication is a reduction distribution;

[0256] Based on the third index mapping relationship, the fifth logical view is rearranged in a third manner based on the first logical dimension to obtain the fourth logical view of the second tensor.

[0257] In some embodiments, the communication module 2553 is further configured to determine a third memory offset for rearranging the fifth logical view into a first logical dimension arrangement based on the number of batches, the number of sequences, and the number of hidden states in the second logical dimension arrangement of the fifth logical view.

[0258] Use the third memory offset of each element in the fifth logical view as the third index mapping relationship.

[0259] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the data processing method described in this application.

[0260] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the data processing method provided in this application. For example, ... Figure 3 The data processing method is shown.

[0261] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0262] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0263] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0264] In summary, this application's embodiments directly derive memory offsets using the mapping relationship of the logical dimension, reading source tensor elements at consecutive physical addresses and writing them step-by-step into a temporary communication buffer. This eliminates the redundant step of explicitly generating a complete transposed intermediate tensor, reducing the system's GPU memory usage and global memory read / write bandwidth consumption. Furthermore, the rearrangement strategy that merges the batch dimension and the hidden state dimension strictly maintains the physical locality of the underlying data, effectively triggering the memory merging access mechanism of the computing hardware, reducing cache miss penalties caused by non-contiguous memory access, and ensuring the efficiency of data cross-memory copying. In addition, after completing set communication operations such as full collection or reduction distribution, this application, based on the reverse memory offset calculation or logical view reshaping mechanism, directly restores the flattened arrangement of the communication results to a three-dimensional continuous physical layout adapted to the downstream matrix multiplication engine, eliminating additional physical transport delays in the data recovery stage. Ultimately, this effectively improves the end-to-end communication efficiency and the overall throughput performance of the distributed computing cluster in large language model sequence parallel training scenarios.

[0265] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain a first logical view of the first input tensor, wherein the first logical dimensions of the first logical view are arranged in sequence as batch dimension, sequence dimension and hidden state dimension; The first logical view is rearranged based on the second logical dimension to obtain the second logical view of the first tensor. The second logical dimension arrangement includes the sequence dimension and the merge dimension in sequence. The merge dimension is the product of the batch dimension and the hidden state dimension. Based on the second logical view, set communication is performed on the first tensor in the sequence dimension.

2. The method according to claim 1, characterized in that, The step of performing a first rearrangement based on a second logical dimension on the first logical view to obtain a second logical view of the first tensor includes: Based on the first logical dimension arrangement and the second logical dimension arrangement of the first logical view, a first index mapping relationship is constructed; The first logical view is rearranged based on the second logical dimension according to the first index mapping relationship to obtain the second logical view of the first tensor.

3. The method according to claim 2, characterized in that, The construction of the first index mapping relationship based on the first logical dimension arrangement and the second logical dimension arrangement of the first logical view includes: Based on the number of batches, the number of sequences, and the number of hidden states in the first logical dimension arrangement, the first memory offset for rearranging the first logical view into the second logical dimension arrangement is determined. The first memory offset of each element in the first logical view is used as the first index mapping relationship.

4. The method according to claim 1, characterized in that, The execution of set communication based on the second logical view in the sequence dimension includes: Obtain the original storage address of the first tensor in the first logical view, and obtain the target storage address corresponding to the original storage address in the second logical view; The element to be communicated is read from memory based on the original storage address, and the element to be communicated is written into a temporary communication buffer based on the target storage address. Generate a data block to be interacted that conforms to the second logical view within the temporary communication buffer; The set communication is performed on the data block to be interacted with.

5. The method according to claim 4, characterized in that, The step of writing the element to be communicated into the temporary communication buffer based on the target storage address includes: Based on the number of batches, the number of sequences, and the number of hidden states in the first logical dimension arrangement, the first memory offset for rearranging the first logical view into the second logical dimension arrangement is determined. Based on the first memory offset, determine the second physical address of the temporary communication buffer to be written to the first physical address where each element in the first tensor is located; Read each element in the first tensor in the order of the first physical address, and write it step by step to the second physical address.

6. The method according to claim 5, characterized in that, The step of reading each element in the first tensor in the order of the first physical address and writing it step by step to the second physical address includes: Select multiple starting elements from the elements of the first tensor; Each of the multiple starting elements is used as the first element to be read in parallel. Multiple elements in the first tensor are read in parallel according to the order of the first physical address, and the second physical address corresponding to the multiple elements is written step by step.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the third logical view of the second tensor, and perform a second rearrangement of the third logical view based on the first logical dimension to obtain the fourth logical view of the second tensor. Wherein, the second tensor is obtained by performing set communication on the first tensor in the sequence dimension based on the second logical view; The second tensor is written back based on the fourth logical view.

8. The method according to claim 7, characterized in that, The step of performing a second rearrangement of the third logical view based on the first logical dimension to obtain a fourth logical view of the second tensor includes: In the case that the collection communication is a full collection, a second index mapping relationship is constructed based on the second logical dimension arrangement of the third logical view; The third logical view is rearranged based on the first logical dimension according to the second index mapping relationship to obtain the fourth logical view of the second tensor.

9. The method according to claim 8, characterized in that, The construction of the second index mapping relationship based on the second logical dimension arrangement of the third logical view includes: Based on the number of batches, the number of sequences, and the number of hidden states in the second logical dimension arrangement of the third logical view, a second memory offset is determined to rearrange the third logical view into the first logical dimension arrangement. The second memory offset of each element in the third logical view is used as the second index mapping relationship.

10. The method according to claim 7, characterized in that, The step of performing a second rearrangement of the third logical view based on the first logical dimension to obtain a fourth logical view of the second tensor includes: In the case where the set communication is a reduction distribution, a fifth logical view is selected from the third logical view, and a third index mapping relationship is constructed based on the second logical dimension arrangement of the fifth logical view; The fifth logical view is rearranged in a third manner based on the first logical dimension according to the third index mapping relationship, to obtain the fourth logical view of the second tensor.

11. The method according to claim 10, characterized in that, Based on the second logical dimension arrangement of the fifth logical view, a third index mapping relationship is constructed, including: Based on the number of batches, the number of sequences, and the number of hidden states in the second logical dimension arrangement of the fifth logical view, a third memory offset is determined to rearrange the fifth logical view into the first logical dimension arrangement. The third memory offset of each element in the fifth logical view is used as the third index mapping relationship.

12. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the first logical view of the input first tensor, wherein the first logical dimension arrangement of the first logical view includes the batch dimension, the sequence dimension and the hidden state dimension in sequence. The rearrangement module is used to perform a first rearrangement based on the second logical dimension on the first logical view to obtain a second logical view of the first tensor, wherein the second logical dimension arrangement includes a sequence dimension and a merge dimension in sequence, and the merge dimension is the product of the batch dimension and the hidden state dimension. A communication module is used to perform set communication on the first tensor in the sequence dimension based on the second logical view.

13. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the data processing method according to any one of claims 1 to 11.

15. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the data processing method according to any one of claims 1 to 11.

Images