Data processing method, data processing device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method, a data processing device, electronic equipment and a storage medium. The data processing method includes: receiving a signal about an input tensor and an index tensor; target data are read from a source storage space corresponding to the input tensor based on the index tensor through a general tensor operation interface, the target data are written into a target storage space corresponding to the output tensor, and the general tensor operation interface supports unified tensor operation involving at least two data arrangement modes. According to the data processing method, a universal and efficient index operation implementation scheme is provided, different data arrangement modes can be adapted, and complex index access requirements under different hardware architectures are compatible, so that the generalization ability and execution efficiency of index operators are remarkably improved, and the data processing efficiency is improved. High-efficiency and low-latency indexing operation on heterogeneous computing equipment can be supported.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of artificial intelligence technology, and more particularly, to a data processing method, a data processing device, an electronic device, and a storage medium. Background Art

[0002] Tensor indexing is a common operation in fields such as deep learning, image processing, and large-scale data computing. For example, in feature extraction, label mapping, or attention mechanisms, it is often necessary to extract data at specific locations from a source tensor (input tensor) based on a set of indices to form a new target tensor (output tensor).

[0003] The Index operator is a high-level indexing operation used to extract data from an input tensor of arbitrary dimensions. This operation supports extracting data from any dimension of the input tensor using a given integer index tensor, making it flexibly applicable to data preprocessing and specific model tasks.

[0004] As the scale of models continues to expand and the complexity of operators continues to increase, how to optimize the efficiency of index operations has become an urgent problem to be solved. Summary of the Invention

[0005] At least one embodiment of the present disclosure provides a data processing method, which includes: receiving signals about an input tensor and an index tensor; reading target data from a source storage space corresponding to the input tensor based on the index tensor through a universal tensor operation interface, and writing the target data to a destination storage space corresponding to an output tensor, wherein the universal tensor operation interface supports unified tensor operations involving at least two data arrangement modes.

[0006] In the data processing method provided in at least one embodiment of the present disclosure, the target data is read from the source storage space corresponding to the input tensor based on the index tensor through a general tensor operation interface, and the target data is written to the destination storage space corresponding to the output tensor, including: using the general tensor operation interface to perform the following operations: according to the tensor attribute information of the output tensor, determine the output coordinates of the target data corresponding to the current thread; according to the tensor attribute information of the index tensor and the output coordinates, determine the index value offset corresponding to the current thread to obtain the input coordinates of the target data; according to the tensor attribute information of the input tensor and the input coordinates, determine the address offset of the target data in the input tensor and read the target data from the source storage space; according to the tensor attribute information of the output tensor, write the target data to the destination storage space, wherein the tensor attribute information includes the data arrangement method.

[0007] In the data processing method provided by at least one embodiment of the present disclosure, the current thread is included in a thread bundle, and the output coordinates of the target data corresponding to the current thread are determined based on the tensor attribute information of the output tensor, including: determining a task distribution method that supports continuous writing based on the data arrangement method and shape of the output tensor, so that data blocks written by multiple threads in the same thread bundle are arranged continuously in the destination storage space; determining the output coordinates of the target data corresponding to the current thread based on the task distribution method.

[0008] In the data processing method provided by at least one embodiment of the present disclosure, the index value offset corresponding to the current thread is determined based on the tensor attribute information of the index tensor and the output coordinates to obtain the input coordinates of the target data, including: intercepting the output coordinates according to the index starting dimension and the dimension of the index tensor to determine the coordinates of the index value corresponding to the current thread; determining the index value offset corresponding to the current thread according to the data arrangement method of the index tensor and the coordinates of the index value; and determining the input coordinates of the target data according to the index value offset and the output coordinates.

[0009] In the data processing method provided by at least one embodiment of the present disclosure, determining the input coordinates of the target data based on the index value offset and the output coordinates includes: starting from the 0th dimension, accessing each dimension of the input tensor in sequence: in response to the existence of an index tensor in the current dimension, determining the value of the input coordinate in the current dimension based on the index value offset and the number of index tensors before the current dimension; in response to the absence of an index tensor in the current dimension, determining the value of the input coordinate in the current dimension based on the value of the output coordinate in the corresponding dimension.

[0010] In the data processing method provided by at least one embodiment of the present disclosure, determining the value of the input coordinate in the current dimension based on the index value offset and the number of index tensors before the current dimension includes: determining the index tensor corresponding to the current dimension based on the number of index tensors before the current dimension; reading the index value in the index tensor corresponding to the current dimension based on the index value offset, and using it as the value of the input coordinate in the current dimension.

[0011] In the data processing method provided by at least one embodiment of the present disclosure, determining the value of the input coordinate in the current dimension based on the value of the output coordinate in the corresponding dimension includes: in response to the absence of an index tensor before the current dimension, using the value of the output coordinate in the current dimension as the value of the input coordinate in the current dimension; in response to the existence of an index tensor before the current dimension, using the value of the output coordinate in the Xth dimension as the value of the input coordinate in the current dimension, where X = current dimension + dimension of index tensor - number of index tensors before the current dimension.

[0012] In the data processing method provided in at least one embodiment of the present disclosure, for the same target data, the index value offset corresponding to each index tensor is the same.

[0013] In the data processing method provided in at least one embodiment of the present disclosure, determining the address offset of the target data in the input tensor and reading the target data from the source storage space based on the tensor attribute information of the input tensor and the input coordinates includes: determining the address offset of the target data in the input tensor based on the data arrangement mode of the input tensor and the input coordinates; and reading the target data from the source storage space based on the address offset.

[0014] In the data processing method provided by at least one embodiment of the present disclosure, the current thread is included in a thread bundle, and the target data is written into the destination storage space according to the tensor attribute information of the output tensor, including: according to the data arrangement method of the output tensor, the target data corresponding to the current thread and the target data corresponding to other threads in the thread bundle are merged and written into the destination storage space.

[0015] In the data processing method provided by at least one embodiment of the present disclosure, the current thread is included in a thread bundle. After determining the index value offset corresponding to the current thread, it also includes: in response to the absence of index tensors on continuous dimensions of the output tensor data arrangement storage, determining to broadcast the index value corresponding to the index value offset to multiple threads in the thread bundle.

[0016] In the data processing method provided by at least one embodiment of the present disclosure, in response to the absence of an index tensor on a continuous dimension of the output tensor data arrangement storage, determining to broadcast the index value corresponding to the index value offset to multiple threads in the thread bundle includes: in response to the dimension of the output tensor obtained by indexing not covering the continuous dimension of the output tensor data arrangement storage, determining to broadcast the index value corresponding to the index value offset to multiple threads in the thread bundle.

[0017] In the data processing method provided in at least one embodiment of the present disclosure, the general tensor operation interface includes at least one of a general tensor read and write interface or a general coordinate processing interface, the general tensor read and write interface includes a thread-granularity read and write interface and a warp-granularity read and write interface, the thread-granularity read and write interface is configured to support broadcast operations, the warp-granularity read and write interface is configured to support merged reading and writing, the general coordinate processing interface includes at least one of a coordinate conversion interface or a coordinate calculation interface, the coordinate conversion interface is configured to realize coordinate conversion under different data arrangements, and the coordinate calculation interface is configured to calculate the address offset corresponding to the coordinate.

[0018] At least one embodiment of the present disclosure provides a data processing device, which includes: a receiving module, configured to receive signals about an input tensor and an index tensor; an execution module, configured to read target data from a source storage space corresponding to the input tensor based on the index tensor through a universal tensor operation interface, and write the target data to a destination storage space corresponding to the output tensor, wherein the universal tensor operation interface supports unified tensor operations involving at least two data arrangement modes.

[0019] At least one embodiment of the present disclosure provides an electronic device, comprising: at least one processor; at least one memory, comprising one or more computer program modules; wherein the one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the data processing method provided in at least one embodiment above.

[0020] At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by at least one processor, perform the data processing method provided by at least one embodiment above.

[0021] At least one embodiment of the present disclosure provides a data processing method, data processing device, electronic device, and non-transitory computer-readable storage medium, which proposes a universal and efficient indexing operation implementation scheme that can adapt to different data arrangements and is compatible with complex index access requirements under different hardware architectures. This significantly improves the generalization capability and execution efficiency of index operators, and can support high-efficiency, low-latency indexing operations on heterogeneous computing devices. This solves the problem that there is currently no generalized solution that supports indexing operations under multiple different data arrangements, and the high complexity of enumeration implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure.

[0023] Figure 1 A flowchart of a data processing method provided by at least one embodiment of the present disclosure;

[0024] Figure 2 A flowchart of a data processing method provided by at least one embodiment of the present disclosure;

[0025] Figure 3 A schematic diagram of the structure of a general-purpose graphics processor;

[0026] Figure 4 A schematic block diagram of a data processing device provided in at least one embodiment of the present disclosure;

[0027] Figure 5 A schematic block diagram of an electronic device provided in at least one embodiment of the present disclosure;

[0028] Figure 6 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure;

[0029] Figure 7 A schematic block diagram of a non-transitory computer-readable storage medium is provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0031] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by persons of ordinary skill in the field to which this disclosure belongs. The words "first", "second" and similar terms used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0032] The present disclosure is described below using several specific embodiments. To maintain clarity and conciseness in the following description of the embodiments of the present disclosure, detailed descriptions of known functions and components may be omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, that component is represented by the same or similar reference numeral in each drawing.

[0033] A tensor generally refers to a multidimensional array that can represent a variety of data structures, including scalars (zero-dimensional tensors), vectors (one-dimensional tensors), matrices (two-dimensional tensors), and higher-dimensional arrays. A tensor consists of multiple elements. The specific location of these elements in the tensor can be represented by coordinates. For example, a two-dimensional tensor with a shape of 1×8190 indicates that the tensor has one row and 8190 columns. Any element can be accessed based on the coordinates (i, j), where i is the row coordinate value and j is the column coordinate value. For example, a row can represent a batch dimension, and a column can represent a height dimension. For multidimensional tensors, the number and order of coordinates are related to the dimension and shape of the tensor.

[0034] Tensor indexing is a common operation in fields such as deep learning, image processing, and large-scale data computation. For example, in feature extraction, label mapping, or attention mechanisms, it is often necessary to extract data at specific locations from a source tensor (input tensor) based on a set of indices to form a new target tensor (output tensor).

[0035] The index operator is a high-level indexing operation used to extract data from an input tensor of arbitrary dimension. This operation supports data extraction according to a given integer index tensor on any dimension of the input tensor, and can be flexibly applied to data preprocessing and specific model tasks. In the field of deep learning, the index operator enables the model to efficiently access specific elements or sub-tensors in a tensor without traversing the entire data structure, thereby significantly improving computational efficiency and providing greater flexibility for complex data operations. In large-scale models such as Mixture-of-Experts (MoE), hundreds or even thousands of expert networks are usually included. In this case, an efficient index operator can achieve precise routing and data distribution between input samples and corresponding experts, supporting horizontal expansion of the model.

[0036] Take a three-dimensional input tensor of shape (x, y, z) as an example, which means that there are x elements in the 0th dimension, y elements in the 1st dimension, and z elements in the 2nd dimension. Next, we perform indexing operations on the 0th and 2nd dimensions of the input tensor. The shape of the given (for example, user-specified) index tensor is (2), that is, each index tensor is a one-dimensional tensor containing two elements.

[0037] The following is a code implementation example that extracts elements with indices [0, 2] and [1, 3] from the 0th and 2nd dimensions of the input tensor respectively:

[0038] indices_0 = [0, 2]

[0039] indices_2 = [1, 3]

[0040] output_tensor = input_tensor[indices_0, : ,indices_2]

[0041] Among them, indices_0 is the index tensor corresponding to the 0th dimension, indices_2 is the index tensor corresponding to the 2nd dimension, input_tensor is the input tensor, and output_tensor is the output tensor generated after extraction according to the specified index.

[0042] The output tensor output_tensor is a two-dimensional tensor with a shape of (2, y). Specifically, the first row of the output tensor, output_tensor[0, :], corresponds to input_tensor[0, :, 1], and the second row of the output tensor, output_tensor[1, :], corresponds to input_tensor[2, :, 3]. Input_tensor[0, :, 1] selects the 0th element of the 0th dimension, all elements of the 1st dimension, and the 1st element of the 2nd dimension in input_tensor, resulting in a one-dimensional tensor of length y. Input_tensor[2, :, 3] selects the 2nd element of the 0th dimension, all elements of the 1st dimension, and the 3rd element of the 2nd dimension in input_tensor, also a one-dimensional tensor of length y. Therefore, the shape of the output tensor output_tensor is determined to be (2, y).

[0043] The above is a logical implementation. The underlying hardware implementation involves reading and writing data. The index operator can convert the index into a linear address offset based on the dimension and stride information of the input tensor, thereby implementing the tensor indexing operation of the data.

[0044] The following is a low-level implementation of an index operator, including steps S1 to S5.

[0045] S1: Traverse all output index items and execute the following steps S2 to S5 for each output index item.

[0046] For example, an output index item refers to each addressable element in the output tensor. For example, in the above example, the output tensor shape is (2, y), which contains 2×y output index items. When the index operation is performed by a single thread, the thread will process these 2×y output index items one by one in sequence. Each output index item corresponds to a set of address offsets, including the output tensor write position offset offsets[0] (indicating where the current thread should write the index result in the output tensor), the address offset offsets[1] of the non-index dimension in the input tensor (for example, the access offset of the first dimension in the input tensor in the above example), and the address offset offsets[2] for reading the index value in the index tensor (used to indicate where the current thread should read the index value from the index tensor). The above "position" refers to the address position in the storage space.

[0047] S2: Calculate the input address and output address based on the address offset corresponding to the output index item.

[0048] For example, the output address is obtained by adding offsets[0] to the data start address of the output tensor, and the input address is obtained by adding offsets[1] to the data start address of the input tensor. The input address points to the starting access location of the non-indexed dimension in the input tensor, and the output address points to the location in the output tensor where the current thread should write the result.

[0049] S3: Calculate the cumulative address offset of the input tensor in the index dimension.

[0050] For example, traverse all index dimensions (such as the 0th and 2nd dimensions in the above example), read the index value of the output index item under the index dimension from the index tensor according to offsets[2], and calculate its address offset in the storage space according to the step size corresponding to the index dimension, and accumulate the address offsets of each index dimension to obtain the accumulated address offset.

[0051] For example, for a three-dimensional tensor with a shape of (x, y, z), the stride of the 0th dimension is y×z, the stride of the 1st dimension is z, and the stride of the 2nd dimension is 1. If the current output index item has an index value of 2 in the 0th dimension and an index value of 3 in the 2nd dimension, the calculated accumulated address offset is 2×y×z + 3×1.

[0052] S4: Obtain the total offset according to the input address and the accumulated address offset.

[0053] For example, assuming that the input address is recorded as in_data and the accumulated address offset is recorded as idx_offset, the total offset can be expressed as in_data[idx_offset].

[0054] S5: Read data from the input tensor according to the total offset and write back to the corresponding address of the output tensor.

[0055] For example, the data at in_data[idx_offset] in the input tensor is read and written to the location pointed to by the output address, thus completing the indexing operation of an element.

[0056] The arrangement of tensor elements in storage space (such as memory or video memory) is called the tensor's layout, also referred to as the tensor's data arrangement in this disclosure. A tensor's layout indicates the order and dimensionality of the tensor within the storage space, which can affect data access efficiency and the throughput of hardware computing units, thereby affecting the efficiency of hardware parallel computing.

[0057] Heterogeneous operators can be tensor operations executed on different types of hardware devices (such as graphics processing units (GPUs), tensor processing units (TPUs), deep learning processing units (DPUs), accelerated processing units (APUs), and neural network processing units (NPUs)). Heterogeneous operators can also be tensor operations executed on the same type of hardware devices from different vendors, which may differ in terms of architecture, memory hierarchy, and compute unit layout. These operations are assigned to the most appropriate device based on their computational characteristics. A well-designed layout can effectively improve the data read and write bandwidth of heterogeneous operators, thereby achieving higher performance. In theory, the layout can be format converted through software, allowing the hardware to process input data of different layouts by supporting only a single base format. However, with the continuous expansion of deep learning model size and the increasing demand for inference and training efficiency, relying solely on software conversion is no longer sufficient to meet application requirements. Therefore, to fully utilize the bandwidth potential of hardware and achieve higher read and write bandwidth, it is necessary to design and understand the tensor layout at the hardware level. In other words, by directly supporting different data arrangements through hardware, unnecessary memory access overhead and data transfer delays can be reduced.

[0058] Common tensor layout types include but are not limited to the following:

[0059] Row-major layout: Consecutive rows of data are stored sequentially in consecutive locations in storage space (e.g., memory). For example, in a two-dimensional array, all data within a row is stored contiguously, while data between different rows may have large address jumps.

[0060] Column-major layout: Consecutive columns of data are stored sequentially in consecutive memory locations. Taking a two-dimensional array as an example, all data within a column is stored contiguously, while data between different columns may have large address jumps.

[0061] Stride Layout: Defines the jump distance of tensor elements in the storage space by specifying the stride of each dimension. That is, the stride of each dimension defines how many bytes to jump to reach the next element. It can flexibly support custom layouts of any dimension.

[0062] Dimension order layout: By specifying the order in which the dimensions of a tensor are arranged, its layout in the storage space is determined. For example, taking a four-dimensional tensor as an example, including the batch (B) dimension, the channel (C) dimension, the height (H) dimension, and the width (W) dimension, the two common dimension order layouts are as follows:

[0063] NCHW layout: Arranged in the order of Batch, Channel, Height, and Width dimensions, suitable for mathematical operations on the CPU and convenient for channel-level operations.

[0064] NHWC layout: Arranged in the order of Batch, Height, Width, and Channel, suitable for GPU memory access mode.

[0065] The above-mentioned tensor layout types are merely illustrative. The tensor layouts (i.e., data arrangement methods) involved in this disclosure are not limited to the aforementioned examples. Various tensor layouts can be flexibly defined and expanded according to different application scenarios and hardware characteristics, and the embodiments of this disclosure do not impose any restrictions on this.

[0066] In hardware architectures that support multiple data layouts, address offset calculation cannot simply follow the traditional step-by-step accumulation method. Instead, it must be remapped and calculated based on the current tensor data layout and specific coordinate information. Therefore, the above traditional implementation method cannot be directly applied to tensor processing scenarios with specific data layouts.

[0067] Furthermore, index operators that support multiple data arrangements must process inputs that include not only the index representations for different data arrangements but also the combinations between multiple index dimensions. Due to the lack of a unified and universal implementation and the high complexity of the operators, enumeration-based implementations are difficult to cover all possible data arrangements and index dimension combinations. They can only handle a few typical scenarios, resulting in weak generalization of index operators.

[0068] Furthermore, layout-based hardware and software architectures often excel in compute-intensive tasks but struggle with memory-intensive operators, particularly those with discontinuous accesses, such as indexing operators. The powerful flexibility of indexing operators relies on complex processing logic, and layout design increases this complexity by nearly N cubed (N being the number of layouts). Therefore, a universal indexing solution is urgently needed in layout-based hardware and software architectures.

[0069] In summary, there is currently no universal implementation mechanism that can effectively support the functional implementation of index operators in a hardware environment where multiple data arrangements coexist. Moreover, as the complexity of operators increases, the enumeration implementation method faces significant challenges in practical applications.

[0070] At least one embodiment of the present disclosure provides a data processing method, which includes: receiving signals about an input tensor and an index tensor; reading target data from a source storage space corresponding to the input tensor based on the index tensor through a universal tensor operation interface, and writing the target data to a destination storage space corresponding to the output tensor, wherein the universal tensor operation interface supports unified tensor operations involving at least two data arrangement methods.

[0071] The data processing method provided in at least one embodiment of the present disclosure proposes a universal and efficient indexing operation implementation scheme that can adapt to different data layouts and is compatible with complex index access requirements under different hardware architectures. This significantly improves the generalization capability and execution efficiency of index operators, enabling high-efficiency, low-latency indexing operations on heterogeneous computing devices. This solves the current problem of a lack of generalized solutions supporting indexing operations under multiple data layouts and the high complexity of enumeration implementations.

[0072] Furthermore, the data processing method provided in at least one embodiment of the present disclosure proposes a broadcast strategy and a merge write strategy based on the data arrangement method, which reduces redundant tensor read and write operations, effectively improves memory access efficiency and the overall performance of the operator, and gives full play to the performance advantages of modern computing devices in large-scale parallel computing, and realizes support for index operators in high-performance deep learning scenarios.

[0073] For example, the data processing method provided in at least one embodiment of the present disclosure can be applied to a processor, and the processor may include a graphics processing unit (GPU), a tensor processing unit (TPU), a general-purpose graphics processing unit (GPGPU), a deep learning processing unit (DPU), an accelerated processing unit (APU), a neural network processing unit (NPU), etc. Of course, the embodiments of the present disclosure are not limited to this, and the processor may also be any other type of processor.

[0074] The data processing method provided in at least one embodiment of the present disclosure can be used to implement an index operator.

[0075] Figure 1 A flowchart of a data processing method provided in at least one embodiment of the present disclosure.

[0076] For example, Figure 1 As shown, the data processing method provided by at least one embodiment of the present disclosure includes the following steps S101-S102. For example, the data processing method provided by at least one embodiment of the present disclosure can be applied to a processor, and the processor can include a graphics processor or a general-purpose graphics processor. Of course, the embodiments of the present disclosure are not limited thereto, and the processor can also be any other type of processor.

[0077] Step S101: Receive signals about input tensors and index tensors.

[0078] For example, in step S101, the input tensor refers to the source tensor involved in the indexing operation, contains the data elements to be processed, and is the target object to be accessed by the indexing operation. For example, the input tensor may include at least one of voice data, image data, text data, video data, and biomedical data. The index tensor is used to specify which elements to read from the input tensor, and its content represents the index position information on a certain dimension to be indexed in the input tensor. Specific examples of input tensors and index tensors have been described above and will not be repeated here.

[0079] For example, a signal about an input tensor can be a generalized representation of the input tensor, and a signal about an index tensor can be a generalized representation of the index tensor. This generalized representation includes, but is not limited to, properties of a tensor such as its dimension, shape, data type, and data layout. For example, this generalized representation can be implemented using a structure, with the aforementioned properties serving as member variables. It should be noted that the implementation of this generalized representation is not limited to structures; other data structures or object models (such as classes) can also be used. This generalized representation helps shield underlying implementation differences, enabling unified processing and interface compatibility for various data layouts.

[0080] For example, the input to an index operator can include the tensor to be indexed (the input tensor), one or more index tensors (whose dimensions are less than or equal to the dimensions of the tensor to be indexed), and the index dimensions corresponding to each index tensor. Each index tensor has the same shape and data is arranged in the same way.

[0081] Step S102: Through the general tensor operation interface, based on the index tensor, read the target data from the source storage space corresponding to the input tensor, and write the target data to the destination storage space corresponding to the output tensor, wherein the general tensor operation interface supports unified tensor operations involving at least two data arrangement methods.

[0082] In some examples, at least two of the input tensor, index tensor, and output tensor have different data layouts in the storage space. For example, the input tensor uses a row-major layout, while the index tensor and output tensor use a column-major layout. Furthermore, the input tensor, index tensor, and output tensor can all be arranged differently in the storage space.

[0083] In other examples, the data layout of the input tensor, index tensor, and output tensor in the storage space can be exactly the same. For example, the input tensor, index tensor, and output tensor all use row-major layout.

[0084] The above is only an example. The embodiments of the present disclosure do not limit the specific data arrangement of the input tensor, index tensor, and output tensor. Each tensor can adopt any applicable layout form based on the actual application scenario.

[0085] For example, in step S102, the output tensor is the target tensor of the index operation, which is used to store the data indexed from the input tensor, and its shape generally depends on the shape of the index tensor, the dimension of the index tensor, and the shape of the input tensor. The storage space can be located in, for example, memory, video memory, cache, or other types of storage media. The source storage space refers to the storage area for storing the input tensor, such as a section of system memory; the destination storage space refers to the storage area for storing the output tensor. The source storage space and the destination storage space can be located in the same storage medium or in different storage media, depending on the actual application scenario and system architecture. The target data refers to the specific data extracted from the input tensor through the index operation and finally written to the output tensor.

[0086] The data arrangement method of the tensor is the tensor layout described above, which defines the arrangement rules of multi-dimensional tensors in the storage space, including but not limited to row-major layout, column-major layout, stride layout, and dimension-order layout. The specific description of the tensor layout has been explained in detail in the previous article and will not be repeated here. Different data arrangements differ in data access mode, memory address mapping rules, and hardware access characteristics. To adapt to diverse storage structures and hardware architectures, in actual applications, the data arrangement methods of input tensors, index tensors, and output tensors may be different.

[0087] For example, the general tensor operation interface can provide tensor operation capabilities to upper-level callers in the form of functions, providing unified operational semantics in at least two different data arrangements. The general tensor operation interface is configured to adapt to multiple data arrangements and has compatibility with multiple data arrangements, shielding the underlying data arrangement differences and thus supporting different data arrangements.

[0088] For example, unified tensor operations mean that in the upper-level logic, regardless of the data layout of the inputs to the universal tensor operation interface (such as input tensors, index tensors, and output tensors), the external operation format remains consistent. When users call the universal tensor operation interface, they can complete the required data processing tasks without having to worry about the underlying data layout of the tensor. In other words, users do not need to use different operation interfaces or adjust operation logic for different data layouts, nor do they need to write different code for different data layouts.

[0089] It should be noted that the "through the general operation interface" mentioned in step S102 refers to the unified access and operation of tensor data from the perspective of the upper-level caller with the help of this interface (that is, reading the target data from the source storage space corresponding to the input tensor based on the index tensor, and writing the target data to the destination storage space corresponding to the output tensor). It reflects the consistency of the operational semantics provided by the interface, rather than the specific underlying implementation details.

[0090] Figure 2 A flowchart of a data processing method provided in at least one embodiment of the present disclosure.

[0091] An example of step S102 includes using the general tensor operation interface to perform the following operations. For the convenience of description, each operation is recorded as step S201 to step S204, such as Figure 2 shown.

[0092] It's important to note that in this example, "Using the general tensor operation interface to perform the following operations" refers to the underlying implementation of the interface, meaning that the following operations are encapsulated within the interface. In other words, the general tensor operation interface encapsulates the actual read and write logic for different data arrangements, including specific implementations such as address offset calculations and coordinate calculations.

[0093] Step S201: Determine the output coordinates of the target data corresponding to the current thread according to the tensor attribute information of the output tensor.

[0094] Step S202: According to the tensor attribute information and output coordinates of the index tensor, determine the index value offset corresponding to the current thread to obtain the input coordinates of the target data.

[0095] Step S203: According to the tensor attribute information and input coordinates of the input tensor, determine the address offset of the target data in the input tensor and read the target data from the source storage space.

[0096] Step S204: writing the target data into the destination storage space according to the tensor attribute information of the output tensor.

[0097] As the scale of data grows, the efficiency of single-threaded indexing operations gradually becomes a performance bottleneck. In order to increase the speed of indexing operations, multi-threading technology can be used for parallel acceleration. That is, the indexing task can be divided into multiple continuous intervals, and each thread is responsible for processing the indexing task of one interval. Since each indexing operation is independent of each other and there is no cross-thread data dependency, it can be split into multiple threads for concurrent execution. The indexing operation provided by the embodiment of the present disclosure can be executed serially by a single thread, or can be processed in parallel through a multi-threading mechanism.

[0098] For example, in a single-threaded execution scenario, the "current thread" refers to the thread currently executing the indexing task; in a multi-threaded parallel execution scenario, the "current thread" refers to the thread currently being described among multiple concurrent threads. The target data corresponding to the current thread refers to the data elements processed by the thread, read from the source storage space where the input tensors are located, and written to the destination storage space where the output tensors are located.

[0099] For example, tensor attribute information may include the data arrangement of the tensor. Since the embodiments of the present disclosure can support the situation where at least two of the input tensor, index tensor, and output tensor have different data arrangement modes in the storage space, in the underlying operations recorded in the above steps S201 to S204, it is necessary to determine the corresponding processing mode according to the data arrangement mode, so as to achieve support and compatibility for different data arrangement modes. In some examples, tensor attribute information may also include the dimension, shape, data type, etc. of the tensor, which is not limited by the embodiments of the present disclosure.

[0100] For example, in step S201, the output coordinates are logical coordinates that are independent of physical storage. That is, the logical coordinates of the target data that the current thread is responsible for processing in the output tensor can be determined based on the tensor attribute information of the output tensor. For example, in a two-dimensional tensor with a shape of (2, 3), the logical coordinates [1, 2] are used to represent the elements in the first row and second column, regardless of their actual storage method in the storage space. This use of logical coordinates helps to shield the differences brought about by different data arrangement methods and achieve unified support for multiple data arrangement methods. For example, in a single-threaded execution scenario, each data item can be processed in sequence according to the actual arrangement order of the output tensor in the storage space; in a multi-threaded execution scenario, a task division method that supports memory access merging can be adopted based on the data arrangement method of the output tensor to improve memory access efficiency and overall execution performance.

[0101] For example, in step S202, the index value offset corresponding to the current thread can be determined based on the tensor attribute information of the index tensor and the output coordinates obtained in step S201. The index value offset is an address offset used to read the corresponding index value from the index tensor. Under the premise that the first address of the index tensor is known, the storage address of the index value can be determined by the index value offset. The index tensor can be stored in any storage medium, can be stored in the same storage medium as the input tensor or output tensor, or can be stored in different storage media. The embodiment of the present disclosure does not limit this. The input coordinates of the target data can be determined based on the index value offset and the output coordinates. The input coordinates are also logical coordinates that are independent of physical storage, that is, the logical coordinates of the target data that the current thread is responsible for processing in the input tensor.

[0102] For example, in step S203, the address offset of the target data in the input tensor can be determined based on the tensor attribute information of the input tensor and the input coordinates obtained in step S202. Given that the first address of the input tensor is known, the address offset can be used to determine the storage address of the target data in the source storage space. Based on this storage address, the target data can be read from the source storage space.

[0103] For example, in step S204 , the target data read from the source storage space may be written into the destination storage space in a correct format and order according to the tensor attribute information of the output tensor.

[0104] The data processing method provided by at least one embodiment of the present disclosure performs mapping and calculation according to the data arrangement and specific coordinate information of each tensor in the underlying implementation of the general tensor operation interface, thereby realizing an efficient generalization scheme for index operators that support different data arrangements.

[0105] In the data processing method provided in at least one embodiment of the present disclosure, a warp-level parallel mechanism may be used to implement indexing operations, thereby fully utilizing the parallel computing capabilities of a parallel computing processor (eg, a general-purpose graphics processor).

[0106] Figure 3 A schematic structural diagram of a general-purpose graphics processor.

[0107] like Figure 3 As shown, the general purpose graphics processor is actually an array of programmable multiprocessors. For example, the programmable multiprocessor can be a streaming processor cluster (SPC), including Figure 3Streaming processor clusters 1, ..., and M are shown, where M is a positive integer greater than 1. In a general-purpose graphics processor, one streaming processor cluster processes one computing task, or multiple streaming processor clusters process one computing task. Multiple streaming processor clusters share data through a global cache or global memory.

[0108] like Figure 3 As shown, taking stream processor cluster 1 as an example, a stream processor cluster includes multiple computing units, such as Figure 3 In the CU, there are CU 1, CU 2, ..., CU N, where N is a positive integer. Each CU is used to perform arithmetic and logical operations, such as accumulation, reduction, conventional addition, subtraction, multiplication, and division. A CU includes multiple cores (also called computing cores or computing cores), each of which includes an arithmetic and logical unit (ALU), a floating-point computing unit, etc. The computing core is used to perform specific computing tasks. In addition, the CU also includes registers (e.g. Figure 3 The register file in the computing unit and the shared memory are used to hierarchically store source data and destination data related to computing tasks. The shared memory in a computing unit is used to share data between the cores of the computing unit.

[0109] like Figure 3 As shown, each stream processor cluster also provides a buffer area (Buffer) for caching data of N computing units in the stream processor cluster.

[0110] In the process of high-performance parallel computing using parallel computing processors such as Graphics Processing Unit (GPU) or general-purpose graphics processors, a large number of threads are used to achieve high concurrency. For example, computing tasks are generally executed by multiple threads. These threads are divided into multiple thread blocks before being executed in the parallel computing processor, and then distributed through the thread block distribution module ( Figure 3(not shown) distributes multiple thread blocks to various computing units. All threads in a thread block are assigned to the same computing unit for execution. Multiple thread blocks can be executed in the same computing unit or in different computing units. At the same time, the thread blocks are split into minimum execution thread warps (or simply warps), which are the basic units of hardware scheduling. Each warp includes a fixed number of threads (or less than this fixed number), for example, 32 threads. These threads execute the same instructions in a single instruction multiple thread (SIMT) manner, but process different data. In a warp, each thread can share some register resources with other threads, but also has independent register resources for loading or calculating different data.

[0111] In each computational unit, the warp scheduling / dispatching module ( Figure 3 (not shown) schedules and allocates thread warps so that multiple cores of the compute unit can execute the warps. Depending on the number of cores in the compute unit, multiple warps in a thread block can execute simultaneously or in a time-sharing manner. Multiple threads in each warp execute the same instruction. Memory execution instructions are emitted to the shared memory of the compute unit or further emitted to the intermediate level cache, global cache, or global memory (e.g., Figure 3 The high bandwidth memory (HBM) in the memory is used for read and write operations.

[0112] In a scenario where the indexing operation is implemented using a warp-level parallel mechanism, an example of step S201 may include the following steps S2011 - S2012 .

[0113] Step S2011: Determine a task distribution method that supports continuous writing based on the data arrangement and shape of the output tensor, so that data blocks written by multiple threads in the same warp are arranged continuously in the destination storage space.

[0114] For example, in step S2011, multiple threads within the same warp need to process consecutively arranged data elements, thereby maximizing the warp's memory access consistency, reducing memory access latency, and improving bandwidth utilization. Because the shapes of the input tensors and index tensors are known, the shape of the output tensor can also be uniquely determined.

[0115] For example, in a warp containing 32 threads, if the output tensor is a two-dimensional tensor of shape (2, 64) and is stored using row-major layout, the output tensor can be divided into four consecutive data segments, each containing 32 consecutive elements. Each of the 32 threads in the warp is responsible for processing (writing) one element in its own data segment, with adjacent threads processing consecutive elements. This task distribution method aligns with the hardware characteristics of the warp, fully leveraging the advantages of memory access coalescing and improving overall execution efficiency.

[0116] Different data arrangements can lead to differences in the order and distribution of data elements in the storage space. For example, in some data arrangements, elements with adjacent logical coordinates may have relatively large address gaps in the storage space. Therefore, when performing warp-level parallel processing, it is necessary to dynamically determine the task distribution strategy suitable for the data arrangement based on the data arrangement and shape information of the output tensor to support the continuity of write operations.

[0117] Through the above method, data blocks written by multiple threads in the same warp can be arranged continuously in the destination storage space, thereby improving memory access efficiency and giving full play to the hardware parallel capability.

[0118] Step S2012: Determine the output coordinates of the target data corresponding to the current thread according to the task distribution method.

[0119] For example, in step S2012, the task distribution method may indicate the storage space area where the current thread will ultimately write the target data. Based on the data arrangement of the output tensor, a mapping relationship between the physical storage address offset and the logical coordinates of each element in the output tensor can be obtained. Based on this mapping relationship, the logical coordinates of the target data in the output tensor for which the current thread is responsible can be further determined.

[0120] In the data processing method provided in at least one embodiment of the present disclosure, an example of step S202 includes the following steps S2021 to S2023.

[0121] Step S2021: intercept the output coordinates according to the index starting dimension and the dimension of the index tensor to determine the coordinates of the index value corresponding to the current thread.

[0122] For example, the coordinates of an index value refer to the logical coordinates of the index value in the index tensor. The output coordinates can be truncated based on the index starting dimension and the dimensions of the index tensor to determine the logical coordinates of the index value that the current thread needs to read in the index tensor. The number of dimensions of the index value coordinates is equal to the dimensions of the index tensor.

[0123] For example, the index starting dimension refers to the dimension where the indexing operation is performed first. Assuming the indexing operation is performed on the 0th and 1st dimensions of the input tensor, the index starting dimension is 0. Assuming the shape of the index tensor is (x, y), the dimension of the index tensor is 2. You can extract two consecutive dimensions starting from the index starting dimension from the output coordinates and use them as the logical coordinates of the index value in the index tensor.

[0124] For example, in the above example, if the output coordinates corresponding to the current thread are [o1, o2, o3, o4], since the index starting dimension is 0 and the dimension of the index tensor is 2, the coordinates of the index value corresponding to the current thread can be determined to be [o1, o2].

[0125] Step S2022: Determine the index value offset corresponding to the current thread based on the data arrangement of the index tensor and the coordinates of the index value.

[0126] For example, the data layout of the index tensor allows us to derive the mapping between the physical storage address offset and the logical coordinates of each element in the index tensor. The logical coordinates of the index value can be used to determine the corresponding address offset, also known as the index value offset. The index value offset can be used to read the corresponding index value from the index tensor.

[0127] In the data processing method provided in at least one embodiment of the present disclosure, for the same target data, each index tensor has the same index offset. Therefore, the current thread can use the same index offset to read the required index value from different index tensors, effectively improving computational efficiency.

[0128] Step S2023: Determine the input coordinates of the target data according to the index value offset and the output coordinates.

[0129] An example of step S2023 includes: starting from the 0th dimension, accessing each dimension of the input tensor in sequence, and executing the following steps S2023a to S2023b.

[0130] Step S2023a: In response to the existence of an index tensor in the current dimension, the value of the input coordinate in the current dimension is determined based on the index value offset and the number of index tensors before the current dimension.

[0131] For example, in step S2023a, if the current dimension requires index processing, it is considered that an index tensor exists for the current dimension. The number of index tensors before the current dimension indicates how many dimensions before the current dimension require index processing.

[0132] For example, a binary representation can be used to predefine which dimensions need to be indexed, where each bit corresponds to a dimension of the input tensor. A value of 1 indicates that the dimension needs to be indexed, that is, an index tensor exists for the dimension; a value of 0 indicates that the dimension does not need to be indexed, that is, an index tensor does not exist for the dimension. For example, if the binary representation dims=0b011 is set, it means that the 0th and 1st dimensions need to be indexed, while the 2nd dimension does not need to be indexed. This method provides a flexible and compact multi-dimensional index configuration, which helps to quickly parse and determine the index requirements of the current dimension, and further determine the corresponding index tensor. In step S2023a, it can be determined whether there is an index tensor for the current dimension by analyzing the above binary representation.

[0133] For example, suppose an index operation is performed on the 0th and 2nd dimensions of a three-dimensional input tensor. If the current dimension is the 0th or 2nd dimension, it is determined that an index tensor exists for the current dimension; if the current dimension is the 1st dimension, it is determined that no index tensor exists for the current dimension.

[0134] For example, suppose the index operation is performed on the 0th and 2nd dimensions of a four-dimensional input tensor, and the current dimension is the 0th dimension, then the number of index tensors before the current dimension is 0. If the current dimension is the 1st or 2nd dimension, then the number of index tensors before the current dimension is 1 (referring to the index tensor corresponding to the 0th dimension). If the current dimension is the 3rd dimension, then the number of index tensors before the current dimension is 2 (referring to the index tensors corresponding to the 0th and 2nd dimensions).

[0135] An example of "determining the value of the input coordinate in the current dimension based on the index value offset and the number of index tensors before the current dimension" in step S2023a may include the following steps S301 to S302.

[0136] Step S301: Determine the index tensor corresponding to the current dimension based on the number of index tensors before the current dimension.

[0137] For example, the number of index tensors preceding the current dimension indicates how many dimensions require indexing. Based on this, we can determine which index tensor to read the index value from for the current dimension, based on multiple index tensors (corresponding to different dimensions to be indexed). Multiple index tensors can be organized sequentially into index tensor sets to enable ordered access to index tensors.

[0138] For example, assuming that the number of index tensors before the current dimension is 2, the index tensor corresponding to the current dimension can be the third index tensor among the multiple index tensors, that is, the third item in the index tensor set.

[0139] Step S302: Read the index value in the index tensor corresponding to the current dimension according to the index value offset, and use it as the value of the input coordinate in the current dimension.

[0140] For example, in step S301, it has been determined which index tensor the current thread should read the index value from. Since the current thread can use the same index value offset to read the required index value from different index tensors, the index value can be read from the determined index tensor according to the index value offset obtained in step S2022. Further, the read index value is used as the value of the input coordinate in the current dimension.

[0141] For example, assuming that the current dimension is the first dimension, the index value read through the above step S302 is idx_value, and the input coordinates corresponding to the current thread are [i1, i2, i3], then the value of the input coordinate in the current dimension (first dimension) i2=idx_value.

[0142] Step S2023b: In response to the absence of an index tensor in the current dimension, the value of the input coordinate in the current dimension is determined based on the value of the output coordinate in the corresponding dimension.

[0143] The above section describes how to determine whether an index tensor exists in the current dimension, so we will not go into details here.

[0144] An example of “determining the value of the input coordinate in the current dimension according to the value of the output coordinate in the corresponding dimension” in step S2023b may include the following steps S303 to S304.

[0145] Step S303: In response to the absence of an index tensor before the current dimension, the value of the output coordinate in the current dimension is used as the value of the input coordinate in the current dimension.

[0146] For example, in step S303, if there is no index tensor in the current dimension and there is no index tensor before the current dimension, that is, no index processing is required in the current dimension and the dimensions before the current dimension, then the value of the output coordinate in the current dimension is used as the value of the input coordinate in the current dimension.

[0147] For example, suppose an index operation is performed on the first dimension of a three-dimensional input tensor. If the current dimension is the 0th dimension, then it is assumed that no index tensor exists for the current dimension or the dimension before it. If the current dimension is the 2nd dimension, then it is assumed that no index tensor exists for the current dimension, but an index tensor exists for the dimension before it (referring to the index tensor corresponding to the 1st dimension). If the output coordinates corresponding to the current thread are [o1, o2, o3, o4] and the input coordinates corresponding to the current thread are [i1, i2, i3], then the value i1 of the input coordinate in the 0th dimension is equal to the value o1 of the output coordinate in the 0th dimension.

[0148] Step S304: In response to the existence of an index tensor before the current dimension, the value of the output coordinate in the Xth dimension is used as the value of the input coordinate in the current dimension, where X = current dimension + dimension of the index tensor - number of index tensors before the current dimension.

[0149] For example, in step S304, if there is no index tensor in the current dimension, but there is an index tensor before the current dimension, that is, index processing is not required in the current dimension and index processing is required in the dimension before the current dimension, then the value of the output coordinate in the Xth dimension is used as the value of the input coordinate in the current dimension.

[0150] For example, we can assume that an index operation is performed on the first dimension of a three-dimensional input tensor. If the current dimension is the second dimension, it is assumed that there is no index tensor for the current dimension, but there is an index tensor before the current dimension (referring to the index tensor corresponding to the first dimension). Assuming the dimension of the index tensor is 2, since the current dimension is 2, the number of index tensors before the current dimension is 1 (referring to the index tensor corresponding to the first dimension), then X = current dimension + dimension of index tensor - number of index tensors before the current dimension = 2 + 2 - 1 = 3. If the output coordinates corresponding to the current thread are [o1, o2, o3, o4] and the input coordinates corresponding to the current thread are [i1, i2, i3], then the value i3 of the input coordinate in the second dimension is equal to the value o4 of the output coordinate in the X dimension (i.e., the third dimension).

[0151] For example, we can assume that the indexing operation is performed on the 0th and 2nd dimensions of a 4-D input tensor.

[0152] If the current dimension is the 1st dimension, it is assumed that there is no index tensor for the current dimension, but there is an index tensor before the current dimension (referring to the index tensor corresponding to the 0th dimension). Assuming the dimension of the index tensor is 2, since the current dimension is 1, the number of index tensors before the current dimension is 1 (referring to the index tensor corresponding to the 0th dimension), then X = current dimension + dimension of index tensor - number of index tensors before the current dimension = 1 + 2 - 1 = 2. If the output coordinates corresponding to the current thread are [o1, o2, o3, o4], and the input coordinates corresponding to the current thread are [i1, i2, i3, i4], then the value i2 of the input coordinate in the 1st dimension is equal to the value o3 of the output coordinate in the Xth dimension (i.e., the 2nd dimension).

[0153] If the current dimension is the third, it is assumed that there is no index tensor for the current dimension, but there is an index tensor before the current dimension (referring to the index tensors corresponding to the 0th and 2nd dimensions). Assuming the dimension of the index tensor is 2, since the current dimension is 3, the number of index tensors before the current dimension is 2 (referring to the index tensors corresponding to the 0th and 2nd dimensions). Then X = current dimension + dimension of index tensor - number of index tensors before the current dimension = 3 + 2 - 2 = 3. If the output coordinates corresponding to the current thread are [o1, o2, o3, o4], and the input coordinates corresponding to the current thread are [i1, i2, i3, i4], then the value i4 of the input coordinate in the third dimension is equal to the value o4 of the output coordinate in the X dimension (also the third dimension).

[0154] The total output dimension can also be calculated by referring to the above formula, that is, the total output tensor dimension = the total input tensor dimension + the dimension of the index tensor - the number of index tensors before the total input tensor dimension.

[0155] An example of step S203 includes the following steps S2031 and S2032.

[0156] Step S2031: Determine the address offset of the target data in the input tensor according to the data arrangement mode and input coordinates of the input tensor.

[0157] For example, the data layout of the input tensor can be used to determine the mapping between the physical storage address offset and the logical coordinates of each element in the input tensor. Based on this mapping, the address offset of the target data corresponding to the current thread in the input tensor can be further determined based on the input coordinates.

[0158] Step S2032: Read target data from the source storage space according to the address offset.

[0159] For example, under the premise that the first address of the input tensor is known, the storage address of the target data in the source storage space can be determined by the above address offset, so that the target data can be read from the source storage space according to the storage address.

[0160] In a scenario where the indexing operation is implemented using a warp-level parallel mechanism, an example of step S204 may include: merging target data corresponding to the current thread and target data corresponding to other threads in the warp into a destination storage space according to a data arrangement of the output tensor.

[0161] If a task distribution method supporting continuous writes was adopted in step S2011, the data to be written by multiple threads within the current warp is contiguous in the destination storage space. Accordingly, in this example, this target data can be consolidated into a single warp-sized batch write operation, significantly improving memory access efficiency. This approach not only enhances the universal tensor operation interface's support for heterogeneous hardware architectures but also effectively improves execution performance for large-scale parallel tasks.

[0162] In a scenario where the warp-level parallelism mechanism is used to implement the index operation, after the index value offset corresponding to the current thread is determined in step S203 , the following step S205 may be executed.

[0163] Step S205 : In response to the absence of an index tensor on a continuous dimension of the output tensor data arrangement storage, determine to broadcast the index value corresponding to the index value offset to multiple threads in the warp.

[0164] For example, in step S205, the dimension in which the output tensor data is arranged and stored continuously refers to which dimension of the output tensor data is stored continuously in the storage space. This dimension can be determined based on the data arrangement of the output tensor. For different data arrangement modes, the dimension in which the data is arranged and stored continuously may be different.

[0165] For example, if it is determined that a broadcast operation can be performed, thread No. 0 in the warp can read the index values ​​of all index tensors and broadcast them to other threads in the warp.

[0166] Take an output tensor of shape (n, h, w) with row-major layout as an example, which stores contiguous values ​​along the w dimension.

[0167] If the output tensor data is arranged and stored in continuous dimensions, that is, if there is no index tensor for the input dimension corresponding to the w dimension, thread 0 in the warp can read the index values ​​of all index tensors and broadcast them to other threads in the warp. This is because each thread in the same warp processes a continuous block of output memory, and adjacent threads will access continuous w-dimensional data. Therefore, thread 0 can read the corresponding index values ​​for all n and h dimensions and broadcast them to other threads in the warp, reducing repeated read requests for index tensors and improving memory access efficiency.

[0168] On the contrary, if there is an index tensor on the input dimension corresponding to the w dimension, each thread needs to read the index tensor to determine the coordinates on the w dimension. The index values ​​corresponding to each thread on the w dimension are different, so the index values ​​cannot be shared, and no broadcast operation is performed at this time.

[0169] By broadcasting index values, multiple threads within a warp can share index values, effectively reducing repeated read requests for index tensors. This approach significantly reduces memory bandwidth pressure and improves parallel execution efficiency.

[0170] An example of step S205 includes: in response to the dimension of the output tensor obtained by indexing not covering a continuous dimension of the output tensor data arrangement storage, determining to broadcast the index value corresponding to the index value offset to multiple threads in the thread warp.

[0171] For example, assuming the input tensor has a shape of (3, 5) and the two-dimensional index tensor has a shape of (2, 2), performing the indexing operation on the first dimension will result in a three-dimensional output tensor with a shape of (3, 2, 2). In this example, the indexing starts at 1 and the index tensor dimension size is 2, so the first and second dimensions of the output tensor are both the dimensions of the output tensor obtained through indexing. Assuming that the output tensor is stored in a row-major layout, the second dimension of the output tensor is a continuous dimension for data arrangement and storage. At this point, it can be determined that the dimensions of the output tensor obtained through indexing cover the continuous dimensions of the output tensor data arrangement and storage, and the index value is determined not to be broadcast.

[0172] For example, assuming the input tensor has a shape of (3, 5) and the two-dimensional index tensor has a shape of (2, 2), performing the index operation on the 0th dimension will result in a three-dimensional output tensor with a shape of (2, 2, 5). In this example, the index starts at 0 and the index tensor dimension size is 2, so the 0th and 1st dimensions of the output tensor are both the dimensions of the output tensor obtained through indexing. Assuming that the output tensor is stored in a row-major layout, the 2nd dimension of the output tensor is a continuous dimension for data arrangement and storage. At this point, it can be determined that the dimension of the output tensor obtained through indexing does not overlap the continuous dimension of the output tensor data arrangement and storage, and the broadcast index value is determined.

[0173] In the data processing method provided in at least one embodiment of the present disclosure, a general tensor operation interface can be encapsulated in the generalized representation described above. The generalized representation described above is a data structure or object model that can describe tensor attributes and operation behaviors. For example, a class can be defined that includes data members that describe tensor attributes and functions for operating on these data, thereby providing better scalability and compatibility.

[0174] Therefore, the data processing method provided by the embodiment of the present disclosure can support all permutations and combinations (including combinations of different data permutations and multiple index dimension combinations) in the form of template functions, for example, and can solve the problem of low support rate of index operators in enumeration schemes, while reducing a large amount of development manpower investment and reducing the learning cost of complex operators.

[0175] For example, the general tensor operation interface may include at least one of a general tensor read and write interface or a general coordinate processing interface, the general tensor read and write interface includes a thread-granularity read and write interface and a warp-granularity read and write interface, the thread-granularity read and write interface is configured to support broadcast operations, and the warp-granularity read and write interface is configured to support merged reading and writing.

[0176] Among them, the warp-granularity read and write interface supports read and write merging, which helps improve memory access efficiency under large-scale parallel computing and can be dynamically adapted according to the read and write characteristics of different hardware platforms; the thread-granularity interface can implement flexible read and write modes, such as supporting broadcast mechanisms, and usually uses the traditional memory access mode based on the first address pointer plus address offset to implement reading and writing.

[0177] For example, the universal coordinate processing interface may include at least one of a coordinate conversion interface or a coordinate calculation interface, the coordinate conversion interface is configured to implement coordinate conversion under different data arrangements, and the coordinate calculation interface is configured to calculate the address offset corresponding to the coordinate.

[0178] The following is a pseudo code example of a data processing method provided by at least one embodiment of the present disclosure, which may correspond to the above step S202:

[0179] int idx_count = 0;

[0180] int out_count = 0;

[0181] size _t pos = index.toAbsPos(&coord_out[index_at_dim])

[0182] for (int i = 0; i < dims_in; ++i) {

[0183] if ((dims & (1 << i))) {

[0184] coord_in[i] = index.template loadPos <broadcastidx>(pos, idx_count);

[0185] if (i == index_at_dim) {

[0186] out_count += dims_idx;

[0187] }

[0188] idx_count += 1;

[0189] } else {

[0190] coord_in[i] = coord_out[out_count];

[0191] out_count += 1;

[0192] }

[0193] }

[0194] The variable index in the pseudocode above is an example of a generalized representation of an index tensor, encapsulating its attributes (such as data layout, dimensions, shape, data type, etc.) and a general tensor operation interface. toAbsPos() is an example of a coordinate calculation interface, which internally calculates index value offsets based on the data layout and coordinate parameters. loadPos() is a data layout-independent wrapper used to read index values ​​from an index tensor, enabling broadcasting. The variables idx_count and out_count are used to count and implement the solution in step S2023. The variable idx_count indicates the number of index tensors preceding the current dimension; the variable pos indicates the index value offset; the variable dims_in indicates the dimension of the input tensor; the variable dims is the binary representation described in step S2023a, indicating the dimension to index; the variable coord_in indicates the input coordinates; the variable coord_out indicates the output coordinates; and the variable index_at_dim indicates the starting dimension for indexing.

[0195] The following is a specific example of the underlying operations of a general tensor operation interface provided by the data processing method provided by at least one embodiment of the present disclosure and the above pseudo code.

[0196] In this example, assume the input tensor Input has a shape of (3, 4, 5), the two index tensors index_0 and index_1 index the 0th and 1st dimensions, respectively, and both have shapes of (2, 2), and the output tensor Output has a shape of (2, 2, 5). Correspondingly, the binary representation of dims has a value of 0b011, indicating that there are indices in the 0th and 1st dimensions. In this example, the data layout of any two of the input tensor, index tensor, and output tensor is different. It should be noted that in some examples, the data layout of the input tensor, index tensor, and output tensor can also be the same.

[0197] Execute step S2011 to determine a distribution method that supports continuous writing based on the data arrangement and shape of the output tensor, so that the data blocks written by multiple threads in the same thread warp are arranged continuously in the destination storage space. In this example, assuming that the output tensor is stored in row-major order and each thread warp includes 32 threads, the task can be distributed to multiple thread warps, with each thread warp processing 32 consecutive data. For example, in a thread warp, the output coordinates corresponding to a thread may be [0, 0, 0], and the output coordinates corresponding to the next adjacent thread may be [0, 0, 1]. It should be noted that since the output tensor can have various data arrangements, the output coordinates corresponding to the next adjacent thread may also be coordinates other than [0, 0, 1], such as [0, 1, 0] or [1, 0, 0].

[0198] Execute step S2012 to determine the output coordinates of the target data corresponding to the current thread based on the task distribution method. It can be assumed that the output coordinates coord_out corresponding to the current thread are determined to be [o1, o2, o3] based on the task distribution method. In other words, the current thread needs to store the target data at the address corresponding to the output coordinates [o1, o2, o3] in the destination storage space.

[0199] Execute step S2021, intercept the output coordinates [o1, o2, o3] according to the index starting dimension 0 and the dimension 2 of the index tensor, that is, intercept two consecutive dimensions starting from the index starting dimension 0 from the output coordinates to obtain the index value coordinates [o1, o2] corresponding to the current thread.

[0200] Execute step S2022 to determine the index value offset corresponding to the current thread based on the arrangement of the index tensor and the coordinates of the index value.

[0201] For example, the operations of steps S2021 to S2022 may be implemented by the toAbsPos() interface in the above pseudo-code example.

[0202] Next, we access each dimension of the input tensor in turn, starting from dimension 0.

[0203] Since there is an index tensor in dimension 0, step S2023a is executed to determine the index tensor corresponding to dimension 0, that is, index_0, based on the number of index tensors before dimension 0. The index value idx_value_0 is read from index_0 based on the index value offset as the value of the input coordinate in dimension 0. Assuming that the input coordinates corresponding to the current thread are coord_in = [i1, i2, i3], then the value of the input coordinates in dimension 0, coord_in[0] = i1 = idx_value_0.

[0204] Since there is an index tensor in the first dimension, step S2023a is executed to determine the index tensor corresponding to the first dimension, that is, index_1, based on the number of index tensors before the first dimension, and read the index value idx_value_1 in index_1 according to the index value offset as the value of the input coordinate in the first dimension, that is, coord_in[1]=i2=idx_value_1.

[0205] For example, the operation of step S2023a can be implemented by encapsulating loadPos() in the pseudocode example above. In some examples, step S205 can be performed before executing step S2023a. If it is determined that no index tensors exist in the continuous dimension of the output tensor data arrangement, then the index value corresponding to the index value offset can be broadcast to multiple threads in the warp. In this case, thread 0 in the warp can read the index values ​​of all index tensors and broadcast them to other threads in the warp.

[0206] Since there is no index tensor in the second dimension, step S2023b is executed. If there is an index tensor before the second dimension, step S304 is executed, and the value of the output coordinate in the Xth dimension is used as the value of the input coordinate in the second dimension. X = current dimension + dimension of index tensor - number of index tensors before current dimension = 2 + 2 - 2 = 2, so coord_in[2] = i3 = coord_out[2] = o3.

[0207] Therefore, we can get the input coordinates coord_in = [i1, i2, i3] = [idx_value_0, idx_value_1, o3]. That is, the target data read from the source storage space corresponding to the input coordinates coord_in = [idx_value_0, idx_value_1, o3] will be written into the destination storage space corresponding to the output coordinates coord_out = [o1, o2, o3].

[0208] Next, execute step S2031 to determine the address offset of the target data in the input tensor according to the data arrangement of the input tensor and the input coordinates [idx_value_0, idx_value_1, o3]; execute step S2032 to read the target data from the source storage space according to the address offset.

[0209] Next, step S204 is executed to merge the target data to be written by multiple threads in the current warp into a batch write operation of warp granularity according to the data arrangement of the output tensor, and write the data into the corresponding destination storage space.

[0210] The data processing method provided in at least one embodiment of the present disclosure proposes a universal and efficient indexing operation implementation scheme that can adapt to different data layouts and is compatible with complex index access requirements under different hardware architectures. This significantly improves the generalization capability and execution efficiency of index operators, enabling high-efficiency, low-latency indexing operations on heterogeneous computing devices. This solves the current problem of a lack of generalized solutions supporting indexing operations under multiple data layouts and the high complexity of enumeration implementations.

[0211] Furthermore, combined with the broadcast strategy and merge write strategy based on data arrangement, this method effectively reduces redundant tensor read and write operations, improves memory access efficiency and the overall performance of the operator, fully leverages the performance advantages of modern computing devices in large-scale parallel computing, and realizes support for index operators in high-performance deep learning scenarios.

[0212] It should also be noted that, in the various embodiments of the present disclosure, the order in which the steps of the data processing method are executed is not limited. Although the execution process of the steps is described above in a specific order, this does not constitute a limitation on the embodiments of the present disclosure. The steps in the data processing method can be executed serially or in parallel, depending on actual needs.

[0213] For example, compared with the above description, the data processing method provided by at least one embodiment of the present disclosure may also include more or fewer steps, and the embodiments of the present disclosure are not limited to this.

[0214] Figure 4 A schematic block diagram of a data processing device provided in at least one embodiment of the present disclosure.

[0215] The data processing device may be, for example, a processor or be arranged in a processor. The processor may include a graphics processing unit (GPU), a tensor processing unit (TPU), a general-purpose graphics processing unit (GPGPU), a deep learning processing unit (DPU), an accelerated processing unit (APU), a neural network processing unit (NPU), etc. Of course, the embodiments of the present disclosure are not limited to this, and the processor may also be any other type of processor.

[0216] The data processing device provided by at least one embodiment of the present disclosure can be applied to the implementation of index operators in deep learning networks.

[0217] For example, Figure 4 As shown, the data processing device provided by at least one embodiment of the present disclosure may include a receiving module 401 and an execution module 402 .

[0218] For example, the receiving module 401 is configured to receive signals regarding an input tensor and an index tensor.

[0219] For example, the execution module 402 is configured to read the target data from the source storage space corresponding to the input tensor based on the index tensor through a general tensor operation interface, and write the target data to the destination storage space corresponding to the output tensor, wherein the general tensor operation interface supports unified tensor operations involving at least two data arrangement methods.

[0220] For example, in at least one embodiment of the present disclosure, the execution module 402 includes an output determination unit, an input determination unit, a reading unit, and a writing unit. The output determination unit is configured to determine the output coordinates of the target data corresponding to the current thread based on the tensor attribute information of the output tensor; the input determination unit is configured to determine the index value offset corresponding to the current thread based on the tensor attribute information and the output coordinates of the index tensor to obtain the input coordinates of the target data; the reading unit is configured to determine the address offset of the target data in the input tensor based on the tensor attribute information and the input coordinates of the input tensor and read the target data from the source storage space; the writing unit is configured to write the target data to the destination storage space based on the tensor attribute information of the output tensor, wherein the tensor attribute information includes the data arrangement method. The output determination unit, the input determination unit, the reading unit, and the writing unit are the underlying implementations of the general tensor operation interface.

[0221] For example, in at least one embodiment of the present disclosure, the output determination unit is further configured to: determine a task distribution method that supports continuous writing based on the data arrangement and shape of the output tensor, so that data blocks written by multiple threads in the same thread bundle are arranged continuously in the destination storage space; and determine the output coordinates of the target data corresponding to the current thread based on the task distribution method.

[0222] For example, in at least one embodiment of the present disclosure, the input determination unit is further configured to: intercept the output coordinates according to the index starting dimension and the dimension of the index tensor to determine the coordinates of the index value corresponding to the current thread; determine the index value offset corresponding to the current thread according to the data arrangement method of the index tensor and the coordinates of the index value; determine the input coordinates of the target data according to the index value offset and the output coordinates.

[0223] For example, in at least one embodiment of the present disclosure, the input determination unit is further configured to: starting from the 0th dimension, access each dimension of the input tensor in sequence: in response to the existence of an index tensor in the current dimension, determine the value of the input coordinate in the current dimension according to the index value offset and the number of index tensors before the current dimension; in response to the absence of an index tensor in the current dimension, determine the value of the input coordinate in the current dimension according to the value of the output coordinate in the corresponding dimension.

[0224] For example, in at least one embodiment of the present disclosure, the input determination unit is further configured to: determine the index tensor corresponding to the current dimension based on the number of index tensors before the current dimension; read the index value in the index tensor corresponding to the current dimension based on the index value offset, and use it as the value of the input coordinate in the current dimension.

[0225] For example, in at least one embodiment of the present disclosure, the input determination unit is further configured to: in response to the absence of an index tensor before the current dimension, use the value of the output coordinate in the current dimension as the value of the input coordinate in the current dimension; in response to the presence of an index tensor before the current dimension, use the value of the output coordinate in the Xth dimension as the value of the input coordinate in the current dimension, where X = current dimension + dimension of the index tensor - number of index tensors before the current dimension.

[0226] For example, in at least one embodiment of the present disclosure, for the same target data, the index value offset corresponding to each index tensor is the same.

[0227] For example, in at least one embodiment of the present disclosure, the reading unit is further configured to: determine the address offset of the target data in the input tensor based on the data arrangement mode and input coordinates of the input tensor; and read the target data from the source storage space based on the address offset.

[0228] For example, in at least one embodiment of the present disclosure, the current thread is included in a thread bundle, and the write unit is further configured to: merge the target data corresponding to the current thread and the target data corresponding to other threads in the thread bundle into the destination storage space according to the data arrangement of the output tensor.

[0229] For example, in at least one embodiment of the present disclosure, the current thread is included in a thread bundle, and the execution module further includes a broadcast unit, which is configured to, after determining the index value offset corresponding to the current thread, determine to broadcast the index value corresponding to the index value offset to multiple threads in the thread bundle in response to the absence of index tensors on continuous dimensions of the output tensor data arrangement storage.

[0230] For example, in at least one embodiment of the present disclosure, the broadcast unit is further configured to determine to broadcast the index value corresponding to the index value offset to multiple threads in the thread warp in response to the dimension of the output tensor obtained by indexing not covering the continuous dimension of the output tensor data arrangement storage.

[0231] For example, in at least one embodiment of the present disclosure, the general tensor operation interface includes at least one of a general tensor read and write interface or a general coordinate processing interface, the general tensor read and write interface includes a thread-granularity read and write interface and a warp-granularity read and write interface, the thread-granularity read and write interface is configured to support broadcast operations, the warp-granularity read and write interface is configured to support merged reading and writing, the general coordinate processing interface includes at least one of a coordinate conversion interface or a coordinate calculation interface, the coordinate conversion interface is configured to implement coordinate conversion under different data arrangements, and the coordinate calculation interface is configured to calculate the address offset corresponding to the coordinate.

[0232] It should be noted that the above-mentioned various modules and units can be implemented by software, hardware, firmware or any combination thereof. For example, the receiving module and the execution module can be implemented as a receiving circuit and an execution circuit respectively. The embodiments of this disclosure do not limit their specific implementation methods.

[0233] It should be understood that the data processing device 400 provided in at least one embodiment of the present disclosure can be used to implement the aforementioned data processing method, and can also achieve technical effects similar to those of the aforementioned data processing method, which will not be elaborated here.

[0234] It should be noted that in the embodiments of the present disclosure, the data processing device 400 may include more or fewer modules or units, and the connection relationship between the modules or units is not limited and can be determined according to actual needs. The specific configuration of each module or unit is not limited and can be composed of analog devices according to circuit principles, or can be composed of digital chips, or constructed in other applicable ways.

[0235] Figure 5 A schematic block diagram of an electronic device provided in accordance with at least one embodiment of the present disclosure.

[0236] For example, Figure 5 As shown, electronic device 500 includes at least one processor 501 and at least one memory 502. The at least one memory 502 includes one or more computer program modules. The one or more computer program modules are stored in memory 502 and configured to be executed by the at least one processor 501. The one or more computer program modules include instructions for executing the above-described data processing method. When executed by the at least one processor 501, the one or more computer program modules can perform one or more steps of the data processing method provided in at least one embodiment of the present disclosure. The memory 502 and processor 501 can be interconnected via a bus system and / or other form of connection mechanism (not shown).

[0237] For example, the processor 501 may be a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an artificial intelligence (AI) accelerator, or other processing units with data processing and / or program execution capabilities, such as a field programmable gate array (FPGA). For example, the central processing unit (CPU) may be an X86, ARM, or RISC-V architecture. The processor 501 may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device 500 to perform desired functions.

[0238] For example, memory 502 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, a flash memory, and the like.

[0239] Figure 6 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0240] The electronic devices in at least one embodiment of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0241] The electronic device includes at least one processor and memory. The processor may be referred to as processing device 601 below, and the memory may include at least one of read-only memory (ROM), random access memory (RAM), and storage device 608 below. The memory is used to store programs for executing the methods described in the above-mentioned method embodiments; the processor is configured to execute the programs stored in the memory. The processor may include a central processing unit (CPU) or other processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0242] like Figure 6 As shown, electronic device 600 may include a processing device 601 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) or programs loaded from a storage device 608 into a random access memory (RAM). Various programs and data required for the operation of electronic device 600 are also stored in RAM 603. Processing device 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface is also connected to bus 604.

[0243] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a display, a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0244] In particular, according to at least one embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, at least one embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of at least one embodiment of the present disclosure are performed.

[0245] It should be noted that the computer-readable medium described above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In at least one embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. Furthermore, in at least one embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, radio frequency (RF), or any suitable combination thereof.

[0246] The computer-readable medium may be included in the electronic device 600 , or may exist independently without being incorporated into the electronic device 600 .

[0247] Figure 7 A schematic block diagram of a non-transitory computer-readable storage medium is provided for at least one embodiment of the present disclosure.

[0248] For example, Figure 7 As shown, a non-transitory computer-readable storage medium 700 stores computer-readable instructions 701 , which, when executed by at least one processor, perform one or more steps of the above-mentioned data processing method.

[0249] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a flash memory, or any combination of the above storage media, or other suitable storage media. For example, the readable storage medium may also be Figure 5 For the memory 502 in the embodiment, the related description can be referred to the aforementioned content and will not be repeated here.

[0250] Although the present disclosure has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications or improvements may be made based on the embodiments of the present disclosure. Therefore, such modifications or improvements, as long as they do not depart from the spirit of the present disclosure, are within the scope of protection claimed by the present disclosure.

[0251] Regarding this disclosure, the following points need to be explained:

[0252] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure. Other structures may refer to conventional designs.

[0253] (2) For the sake of clarity, the thickness of layers or regions in the drawings used to describe the embodiments of the present disclosure are enlarged or reduced, that is, these drawings are not drawn according to the actual scale.

[0254] (3) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to form new embodiments.

[0255] The above description is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.< / broadcastidx>

Claims

1. A data processing method, characterized in that: The data processing method includes: Receive signals about input tensors and index tensors; Through the general tensor operation interface, based on the index tensor, the target data is read from the source storage space corresponding to the input tensor, and the target data is written to the destination storage space corresponding to the output tensor. The general tensor operation interface supports unified tensor operations involving at least two data arrangement modes.

2. The data processing method according to claim 1, wherein: The method of reading target data from a source storage space corresponding to the input tensor based on the index tensor through a general tensor operation interface and writing the target data to a destination storage space corresponding to the output tensor includes: Use the general tensor operation interface to perform the following operations: Determine the output coordinates of the target data corresponding to the current thread according to the tensor attribute information of the output tensor; Determine, according to the tensor attribute information of the index tensor and the output coordinates, an index value offset corresponding to the current thread to obtain the input coordinates of the target data; Determine, according to the tensor attribute information of the input tensor and the input coordinates, an address offset of the target data in the input tensor and read the target data from a source storage space; Writing the target data into the destination storage space according to the tensor attribute information of the output tensor, The tensor attribute information includes data arrangement.

3. The data processing method according to claim 2, characterized in that: The current thread is included in a warp, and determining output coordinates of target data corresponding to the current thread according to tensor attribute information of the output tensor includes: Determining a task distribution method that supports continuous writing based on the data arrangement and shape of the output tensor, so that data blocks written by multiple threads in the same warp are continuously arranged in the destination storage space; According to the task distribution mode, output coordinates of the target data corresponding to the current thread are determined.

4. The data processing method according to claim 2, wherein: The determining, based on the tensor attribute information of the index tensor and the output coordinates, an index value offset corresponding to the current thread to obtain the input coordinates of the target data includes: Intercepting the output coordinates according to the index starting dimension and the dimension of the index tensor to determine the coordinates of the index value corresponding to the current thread; Determine the index value offset corresponding to the current thread according to the data arrangement of the index tensor and the coordinates of the index value; The input coordinates of the target data are determined according to the index value offset and the output coordinates.

5. The data processing method according to claim 4, characterized in that: The step of determining the input coordinates of the target data according to the index value offset and the output coordinates includes: Starting from dimension 0, access each dimension of the input tensor in turn: In response to the existence of an index tensor in the current dimension, determining a value of the input coordinate in the current dimension according to the index value offset and the number of index tensors before the current dimension; In response to the absence of an index tensor in the current dimension, the value of the input coordinate in the current dimension is determined according to the value of the output coordinate in the corresponding dimension.

6. The data processing method according to claim 5, characterized in that: The determining the value of the input coordinate in the current dimension according to the index value offset and the number of index tensors before the current dimension includes: Determine the index tensor corresponding to the current dimension according to the number of index tensors before the current dimension; The index value is read from the index tensor corresponding to the current dimension according to the index value offset, and is used as the value of the input coordinate in the current dimension.

7. The data processing method according to claim 5, characterized in that: The determining the value of the input coordinate in the current dimension according to the value of the output coordinate in the corresponding dimension includes: In response to the absence of an index tensor before the current dimension, taking the value of the output coordinate in the current dimension as the value of the input coordinate in the current dimension; In response to the existence of an index tensor before the current dimension, the value of the output coordinate in the Xth dimension is used as the value of the input coordinate in the current dimension, where X = current dimension + dimension of index tensor - number of index tensors before the current dimension.

8. The data processing method according to claim 2, wherein: For the same target data, the index value offset corresponding to each index tensor is the same.

9. The data processing method according to claim 2, wherein: The determining, based on the tensor attribute information of the input tensor and the input coordinates, the address offset of the target data in the input tensor and reading the target data from the source storage space includes: Determining an address offset of the target data in the input tensor according to a data arrangement mode of the input tensor and the input coordinates; The target data is read from the source storage space according to the address offset.

10. The data processing method according to claim 2, wherein: The current thread is included in a warp, and the writing the target data into the destination storage space according to tensor attribute information of the output tensor includes: According to the data arrangement of the output tensor, the target data corresponding to the current thread and the target data corresponding to other threads in the thread warp are merged and written into the destination storage space.

11. The data processing method according to claim 2, wherein: The current thread is included in the warp, and after determining the index value offset corresponding to the current thread, the method further includes: In response to the absence of an index tensor in a continuous dimension of the output tensor data arrangement storage, it is determined to broadcast the index value corresponding to the index value offset to multiple threads in the warp.

12. The data processing method according to claim 11, characterized in that: In response to the absence of an index tensor in a continuous dimension of the output tensor data arrangement storage, determining to broadcast the index value corresponding to the index value offset to multiple threads in the warp includes: In response to the dimension of the output tensor obtained by indexing not covering a continuous dimension of the output tensor data arrangement storage, it is determined to broadcast the index value corresponding to the index value offset to multiple threads in the thread warp.

13. The data processing method according to claim 1, wherein: The general tensor operation interface includes at least one of a general tensor read and write interface or a general coordinate processing interface. The general tensor read and write interface includes a thread-granularity read and write interface and a warp-granularity read and write interface, wherein the thread-granularity read and write interface is configured to support broadcast operations, and the warp-granularity read and write interface is configured to support merged read and write operations. The universal coordinate processing interface includes at least one of a coordinate conversion interface or a coordinate calculation interface. The coordinate conversion interface is configured to implement coordinate conversion under different data arrangements, and the coordinate calculation interface is configured to calculate an address offset corresponding to a coordinate.

14. A data processing device, characterized in that: The data processing device includes: a receiving module configured to receive signals regarding an input tensor and an index tensor; An execution module is configured to read target data from a source storage space corresponding to the input tensor based on the index tensor through a general tensor operation interface, and write the target data to a destination storage space corresponding to the output tensor, The general tensor operation interface supports unified tensor operations involving at least two data arrangement modes.

15. An electronic device, characterized in that: The electronic device comprises: at least one processor; at least one memory including one or more computer program modules; The one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the data processing method according to any one of claims 1 to 13.

16. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions, when executed by at least one processor, perform the data processing method according to any one of claims 1 to 13.

Images