Data processing method, apparatus, computer program product, and storage medium

Abstract

A data processing method, apparatus, computer program product, and computer-readable storage medium are provided. The method equips each of multiple thread bundles executing in parallel on a processor with a dedicated tensor input register. This register is divided into multiple independently configurable fields to hold various parameters required for performing tensor operations. When the processor needs to perform a tensor operation on a particular thread bundle, it first pre-configures each field of the tensor input register for that thread bundle according to the operation configuration data related to the tensor operation. Then, while issuing a tensor operation instruction to a tensor core dedicated to tensor computation, the entire contents of the tensor input register for that thread bundle are also sent. Upon receiving the instruction and configuration information, the tensor core can directly set its working state based on this information and immediately execute the corresponding tensor computation. Furthermore, after executing the tensor operation, no operation configuration of any thread bundle is saved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to a data processing method, apparatus, computer program product, and storage medium. Background Technology

[0002] Tensor cores are hardware modules in modern artificial intelligence (AI) processors and general-purpose graphics processing units (GPGPUs) specifically designed for efficient tensor computation. As coprocessors, tensor cores typically do not directly execute instruction sequences; instead, the main processor sends computational tasks to them to initiate tensor computations, such as matrix multiplication and convolution—typical AI and high-performance computing operations. These operations are central to deep learning training and inference, scientific computing, and graphics processing, and the execution efficiency of tensor cores directly impacts the overall system's computational throughput and energy efficiency. When performing computations, tensor cores need to know the memory addresses, data types, and write-back addresses of operands. They also often require configuring several parameters related to computation organization and data layout, including but not limited to the dimensions of the matrix or tensor, memory synchronization channel identifiers, padding values, and data access stride. As tensor computation patterns become increasingly complex, the required configuration information may be further expanded, for example, by adding additional control fields to support more flexible data transformations, masks, or sparse computations.

[0003] However, AI processors and GPGPUs typically use fixed-length instruction encoding formats, with a single instruction width generally being 64, 128, or 256 bits. Within this limited instruction encoding space, it's difficult to carry all the aforementioned configuration information, and as tensor operations become more complex, it may be necessary to expand support for even more information. On the other hand, to improve hardware utilization and system throughput, tensor kernels often need to be shared by multiple warps and support flexible time-sharing multiplexing.

[0004] Therefore, a data processing method is needed that can efficiently and flexibly transmit complete configuration information to the tensor kernel under limited instruction encoding constraints, and support fast and low-overhead configuration switching when multiple threads share the tensor kernel. Summary of the Invention

[0005] Embodiments of this disclosure provide a data processing method, apparatus, computer program product, and computer-readable storage medium.

[0006] Embodiments of this disclosure provide a data processing method, comprising: equipping each of a plurality of thread bundles on a processor with a corresponding tensor input register, the tensor input register including a plurality of fields; in response to performing a first tensor operation on a first thread bundle among the plurality of thread bundles, configuring the plurality of fields of the tensor input register of the first thread bundle based on operation configuration data related to the first tensor operation of the first thread bundle; sending a first tensor operation instruction from the processor to a tensor core, and the contents of the tensor input register of the first thread bundle, wherein the first tensor operation instruction is used to instruct the tensor core to perform the first tensor operation of the first thread bundle according to the contents of the tensor input register of the first thread bundle, and the tensor core does not save the contents of the tensor input register of the first thread bundle after performing the first tensor operation.

[0007] Embodiments of this disclosure provide a data processing apparatus, comprising: a register allocation module configured to equip each of a plurality of thread bundles on a processor with a corresponding tensor input register, the tensor input register including a plurality of fields; a register configuration module configured to configure a plurality of fields of the tensor input register of the first thread bundle based on operation configuration data related to the first tensor operation of the first thread bundle in response to the execution of a first tensor operation of the first thread bundle; and an information sending module configured to send a first tensor operation instruction and the contents of the tensor input register of the first thread bundle to a tensor core, wherein the first tensor operation instruction is used to instruct the tensor core to perform the first tensor operation of the first thread bundle according to the contents of the tensor input register of the first thread bundle; wherein the tensor core does not save the contents of the tensor input register of the first thread bundle after performing the first tensor operation.

[0008] Embodiments of this disclosure provide a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, are used to implement the data processing method described above.

[0009] Embodiments of this disclosure provide a computer program product stored on a computer-readable storage medium and including computer-executable instructions that, when executed by a processor, are used to implement the data processing method described above.

[0010] The method provided in the embodiments of this disclosure reconstructs the storage, management, and transmission mechanism of configuration information in tensor operations. Specifically, for multiple thread bundles executing in parallel on a processor, each thread bundle is equipped with a dedicated tensor input register. This register is divided into multiple independently configurable fields to carry various parameters required for performing tensor operations. When the processor needs to perform tensor operations on a specific thread bundle (e.g., the first thread bundle), it first pre-configures each field of the tensor input register of that thread bundle according to the operation configuration data related to the tensor operation. Subsequently, the processor sends the entire contents of the tensor input register of that thread bundle along with the tensor operation instruction to the coprocessor (i.e., the tensor core) dedicated to tensor computation. After receiving the instruction and the accompanying configuration information, the tensor core can directly set the working state based on this information and immediately execute the corresponding tensor computation.

[0011] The method described in this disclosure decouples the rich configuration information required for tensor operations from length-limited instruction encoding and stores it in a sufficiently large dedicated register. This overcomes the bottleneck that fixed-length instructions cannot carry all parameters and makes it possible to support more complex and scalable operation configurations. Secondly, by maintaining an independent tensor input register for each thread bundle, thread bundle-specific configuration information can be achieved. This allows the same thread bundle to perform multiple tensor operations consecutively, and if only some of the operation parameters change, only the corresponding fields in the tensor input register need to be updated, while the remaining unchanged configuration is preserved and reused, significantly reducing the instruction overhead required for configuration updates. Furthermore, when the processor scheduler switches between different thread bundles, since the configuration state of each thread bundle is already prepared in its private register, there is no need to perform time-consuming operations within the tensor core to save the configuration state of the previous thread bundle or restore or reprogram the configuration state of the current thread bundle. Thread beam switching only requires sending the register contents of the target thread beam along with the instruction, which can greatly reduce the latency of context switching and enable tensor kernels to be efficiently and flexibly time-division multiplexed by multiple thread beams. As a result, the actual time used for computation and the overall throughput can be significantly improved. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some exemplary embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0013] Figure 1 This is a schematic diagram illustrating the implementation of tensor operations based on the configuration of registers within the tensor kernel;

[0014] Figure 2 This is a flowchart illustrating a data processing method according to an embodiment of the present disclosure;

[0015] Figure 3 This is a schematic diagram illustrating a tensor operation implementation according to an embodiment of the present disclosure;

[0016] Figure 4 This is a schematic flowchart illustrating a data processing method according to embodiments of the present disclosure; and

[0017] Figure 5 This is a schematic diagram illustrating a data processing apparatus according to an embodiment of the present disclosure. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0019] In this specification and accompanying drawings, steps and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these steps and elements are omitted. Furthermore, in the description of this disclosure, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0020] In embodiments of this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works in conjunction with other related parts to achieve a predetermined objective, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0022] As a dedicated hardware module in modern AI processors and general-purpose graphics processors that performs core tensor operations (such as matrix multiplication), the computational performance of the tensor core directly determines the throughput and energy efficiency of critical tasks such as deep learning training and inference. Therefore, efficiently driving the tensor core to perform operations and optimizing its task distribution and configuration processes are crucial for unlocking hardware potential and improving overall system performance. Currently, there are two main implementation methods for the main processor to send computational tasks to the tensor core.

[0023] A common approach is for the main processor to send a tensor operation instruction encoded with a tensor descriptor number to the coprocessor. Upon receiving the instruction, the tensor core first parses the descriptor number, then accesses a dedicated tensor descriptor memory based on this number to retrieve all the configuration information required for the operation, including matrix dimensions, data layout, padding values, and other parameters, before initiating the computation. This method decouples complex configuration information from compact instruction encoding, passing parameters through indirect addressing.

[0024] Another direct approach is to design a set of dedicated configuration registers inside the tensor kernel to store the information needed for tensor operations. Figure 1 This is a schematic diagram illustrating the implementation of tensor operations based on the configuration of registers within the tensor kernel.

[0025] like Figure 1 As shown, the main processor can primarily include an instruction cache, multiple thread bundles (represented by thread bundles 1 to N in the diagram; a thread bundle is the basic scheduling unit in parallel architectures such as GPUs, and multiple thread bundles can alternately use the tensor core to execute their respective computational tasks), an arbitration and issue unit, an execution unit, and a general-purpose register set. The tensor core, acting as a coprocessor, integrates a dedicated tensor input register. In this implementation, the instruction cache stores the program instruction sequence, including configuration and execution instructions. The multiple thread bundles represent independent task flows that can be executed in parallel. The arbitration and issue unit selects instructions from the ready thread bundles and dispatches them to the corresponding execution units. The execution units handle regular arithmetic and logical operations, while the general-purpose register set provides operand storage space for the execution units. Specifically, when a thread bundle needs to perform tensor operations, the main processor sends a series of configuration instructions to the coprocessor (i.e., the tensor core) through the arbitration and issue unit. These instructions write the operand addresses, matrix dimensions, padding values, and other parameters required for the operation into the dedicated configuration register (tensor input register) inside the tensor core, thus setting the working state of the tensor core. Once all configuration information is ready, the main processor issues an execution instruction, and the tensor core then begins the actual tensor computation based on the complete configuration state loaded in its internal registers. This approach provides the software with a direct and flexible configuration interface.

[0026] However, both of the aforementioned existing technical solutions have significant limitations. For the first descriptor-based approach, the space of the tensor descriptor memory is limited, typically only able to store a finite number of tensor descriptors. When the number of different tensor operation configurations used in the program exceeds this limit, the system must load new descriptors from external memory into the tensor descriptor memory before executing subsequent tensor operations. This process not only requires additional data transfer and management instructions, increasing the complexity of software programming, but also causes the tensor core to remain idle while waiting for configuration to be ready, thus reducing its actual utilization. For the second approach based on core register configuration, the problem is particularly prominent in multi-threaded parallel environments. In general-purpose graphics processors or artificial intelligence processors, multiple thread bundles frequently share the tensor core alternately. Whenever the execution context switches from one thread bundle to another, the configuration information set in the tensor core's internal registers by the previous thread bundle usually cannot be directly inherited or reused by the new thread bundle; the main processor must resend the entire set of configuration instructions to overwrite it. Since thread beam switching is a highly frequent event, such repeated and complete register reconfiguration operations introduce significant overhead, causing the valuable computation cycles of the tensor kernel to be occupied by configuration operations, which also severely restricts its effective utilization. Therefore, finding a mechanism that can support fast switching of configuration information between multiple thread beams with low overhead has become a key challenge in improving the parallel efficiency of tensor kernels.

[0027] Based on this, this disclosure proposes a data processing method that introduces a thread-specific, flexibly configurable and reusable tensor input register, which is sent in conjunction with the operation instructions, to construct an efficient and low-overhead tensor core configuration and switching mechanism, thereby effectively improving the utilization and execution efficiency of tensor computation units in parallel processing systems.

[0028] The aforementioned data processing method reconstructs the storage, management, and transmission mechanism of configuration information in tensor operations. Specifically, for multiple thread bundles executing in parallel on the processor, each thread bundle is equipped with a dedicated tensor input register. This register is divided into multiple independently configurable fields to hold various parameters required for performing tensor operations. When the processor needs to perform tensor operations on a specific thread bundle (e.g., the first thread bundle), it first pre-configures each field of the tensor input register of that thread bundle according to the operation configuration data related to the tensor operation. Subsequently, while issuing tensor operation instructions to the coprocessor dedicated to tensor computation (i.e., the tensor core), the processor also sends the entire contents of the tensor input register of that thread bundle. Upon receiving the instruction and the accompanying configuration information, the tensor core can directly set its working state based on this information and immediately execute the corresponding tensor computation.

[0029] The data processing method disclosed herein decouples the rich configuration information required for tensor operations from the length-limited instruction code and stores it in a sufficiently large dedicated register. This overcomes the bottleneck that fixed-length instructions cannot carry all parameters and makes it possible to support more complex and scalable operation configurations. Secondly, by maintaining an independent tensor input register for each thread bundle, thread bundle-specific configuration information can be achieved. This allows the same thread bundle to perform multiple tensor operations consecutively. If only some of the operation parameters change, only the corresponding field in the tensor input register needs to be updated, while the remaining unchanged configuration is preserved and reused, significantly reducing the instruction overhead required for configuration updates. Furthermore, when the processor scheduler switches between different thread bundles, since the configuration state of each thread bundle is already prepared in its private register, there is no need to perform time-consuming operations within the tensor core to save the configuration state of the previous thread bundle or restore or reprogram the configuration state of the current thread bundle. Thread beam switching only requires sending the register contents of the target thread beam along with the instruction, which can greatly reduce the latency of context switching and enable tensor kernels to be efficiently and flexibly time-division multiplexed by multiple thread beams. As a result, the actual time used for computation and the overall throughput can be significantly improved.

[0030] The following will refer to Figures 2 to 4 The data processing methods disclosed herein are described in detail.

[0031] Figure 2 This is a flowchart illustrating a data processing method 200 according to an embodiment of the present disclosure. Figure 3 This is a schematic diagram illustrating a tensor operation implementation according to an embodiment of the present disclosure. Figure 4 This is a schematic flowchart illustrating a data processing method according to an embodiment of the present disclosure.

[0032] like Figure 3 As shown, the data processing method of this disclosure can be implemented on a typical hardware architecture, which may include a processor and a coprocessor (i.e., a tensor core) working in conjunction with it. The processor may contain an instruction cache, multiple thread bundles (thread bundles 1 to N shown in the diagram), arbitration and issue units, an execution unit, and a general-purpose register set. The tensor core, as a coprocessor, is used to perform tensor operations on each thread bundle. Up to this point, it is consistent with the above references. Figure 1The described tensor operation implementations based on tensor core register configurations are structurally consistent. However, after a deep reflection on the inherent contradictions in the tensor core configuration mechanisms of the aforementioned tensor operation implementations and a re-examination of the architecture, the inventors of this disclosure found that in these designs, whether using shared descriptor memory or tensor core internal registers, a common design premise is implicitly present: the operational state of the execution unit (coprocessor) should be maintained and managed by itself. Under this premise, configuration information is regarded as the private state of the tensor core, causing multiple threads to frequently overwrite and rewrite this state when sharing this computing resource in a time-sharing manner, resulting in unavoidable and significant switching overhead. This design constraint has long limited the utilization efficiency of the tensor core in highly parallel environments.

[0033] Based on this, this disclosure breaks through the aforementioned traditional mindset and proposes a fundamental architectural shift: repositioning the configuration information required for tensor operations as a component of the thread-bound execution context. This is similar to a program counter (PC) or general-purpose register set, whose lifecycle is bound to the thread-bound execution context, rather than to a fixed hardware execution unit. Based on this new concept, the tensor kernel can be designed as a stateless or transiently stateful, highly efficient execution engine. All configurations required for each operation are delivered as part of the input data packet along with the instruction, and do not need to be persistently stored within the kernel after the operation is completed.

[0034] Therefore, as Figure 4 As shown, the data processing method of this disclosure can begin with the pre-configuration of the processor's hardware resources, i.e., equipping each thread bundle with a tensor input register. Specifically, in step S201, for multiple thread bundles on the processor, a corresponding tensor input register can be equipped for each thread bundle, and the tensor input register may include multiple fields.

[0035] In other words, in the embodiments of this disclosure, each thread bundle on the processor can be equipped with a corresponding tensor input register to serve as a program state exclusive to each thread bundle. For example... Figure 3As shown, tensor input registers can be integrated within the processor as part of the thread bundle's private state (associated with the contexts shown in "Thread Bundle 1" to "Thread Bundle N" in the diagram). Their capacity is not limited by instruction word length and can accommodate a complete and scalable set of parameters, including operand base addresses, matrix dimensions, data strides, boundary padding values, and synchronization addresses. For example, with one corresponding tensor input register for each thread bundle on the processor, N such tensor input registers (shown as tensor input registers 1 to N in the diagram) can be integrated in a streaming multiprocessor (SM) supporting thread bundles 1 to N. Each register is strictly bound to a thread bundle ID. These registers can be physically integrated into the processor's register file or used as dedicated storage units within the thread bundle context storage area.

[0036] Optionally, each tensor input register can be physically implemented as a set of memory units within the processor chip, and its logical structure can be divided into multiple independently addressable and programmable domains. These domains can collectively form a structured parameter storage container to hold all the configuration information required for any tensor operations initiated by this thread bundle in the future.

[0037] As mentioned above, these tensor input registers, similar to the program counter, status register, and general-purpose register set inherent to each thread bundle, are integrated and arranged in the processor's thread bundle context storage area, becoming part of the thread bundle execution state. That is, from the moment a thread bundle is created or activated, its dedicated tensor input registers are ready, and many of its fields may be initialized to default values, awaiting to be filled by subsequent configuration instructions.

[0038] In step S202, in response to the execution of a first tensor operation of a first thread bundle among the plurality of thread bundles, multiple fields of the tensor input register of the first thread bundle may be configured based on operation configuration data associated with the first tensor operation of the first thread bundle.

[0039] Optionally, based on the hardware foundation established in step S201, when the processor's scheduling logic determines that a tensor operation (referred to as the first tensor operation) of a specific thread bundle (for convenience) among the multiple thread bundles is about to be executed, a configuration process for the private tensor input register of the first thread bundle can be triggered. Specifically, this configuration process can systematically write all the parameters and control information necessary for executing the first tensor operation, i.e., the operation configuration data related to the first tensor operation of the first thread bundle, into the specified fields of the tensor input register corresponding to the first thread bundle.

[0040] According to embodiments of this disclosure, configuring multiple domains of the tensor input register of the first thread bundle based on computation configuration data related to the first tensor operation of the first thread bundle may include: executing a domain configuration instruction through the processor to write the computation configuration data related to the first tensor operation of the first thread bundle into multiple domains of the tensor input register of the first thread bundle according to the domain configuration instruction.

[0041] Optionally, the processor can flexibly configure these tensor input registers at any point in time using independent domain segment configuration instructions. For example, such as Figure 4 As shown, when the target thread bundle is determined to be the first thread bundle, meaning the first thread bundle needs to perform tensor operations, the processor can generate only one tensor operation instruction specifying the operation type to configure the tensor input register field of the target thread bundle as needed. This field configuration instruction can be a dedicated command introduced in the processor's instruction set architecture for managing tensor input registers.

[0042] Optionally, when it is necessary to prepare parameters for the first tensor operation of the first thread bundle, the processor can generate and issue such an instruction. The operands of this instruction can specify several key elements, such as, but not limited to: the identifier of the target thread bundle (used to address the private register of the first thread bundle), the identifier of the target domain (used to select a specific storage location within the register), and the data value to be written or the address of the source register storing the data value. When the execution unit in the processor decodes and executes this instruction, it can obtain the operation configuration data related to the first tensor operation of the first thread bundle according to the guidance of the instruction, and write it into the tensor input register determined by the identifier of the first thread bundle, and store it precisely in the domain specified by the domain identifier.

[0043] For example, as a non-limiting embodiment of a domain configuration instruction, it could be SET_TIR FIELD_B_COLS, 1024, which instructs the processor to write the value 1024 (e.g., possibly representing the number of columns in matrix B) into the "Number of Columns in Matrix B" domain of the tensor input register of thread bundle A. By executing such a domain configuration instruction, all the scattered parameters related to this complex tensor operation can be systematically and flexibly loaded into the configuration memory space private to the first thread bundle.

[0044] The configuration method based on dedicated instructions described above not only provides a clear and intuitive control interface for programming, but also completely decouples the transmission of configuration data from the encoding limitations of tensor operation instructions. This allows the system to adaptably handle any parameters, regardless of their complexity or future expansion, through this general mechanism that is separate from the operation instructions.

[0045] Optionally, this configuration step can be performed at any time before sending tensor operation instructions to prepare for the upcoming computation task. This also allows the tensor operation instructions themselves to remain concise, carrying only core information such as opcodes, while the complex parameter passing responsibility is transferred to the thread-bound private, ample register storage structure.

[0046] According to embodiments of this disclosure, the operation configuration data related to the first tensor operation of the first thread bundle is the data required to perform the first tensor operation of the first thread bundle, including operand addressing information and operation parameter information; wherein, the operand addressing information may include one or more of the following: the base address of the input data in shared memory, the base address of the output data in shared memory, or the address of the memory synchronization channel; wherein, the operation parameter information may include one or more of the following: the dimension of the matrix involved in the operation, the row or column stride of the data in memory, or the boundary padding value used when padding the input data of the first tensor operation.

[0047] Optionally, the operation configuration data related to the first tensor operation of the first thread bundle is essentially a complete set of parameters indispensable for driving the tensor kernel to complete this specific tensor calculation. Logically, this data can be categorized into two main types: one is operand addressing information, which specifies the location of the data involved in the operation in memory, typically including the base address of the input data in shared memory, the base address of the output data in shared memory, and the address of the memory synchronization channel used to coordinate computation and data movement; the other is operation parameter information, which defines the organization and computation rules of the data itself, specifically covering the various dimensions (sizes) of the matrix or tensor involved in the operation, the data type involved in the operation, the synchronization channel number, the accumulation mode, the data multicast mode, the row or column step when traversing data in memory, and the specific boundary padding values ​​used when padding the input data. This data together constitutes a precise description of a tensor operation.

[0048] According to embodiments of this disclosure, the plurality of domain segments may be extensible; wherein, the plurality of domain segments may include a domain segment for storing operand addressing information and a domain segment for storing operation parameter information; wherein, the domain segment for storing operand addressing information may include one or more of the following: a domain segment for storing the base address of input data in shared memory, a domain segment for storing the base address of output data in shared memory, or a domain segment for storing the address of a memory synchronization channel for coordinating computation and data transfer; wherein, the domain segment for storing operation parameter information may include one or more of the following: a domain segment for storing the dimension of the matrix involved in the operation, a domain segment for storing the row or column stride of data in memory, or a domain segment for storing boundary padding values.

[0049] Optionally, in order to efficiently handle the aforementioned diverse data in hardware, the tensor input register in the embodiments of this disclosure can divide its storage space into multiple expandable domain segments. That is, the number and function of the domain segments in the tensor input register are not fixed, but can be increased, decreased, or redefined according to changes in future computational needs.

[0050] Optionally, these fields can be specifically designed to correspond to the aforementioned data types. Specifically, the tensor input register can include fields for storing operand addressing information, such as fields specifically for storing the base address of input or output data in shared memory, and fields for storing the address of the memory synchronization channel used to coordinate computation and data transfer. Simultaneously, the register can also include fields for storing operation parameter information, such as fields for storing the dimensions (sizes) of the matrices involved in the operation, fields for storing the data types involved in the operation, fields for storing the synchronization channel number, fields for storing the accumulation mode, fields for storing the data multicast mode, fields for storing the row or column steps of data in memory, and fields for storing boundary padding values. Through this structured field design, each piece of relevant data required for the first tensor operation can be accurately stored in the corresponding dedicated container (field) within the register. When subsequent computation instructions need to be sent to the tensor kernel, the data pre-stored in each domain segment can be sent out as a complete and ordered configuration package, thereby efficiently completing the instantaneous configuration of the tensor kernel's working state.

[0051] The scalability design of the aforementioned domain provides flexible space to accommodate diverse operational parameters, ranging from basic matrix dimensions and memory addresses to complex boundary padding values ​​and synchronization control, thus overcoming the encoding limitations of fixed-length instructions.

[0052] like Figure 4 As shown, after completing the on-demand configuration of the tensor input register field of the target thread bundle, the instruction sending and execution phase can begin.

[0053] In step S203, a first tensor operation instruction and the contents of the tensor input register of the first thread bundle can be sent from the processor to the tensor core. The first tensor operation instruction is used to instruct the tensor core to perform a first tensor operation of the first thread bundle according to the contents of the tensor input register of the first thread bundle. After performing the first tensor operation, the tensor core does not save the contents of the tensor input register of the first thread bundle.

[0054] Optionally, the processor can delegate its internally prepared computational tasks to the coprocessor (Tensor Core) for execution. This includes sending two core pieces of information from the processor to the Tensor Core: the first tensor operation instruction and the complete contents of the tensor input register of the first thread bundle. This process can be coordinated by the arbitration and issue unit within the processor, ensuring precise binding and efficient transmission of instructions to the corresponding thread bundle's private configuration. This enables rapid loading of the thread bundle's dedicated configuration and avoids the overhead of the Tensor Core's internal state management.

[0055] Optionally, the first tensor operation instruction can be a command that defines a basic operation type, such as matrix multiplication or convolution, which instructs the tensor kernel to initiate a computation of the corresponding type. However, the operation type alone is insufficient for the tensor kernel to function; it also needs to know the specific object and rules of the operation. This information is contained in the contents of the tensor input register of the first thread bundle, which is sent simultaneously. The contents of this register, namely all parameters (such as base address, dimension, stride, etc.) configured in previous steps and stored in multiple domains, serve as a complete configuration package for this operation and arrive at the tensor kernel synchronously with the operation instruction. Therefore, upon receiving this composite information package, the tensor kernel's internal logic can immediately utilize the register contents sent with the instruction to configure its tensor operation unit, including the data path, address generator, and control state of the operation unit, thereby instantly establishing all the working conditions required for the first thread bundle to execute the first tensor operation, and then immediately executing the corresponding tensor operation.

[0056] In embodiments of this disclosure, such as Figure 4 As shown, after the tensor kernel completes the current tensor operation, the program can statically determine whether there are any subsequent tensor operations that need to be performed. If not, the entire processing flow ends naturally; if so, further judgment can be performed to determine whether to switch thread bundles.

[0057] As a non-limiting embodiment, if it is determined that switching thread bundles is not necessary, i.e., the next operation still belongs to the same thread bundle, then the process can return to the step of "configuring the tensor input register fields of the target thread bundle as needed." In this case, since consecutive operations within the same thread bundle often have high parameter similarity, it is not necessary to reconfigure the entire contents of its private tensor input register. Only the relevant fields in the register need to be modified and updated according to the changes in the operation parameters. For example, it may only be necessary to update the memory address fields of the operands, while keeping fields such as matrix dimension and stride unchanged. This achieves a high degree of reuse of configuration information and significantly reduces instruction overhead.

[0058] Conversely, if it is determined that a thread bundle switch is needed (e.g., the arbitration and emission units have switched thread bundles), meaning the next tensor operation belongs to a different thread bundle, the target thread bundle can be redefined. In other words, the execution context will switch from the current first thread bundle to the new thread bundle corresponding to the next tensor operation (e.g., the second thread bundle). The core of this switch is that the new thread bundle will use its own independent, pre-configured tensor input register. Since the register contents of each thread bundle are isolated and do not interfere with each other, there is no need to save the register contents of the first thread bundle being used, nor is there a need for complex restoration or initialization of the registers of the second thread bundle to be used. It can simply be used as the new configuration source. Based on this, the switching between thread bundles can be extremely fast, introducing almost no additional latency, thus ensuring that the tensor kernel can be efficiently and flexibly time-division multiplexed by multiple thread bundles, greatly improving the overall computational throughput.

[0059] The following describes the implementation examples for the different situations corresponding to the above different judgment results.

[0060] For cases where the next operation still belongs to the same thread bundle, according to embodiments of this disclosure, the data processing method of this disclosure may further include: in response to executing a second tensor operation of the first thread bundle, updating at least a portion of a plurality of domain segments of the tensor input register of the first thread bundle based on operation configuration data related to the second tensor operation of the first thread bundle; sending a second tensor operation instruction from the processor to the tensor core, and the updated content of the tensor input register of the first thread bundle, wherein the second tensor operation instruction is used to instruct the tensor core to execute the second tensor operation of the first thread bundle according to the updated content of the tensor input register of the first thread bundle.

[0061] After completing the first tensor operation, if the same thread bundle (i.e., the first thread bundle) needs to immediately perform another computation (referred to as the second tensor operation), the data processing method of this disclosure can further demonstrate its advantage of efficient configuration reuse. Optionally, in response to the upcoming execution of the second tensor operation, the processor can first update the contents of the tensor input register of the first thread bundle based on the operation configuration data related to this new operation. It should be noted that this update does not always rewrite all fields of the tensor input register, but usually updates a portion of those fields.

[0062] According to embodiments of this disclosure, updating at least some of the multiple domain segments of the tensor input register of the first thread bundle based on the operation configuration data related to the second tensor operation of the first thread bundle may include: modifying only the content of the domain segment related to the difference in the multiple domain segments of the tensor input register of the first thread bundle based on the difference between the operation configuration data related to the second tensor operation and the operation configuration data related to the first tensor operation, while keeping the content of the other domain segments unchanged.

[0063] Specifically, the update process described above can be executed based on the parameter changes of the second tensor operation compared to the first tensor operation. That is, only the contents of specific fields in the register related to the change can be modified, while the contents of other unaffected fields remain unchanged. For example, if the two operations (i.e., the first tensor operation and the second tensor operation) only change the base address of the input data, then only one configuration instruction to update the address field needs to be executed, while other fields such as matrix dimension, stride, and padding can retain their original values. Subsequently, the processor can send the second tensor operation instruction for this operation to the tensor core, simultaneously carrying the complete updated contents of the tensor input register of the first thread bundle.

[0064] Alternatively, the tensor kernel can perform second tensor operations based on this new configuration package, which includes the latest partial parameters and a large number of reused parameters, thereby minimizing the number of instructions and bus transfers required for configuration while ensuring computational correctness.

[0065] According to embodiments of this disclosure, the tensor input register of each of the plurality of thread bundles is arranged in the thread bundle state storage area of ​​the processor, such that for each thread bundle, the contents of the tensor input register of the thread bundle remain unchanged during the lifetime of the thread bundle except for the update, wherein the thread bundle state storage area is a hardware storage area in the processor used to store the execution context information required for each thread bundle.

[0066] Optionally, to implement the above mechanism and ensure the independence and persistence of the configuration states of each thread bundle, in the embodiments of this disclosure, the tensor input registers equipped with each thread bundle can be arranged in the thread bundle state storage area of ​​the processor (i.e., the thread bundle context storage area mentioned above). This storage area can be a hardware area within the processor chip dedicated to storing the execution context information required by each thread bundle (e.g., the execution context information required by the program counter, general-purpose registers, status registers, and tensor input registers). By placing the tensor input registers here, the configuration information of each thread bundle can become part of its complete hardware context. Therefore, unless modified by an explicit update instruction, the contents of the tensor input registers of each thread bundle will remain unchanged throughout its lifetime and independent of the operations of other thread bundles. This provides hardware guarantees for partial updates of the configuration within a thread bundle and zero-overhead state management during cross-thread bundle switching.

[0067] In the case where the next tensor operation belongs to a different thread bundle, according to embodiments of this disclosure, the data processing method of this disclosure may further include: in response to switching from the first thread bundle to a second thread bundle among the plurality of thread bundles to perform tensor operations on the second thread bundle, configuring a plurality of domain segments of the tensor input register of the second thread bundle based on operation configuration data related to the tensor operations of the second thread bundle; sending a third tensor operation instruction from the processor to the tensor core, and the contents of the tensor input register of the second thread bundle, wherein the third tensor operation instruction is used to instruct the tensor core to perform tensor operations on the second thread bundle according to the contents of the tensor input register of the second thread bundle.

[0068] Optionally, such as Figure 4 As shown, when a tensor operation (e.g., the first tensor operation of the first thread bundle) is completed, and the scheduling logic (e.g., the arbitration and launch unit) determines that it is necessary to switch from the current first thread bundle to another thread bundle (e.g., the second thread bundle among the multiple thread bundles) to execute the tensor operation corresponding to the second thread bundle, the data processing method of this disclosure can enable an optimization mechanism for efficient switching between thread bundles.

[0069] Specifically, firstly, in response to the switching decision, the processor can configure multiple domains of the tensor input register of the second thread bundle based on the operation configuration data related to the tensor operations of the second thread bundle. This configuration process is similar to the configuration of the first thread bundle described above; it can be accomplished by executing corresponding domain configuration instructions, and this configuration can be pre-executed at any time before the switching occurs, ensuring that the registers of the second thread bundle are ready when the switching is triggered. Next, the processor's arbitration and issue unit can retrieve the tensor operation instructions (i.e., the third tensor operation instructions) from the instruction stream of the second thread bundle and simultaneously read all the configured contents of the tensor input register of the second thread bundle, sending both together to the tensor core. The third tensor operation instructions can be used to instruct the tensor core to perform specific calculations required by the second thread bundle, while the accompanying register contents provide complete instantaneous configuration parameters for this calculation.

[0070] It is important to note that during the complete switching and transmission process described above, the method disclosed herein does not require storing the contents of the tensor input register of the first thread bundle in the tensor core. This is because the configuration information of each thread bundle is persistently and independently maintained in its own private tensor input register on the processor side. The tensor core is designed as a stateless execution engine, responsible only for receiving and executing the currently delivered instructions and their configuration packets, without maintaining any long-term configuration state associated with any specific thread bundle. Therefore, switching from the first thread bundle to the second thread bundle is simply processing a new, independent computation request for the tensor core, completely eliminating the cumbersome overhead of storing the state of the previous thread bundle in internal registers and restoring or rewriting the state for the next thread bundle. This achieves fast, zero-configuration-residue switching between thread bundles, greatly improving the reuse efficiency of the tensor core in parallel tasks and the overall system throughput.

[0071] As an example, since the configuration of each thread bundle is stored independently in its own tensor input register, when the scheduler decides to start from the first thread bundle (e.g., the corresponding thread bundle), Figure 3When switching from "Thread Bundle 1" to the second thread bundle (e.g., "Thread Bundle 2"), simply stop sending the instruction stream of the former and start sending the instruction stream of the latter. When sending a tensor operation instruction from the latter, the configuration contents in the latter's private register are naturally included. For the tensor kernel, it only changes the source of input data; internally, it does not need to perform any saving operations on the state of the previous thread bundle, nor does it need to perform any state restoration or reprogramming for the current thread bundle. For example, in one embodiment, after thread bundle A completes a matrix multiplication, all its parameters are still stored in its own tensor input register (TIR), while the system switches to thread bundle B to perform convolution because it is waiting for memory data. When the scheduler sends the convolution instruction of thread bundle B, it directly carries the complete set of convolution parameters pre-configured in thread bundle B's own TIR. The configuration of thread bundle A remains unchanged in its private TIR and can be used immediately or after partial updates when it is scheduled again.

[0072] As described above, the data processing method of this disclosure equips each thread bundle with a tensor input register with multiple scalable domain segments. The complete set of complex parameters required for the operation (such as address, dimension, stride, padding value, etc.) can be stored in the register with ample capacity, no longer limited by the number of bits of a single instruction. This allows these parameters to be set flexibly and completely through independent domain segment configuration instructions, while the tensor operation instructions themselves remain concise, carrying only opcodes. This achieves decoupling of configuration information and unlimited expansion potential at the hardware instruction set architecture level.

[0073] Building upon this, the data processing method disclosed herein significantly optimizes the execution efficiency of continuous operations within the same thread bundle. Specifically, for subsequent tensor operations within the same thread bundle, it supports incremental updates only to the changed fields in the tensor input register, while the vast majority of unchanged parameters can be automatically inherited and reused. For example, in deep neural networks, when processing different data blocks consecutively, often only the input address changes, while parameters such as convolution kernel weights and strides remain unchanged. The data processing method disclosed herein allows updating only the address field, and the subsequent operation instructions still carry the complete register contents, allowing the tensor kernel to directly utilize most of the reused parameters to perform calculations. This greatly reduces the number of configuration instructions that must be executed to prepare for subsequent operations, lowers instruction sending overhead and bus traffic, enabling the tensor kernel to remain in a computational state more continuously and improving computational throughput per unit time. Furthermore, although the complete contents of the tensor input register are still sent to the tensor kernel, the overhead of updating specific fields of private registers on the processor side is far lower than the overhead of reconfiguring the entire tensor kernel through a series of instructions.

[0074] Furthermore, the data processing method disclosed herein overcomes the problem of high switching overhead when multiple thread bundles share a tensor core. Specifically, since each thread bundle has its own independent and persistent configuration storage (located in the processor's thread bundle state storage area), when the scheduler needs to switch from one thread bundle to another, it does not need to perform any saving operations on the configuration state of the previous thread bundle within the tensor core, nor does it need to perform cumbersome state restoration or reprogramming for the next thread bundle. The switching process is simplified to: stopping the transmission of the instruction stream of the previous thread bundle, starting the transmission of the instruction stream of the next thread bundle, and automatically attaching the register configuration content that the thread bundle itself is already ready when sending its tensor operation instructions. For the tensor core, it is merely a transition from one stateless execution moment to the next stateless execution moment, and the latency caused by thread bundle switching can be significantly reduced. This allows the tensor core to perform fast and flexible time-sharing multiplexing among a large number of thread bundles with extremely high efficiency, and its actual computational utilization is significantly improved.

[0075] Figure 5 This is a schematic diagram illustrating a data processing apparatus 500 according to an embodiment of the present disclosure.

[0076] According to embodiments of the present disclosure, the data processing apparatus 500 may include a register provisioning module 501, a register configuration module 502, and an information transmission module 503.

[0077] The register configuration module 501 can be configured to equip each of multiple thread bundles on the processor with a corresponding tensor input register, the tensor input register including multiple field segments. Optionally, the register configuration module 501 can perform the operations described above with reference to step S201.

[0078] For example, register provisioning module 501 can provide a corresponding tensor input register for each thread bundle on the processor, serving as a program state exclusive to each thread bundle. The tensor input register can be integrated within the processor as part of the thread bundle's private state, and its capacity is not limited by instruction word length, capable of accommodating a complete and scalable set of parameters, including operand base addresses, matrix dimensions, data strides, boundary padding values, and synchronization addresses.

[0079] For example, each tensor input register can be physically implemented as a set of memory units within a processor chip, and its logical structure can be divided into multiple independently addressable and programmable domains. These domains can collectively form a structured parameter storage container, used to hold all the configuration information required for any tensor operations initiated by this thread bundle in the future.

[0080] These tensor input registers, similar to the program counter, status register, and general-purpose register set inherent to each thread bundle, are integrated and arranged in the processor's thread bundle context memory, becoming part of the thread bundle's execution state. In other words, from the moment a thread bundle is created or activated, its dedicated tensor input registers are ready, with multiple fields potentially initialized to default values ​​and awaiting filling by subsequent configuration instructions.

[0081] The register configuration module 502 can be configured to configure multiple fields of the tensor input register of the first thread bundle in response to the execution of a first tensor operation of the first thread bundle among the plurality of thread bundles, based on operation configuration data associated with the first tensor operation of the first thread bundle. Optionally, the register configuration module 502 can perform the operations described above with reference to step S202.

[0082] Based on the hardware foundation established by the register configuration module 501, when the processor's scheduling logic determines that a tensor operation (referred to as the first tensor operation) of a specific thread bundle (for convenience) among the multiple thread bundles is about to be executed, the register configuration module 502 can trigger the configuration process of the private tensor input register for the first thread bundle. Specifically, this configuration process can systematically write all the parameters and control information necessary for executing the first tensor operation, i.e., the operation configuration data related to the first tensor operation of the first thread bundle, into the designated fields of the tensor input register corresponding to the first thread bundle.

[0083] For example, register configuration module 502 can flexibly configure these tensor input registers at any point in time using independent domain segment configuration instructions. For instance, when the target thread bundle is determined to be the first thread bundle, meaning the first thread bundle needs to perform tensor operations, the processor can generate only one tensor operation instruction specifying the operation type to configure the tensor input register domain segment of the target thread bundle as needed. This domain segment configuration instruction can be a dedicated command introduced in the processor instruction set architecture for managing tensor input registers.

[0084] For example, when it is necessary to prepare parameters for the first tensor operation of the first thread bundle, the processor can generate and issue such an instruction. The operands of this instruction can specify several key elements, such as, but not limited to: the identifier of the target thread bundle (used to address the private register of the first thread bundle), the identifier of the target domain (used to select a specific storage location within the register), and the data value to be written or the address of the source register storing the data value. When the execution unit in the processor decodes and executes this instruction, it can obtain the operation configuration data related to the first tensor operation of the first thread bundle according to the instruction's guidance, and write it into the tensor input register determined by the first thread bundle identifier, and store it precisely in the domain specified by the domain identifier.

[0085] For example, this configuration step can be performed at any time before sending tensor operation instructions to prepare for the upcoming computation task. It also allows the tensor operation instructions themselves to remain concise, carrying only core information such as opcodes, while the complex parameter passing responsibility is transferred to the thread-bound private, ample register storage structure.

[0086] For example, the operation configuration data related to the first tensor operation of the first thread bundle is essentially a complete set of parameters indispensable for driving the tensor kernel to complete this specific tensor computation. Logically, this data can be categorized into two main types: one is operand addressing information, which specifies the location of the data involved in the operation in memory, typically including the base address of the input data in shared memory, the base address of the output data in shared memory, and the address of the memory synchronization channel used to coordinate the execution order between multiple parallel thread blocks; the other is operation parameter information, which defines the organization and computation rules of the data itself, specifically covering the various dimensions of the matrix or tensor involved in the operation, the row or column stride when traversing the data in memory, and the specific boundary padding values ​​used when padding the input data. These data together constitute a precise description of a tensor operation.

[0087] For example, to efficiently handle the diverse data mentioned above in hardware, the tensor input register could divide its storage space into multiple expandable domains. In other words, the number and function of the domains in the tensor input register are not fixed, but can be increased, decreased, or redefined according to changes in future computational needs.

[0088] For example, these domains can be specifically designed to correspond to the aforementioned data types. Specifically, the tensor input register can contain domains for storing operand addressing information, such as a domain specifically for storing the base address of input or output data in shared memory, and a domain for storing the address of the memory synchronization channel used to coordinate computation and data transfer. Simultaneously, the register can also contain domains for storing computational parameter information, such as a domain for storing the dimensions of the matrices involved in the computation, a domain for storing the row or column steps of data in memory, and a domain for storing boundary padding values. Through this structured domain design, each piece of relevant data required for the first tensor operation can be precisely stored in the corresponding dedicated container (domain) in the register. When subsequent computation instructions need to be sent to the tensor kernel, this data pre-stored in each domain can be sent out as a complete and ordered configuration package, thereby efficiently completing the instantaneous configuration of the tensor kernel's working state.

[0089] The information sending module 503 can be configured to send a first tensor operation instruction and the contents of the tensor input register of the first thread bundle to the tensor core. The first tensor operation instruction instructs the tensor core to perform a first tensor operation on the first thread bundle based on the contents of the tensor input register of the first thread bundle. After performing the first tensor operation, the tensor core does not save the contents of the tensor input register of the first thread bundle. Optionally, the information sending module 503 can perform the operations described above with reference to step S203.

[0090] For example, in the information sending module 503, two core pieces of information can be sent from the processor to the tensor core: the first tensor operation instruction and the complete contents of the tensor input register of the first thread bundle. For instance, the first tensor operation instruction could be a command defining a basic operation type, such as matrix multiplication or convolution, which instructs the tensor core to initiate a computation of the corresponding type. However, the operation type alone is insufficient for the tensor core to function; it also needs to know the specific object and rules of the operation. This information is contained within the contents of the tensor input register of the first thread bundle, which is sent simultaneously. The contents of this register—all parameters configured in previous steps and stored in multiple domains (such as base address, dimension, stride, etc.)—serve as a complete configuration package for this operation and arrive at the tensor core synchronously with the operation instruction. Therefore, upon receiving this composite information package, the tensor core's internal logic can immediately utilize the register contents sent with the instruction to configure its tensor operation unit, including the data path, address generator, and control state of the operation unit, thereby instantly establishing all the working conditions required for the first thread bundle to execute the first tensor operation, and then immediately executing the corresponding tensor operation.

[0091] According to embodiments of this disclosure, the data processing apparatus of this disclosure may further include a module for performing the following operations: in response to performing a second tensor operation on the first thread bundle, updating at least a portion of a plurality of domain segments of a tensor input register of the first thread bundle based on operation configuration data associated with the second tensor operation on the first thread bundle; sending a second tensor operation instruction from the processor to the tensor core, and the updated content of the tensor input register of the first thread bundle, wherein the second tensor operation instruction is used to instruct the tensor core to perform the second tensor operation on the first thread bundle according to the updated content of the tensor input register of the first thread bundle.

[0092] For example, after completing the first tensor operation, if the same thread bundle (i.e., the first thread bundle) needs to immediately perform another computation (called the second tensor operation), the module can, in response to the upcoming execution of the second tensor operation, first update the contents of the tensor input register of the first thread bundle based on the operation configuration data associated with this new operation. It is important to note that this update does not always rewrite all fields of the tensor input register, but typically updates only a portion of those fields.

[0093] According to embodiments of this disclosure, the data processing apparatus of this disclosure may further include a module for performing the following operations: in response to switching from a first thread bundle to a second thread bundle among the plurality of thread bundles to perform tensor operations on the second thread bundle, configuring a plurality of domain segments of a tensor input register of the second thread bundle based on operation configuration data related to the tensor operations on the second thread bundle; sending a third tensor operation instruction from the processor to the tensor core, and the contents of the tensor input register of the second thread bundle, wherein the third tensor operation instruction is used to instruct the tensor core to perform tensor operations on the second thread bundle according to the contents of the tensor input register of the second thread bundle.

[0094] For example, when a tensor operation (e.g., the first tensor operation of the first thread bundle) is completed, and the scheduling logic (e.g., arbitration and launch unit) determines that it is necessary to switch from the current first thread bundle to another thread bundle (e.g., the second thread bundle among the multiple thread bundles) to execute the tensor operation corresponding to the second thread bundle, this module can execute an optimization mechanism for efficient switching between thread bundles. For example, in response to this switching decision, the module can configure multiple fields of the tensor input register of the second thread bundle based on the operation configuration data related to the tensor operation of the second thread bundle. This configuration process is similar to the configuration of the first thread bundle described above, and can be accomplished by executing the corresponding field configuration instructions. This configuration can be pre-executed at any time before the switching occurs, so that the register of the second thread bundle is already in a ready state when the switching is triggered. Next, the module can extract its tensor operation instruction (i.e., the third tensor operation instruction) from the instruction stream of the second thread bundle, and simultaneously read all the configured contents in the tensor input register of the second thread bundle, and send both of them to the tensor core. The third tensor operation instruction can be used to instruct the tensor kernel to perform a specific computation required by the second thread bundle, while the accompanying register contents can provide complete instantaneous configuration parameters for the computation.

[0095] According to embodiments of this disclosure, the data processing apparatus of this disclosure may further include modules for performing other operations of the data processing method described above, which will not be described in detail here.

[0096] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0097] For example, the methods or apparatus according to embodiments of this disclosure can also be implemented using the architecture of a computing device. The computing device may include a bus, one or more CPUs, read-only memory (ROM), random access memory (RAM), a communication port connected to a network, input / output components, a hard disk, etc. Storage devices in the computing device, such as ROM or hard disk, may store various data or files used for processing and / or communication by the data processing methods provided in this disclosure, as well as program instructions executed by the CPU. The computing device may also include a user interface. Of course, the above architecture is merely exemplary, and one or more components of the computing device described above may be omitted as needed when implementing different devices.

[0098] According to another aspect of this disclosure, a computer-readable storage medium is also provided. The computer storage medium stores computer-executable instructions. When the computer-executable instructions are executed by a processor, a data processing method according to embodiments of this disclosure described with reference to the above-drawn figures can be performed. The computer-readable storage medium in the embodiments of this disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0099] Embodiments of this disclosure also provide a computer program product or computer program including computer-executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the computer device to perform a data processing method according to embodiments of this disclosure.

[0100] Embodiments of this disclosure provide a data processing method, apparatus, computer program product, and computer-readable storage medium.

[0101] The method provided in the embodiments of this disclosure reconstructs the storage, management, and transmission mechanism of configuration information in tensor operations. Specifically, for multiple thread bundles executing in parallel on a processor, each thread bundle is equipped with a dedicated tensor input register. This register is divided into multiple independently configurable fields to carry various parameters required for performing tensor operations. When the processor needs to perform tensor operations on a specific thread bundle (e.g., the first thread bundle), it first pre-configures each field of the tensor input register of that thread bundle according to the operation configuration data related to the tensor operation. Subsequently, the processor sends the entire contents of the tensor input register of that thread bundle along with the tensor operation instruction to the coprocessor (i.e., the tensor core) dedicated to tensor computation. After receiving the instruction and the accompanying configuration information, the tensor core can directly set the working state based on this information and immediately execute the corresponding tensor computation.

[0102] The method described in this disclosure decouples the rich configuration information required for tensor operations from length-limited instruction encoding and stores it in a sufficiently large dedicated register. This overcomes the bottleneck that fixed-length instructions cannot carry all parameters and makes it possible to support more complex and scalable operation configurations. Secondly, by maintaining an independent tensor input register for each thread bundle, thread bundle-specific configuration information can be achieved. This allows the same thread bundle to perform multiple tensor operations consecutively, and if only some of the operation parameters change, only the corresponding fields in the tensor input register need to be updated, while the remaining unchanged configuration is preserved and reused, significantly reducing the instruction overhead required for configuration updates. Furthermore, when the processor scheduler switches between different thread bundles, since the configuration state of each thread bundle is already prepared in its private register, there is no need to perform time-consuming operations within the tensor core to save the configuration state of the previous thread bundle or restore or reprogram the configuration state of the current thread bundle. Thread beam switching only requires sending the register contents of the target thread beam along with the instruction, which can greatly reduce the latency of context switching and enable tensor kernels to be efficiently and flexibly time-division multiplexed by multiple thread beams. As a result, the actual time used for computation and the overall throughput can be significantly improved.

[0103] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing at least one executable instruction for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer-executable instructions.

[0104] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0105] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure.

Claims

1. A data processing method, comprising: For multiple thread bundles on the processor, each thread bundle is equipped with a corresponding tensor input register, which includes multiple field segments; In response to the execution of a first tensor operation of a first thread bundle among the plurality of thread bundles, multiple fields of the tensor input register of the first thread bundle are configured based on operation configuration data associated with the first tensor operation of the first thread bundle. The processor sends a first tensor operation instruction to the tensor core, along with the contents of the tensor input register of the first thread bundle. The first tensor operation instruction instructs the tensor core to perform a first tensor operation on the first thread bundle based on the contents of the tensor input register of the first thread bundle. After performing the first tensor operation, the tensor core does not save the contents of the tensor input register of the first thread bundle.

2. The method of claim 1, further comprising: In response to the execution of a second tensor operation of the first thread bundle, at least a portion of the multiple segments of the tensor input register of the first thread bundle are updated based on the operation configuration data associated with the second tensor operation of the first thread bundle. The processor sends a second tensor operation instruction to the tensor core, along with the updated contents of the tensor input register of the first thread bundle, wherein the second tensor operation instruction is used to instruct the tensor core to perform a second tensor operation on the first thread bundle based on the updated contents of the tensor input register of the first thread bundle.

3. The method as described in claim 2, wherein, Based on the operation configuration data related to the second tensor operation of the first thread bundle, updating at least a portion of multiple segments of the tensor input register of the first thread bundle includes: Based on the difference between the operation configuration data related to the second tensor operation and the operation configuration data related to the first tensor operation, only the content of the field segment related to the difference in the multiple field segments of the tensor input register of the first thread bundle is modified, while the content of the other field segments in the multiple field segments remains unchanged.

4. The method of claim 2, wherein, The tensor input register of each of the plurality of thread bundles is arranged in the thread bundle state storage area of ​​the processor, such that for each thread bundle, the contents of the tensor input register of the thread bundle remain unchanged during the lifetime of the thread bundle except for the update, wherein the thread bundle state storage area is a hardware storage area in the processor used to store the execution context information required for each thread bundle.

5. The method of claim 1, wherein, Based on the operation configuration data related to the first tensor operation of the first thread bundle, the configuration of multiple fields of the tensor input register of the first thread bundle includes: The processor executes a domain segment configuration instruction to write the operation configuration data related to the first tensor operation of the first thread bundle into multiple domain segments of the tensor input register of the first thread bundle, according to the domain segment configuration instruction.

6. The method of claim 1, further comprising: In response to switching from the first thread bundle to the second thread bundle among the plurality of thread bundles to perform tensor operations on the second thread bundle, multiple field segments of the tensor input register of the second thread bundle are configured based on operation configuration data related to the tensor operations of the second thread bundle; The processor sends a third tensor operation instruction to the tensor core, along with the contents of the tensor input register of the second thread bundle, wherein the third tensor operation instruction is used to instruct the tensor core to perform tensor operations on the second thread bundle based on the contents of the tensor input register of the second thread bundle.

7. The method of claim 1, wherein, The operation configuration data related to the first tensor operation of the first thread bundle is the data required to execute the first tensor operation of the first thread bundle, including operand addressing information and operation parameter information; The operand addressing information includes one or more of the following: the base address of the input data in the shared memory, the base address of the output data in the shared memory, or the address of the memory synchronization channel; The operation parameter information includes one or more of the following: the dimension of the matrix involved in the operation, the data type involved in the operation, the synchronization channel number, the accumulation mode, the data multicast mode, the row or column step of the data in memory, or the boundary padding value used when padding the input data of the first tensor operation.

8. The method of claim 1, wherein, The multiple domain segments are extensible; The plurality of field segments include a field segment for storing operand addressing information and a field segment for storing operation parameter information; The field segment used to store operand addressing information includes one or more of the following: a field segment used to store the base address of input data in shared memory, a field segment used to store the base address of output data in shared memory, or a field segment used to store the address of a memory synchronization channel used to coordinate computation and data transfer. The domain segment used to store operation parameter information includes one or more of the following: a domain segment used to store the dimension of the matrix involved in the operation, a domain segment used to store the data type involved in the operation, a domain segment used to store the synchronization channel number, a domain segment used to store the accumulation mode, a domain segment used to store the data multicast mode, a domain segment used to store the row or column step of data in memory, or a domain segment used to store boundary padding values.

9. A data processing apparatus, comprising: The register equipping module is configured to equip each of the multiple thread bundles on the processor with a corresponding tensor input register, the tensor input register including multiple field segments; The register configuration module is configured to configure multiple fields of the tensor input register of the first thread bundle in response to the execution of the first tensor operation of the first thread bundle among the plurality of thread bundles, based on operation configuration data related to the first tensor operation of the first thread bundle; as well as The information sending module is configured to send a first tensor operation instruction to the tensor core and the contents of the tensor input register of the first thread bundle, wherein the first tensor operation instruction is used to instruct the tensor core to perform a first tensor operation of the first thread bundle according to the contents of the tensor input register of the first thread bundle; Wherein, after performing the first tensor operation, the tensor kernel does not save the contents of the tensor input register of the first thread bundle.

10. A computer program product stored on a computer-readable storage medium and comprising computer-executable instructions which, when executed by a processor, are used to implement the method of any one of claims 1-8.

11. A computer-readable storage medium having stored thereon computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of claims 1-8.

Images