Processor, tensor processing method, and device

Abstract

The present application relates to the technical field of data processing, and is used for realizing universal and efficient processing of a multi-dimensional tensor. Provided are a processor, a tensor processing method, and a device. A tensor processing unit in the processor comprises: a preprocessing circuit, which is used for adjusting first dimension information of a multi-dimensional tensor, so as to obtain second dimension information, wherein the product of the size of any dimension in the second dimension information and the capacity of a unit access storage space is smaller than or equal to the capacity of a data cache; an address generation circuit, which is used for generating, on the basis of the second dimension information, a plurality of source addresses corresponding to a plurality of pieces of data in the multi-dimensional tensor; an access control circuit, which is used for determining a plurality of pieces of cache mapping information corresponding to the plurality of source addresses, and sending a plurality of access requests on the basis of the plurality of source addresses, and is used for acquiring from a first memory the plurality of pieces of data of the multi-dimensional tensor; and the data cache, which is used for caching the plurality of pieces of data on the basis of the plurality of pieces of cache mapping information.

Description

A processor, tensor processing method and device

[0001] This application claims priority to Chinese Patent Application No. 202410521298.4, filed with the State Intellectual Property Office of China on April 26, 2024, entitled “A processor, tensor processing method and apparatus”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of data processing technology, and in particular to a processor, tensor processing method and device. Background Technology

[0003] A tensor is a mathematical concept that includes scalars, vectors, and linear operators, and can be expressed using a coordinate system. Multidimensional tensor transpose is a common operation in high-performance computing (HPC) and artificial intelligence (AI) applications. The computational methods and hardware architectures used for this operation are key areas for improving computational performance.

[0004] Currently, due to alignment constraints in chip memory and the lack of a universal transpose instruction, it is necessary to convert the data to aligned granularity using a two-dimensional transpose within the vector unit, then perform small-granular rearrangement, and finally convert it back to unaligned granularity using a two-dimensional transpose. However, this algorithm leads to problems such as small bus transport granularity and overly fragmented instructions, severely limiting performance. Summary of the Invention

[0005] This application provides a processor, tensor processing method, and apparatus for achieving general and efficient processing of multidimensional tensors and improving performance.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, a processor is provided, including a tensor processing unit for implementing multidimensional tensor processing. The tensor processing unit includes: a preprocessing circuit, an address generation circuit, an access control circuit, and a data buffer. The preprocessing circuit is used to adjust first-dimensional information of the multidimensional tensor to obtain second-dimensional information, and to determine an input dimension data interval based on the second-dimensional information. The product of the size of any dimension in the second-dimensional information and the capacity of a unit access memory space (e.g., a cache line) is less than or equal to the capacity of the data buffer. The input dimension data interval is the interval between data of multiple dimensions in the multidimensional tensor. The address generation circuit... The circuit is used to generate multiple source addresses corresponding to multiple data in the multidimensional tensor based on the second dimension information and the input dimension data interval. The multiple source addresses are the addresses corresponding to the multiple data in the multidimensional tensor in the first memory. The access control circuit is used to determine multiple cache mapping information corresponding to the multiple source addresses and send multiple access requests based on the multiple source addresses. The multiple access requests are used to retrieve the multiple data from the first memory. The cache mapping information is used to indicate the cache address corresponding to the multiple source addresses in the data cache. The data cache is used to cache the multiple data based on the multiple cache mapping information.

[0008] In the above technical solution, the preprocessing circuit adjusts the first dimension information of the multidimensional tensor to obtain the second dimension information, and determines the input dimension data interval based on the second dimension information. The product between the size of any dimension in the second dimension information and the capacity of the unit access storage space is less than or equal to the capacity of the data cache. The address generation circuit generates multiple source addresses corresponding to multiple data in the multidimensional tensor based on the second dimension information and the input dimension data interval. The access control circuit determines multiple cache mapping information corresponding to the multiple source addresses, and sends multiple access requests to retrieve multiple data from the first memory based on the multiple source addresses. The data cache caches the multiple data based on the multiple cache mapping information, thereby achieving general and efficient processing of multidimensional tensors, improving cache hit rate, reducing capacity misses, and improving performance in a limited data cache.

[0009] In one possible implementation of the first aspect, the preprocessing circuit is further configured to perform at least one of the following processes on the first dimension information of the multidimensional tensor: reordering, splitting axes, borrowing axes, or padding. In the above possible implementations, the preprocessing circuit, through at least one of the following processes: reordering, splitting axes, borrowing axes, or padding, can improve cache hit rate, reduce capacity misses, and improve performance when tensor processing is performed by the tensor processing unit in the processor.

[0010] In one possible implementation of the first aspect, the preprocessing circuit is further configured to: determine the output dimension data interval based on the second dimension information, the output dimension data interval being the interval between data of multiple dimensions in the processed multidimensional tensor; the address generation circuit is further configured to generate multiple destination addresses corresponding to the multiple data based on the second dimension interval and the output dimension data interval, the multiple destination addresses being addresses for writing the multiple data into the second memory; and the data buffer is further configured to write the multiple data into the second memory based on the multiple destination addresses.

[0011] In one possible implementation of the first aspect, the data cache includes multiple cache lines, and the multiple cache mapping information is used to indicate the cache lines corresponding to the multiple source addresses. The tensor processing unit also includes a state memory; the state memory is used to store cache state information, which indicates the state of the multiple cache lines, namely, a hit or a miss; the access control circuit is further used to query the cache state information, and when the state of the cache line corresponding to any of the multiple source addresses is missing, send an access request corresponding to that source address, and update the state of the cache line corresponding to that source address in the cache state information to a hit. In the above possible implementation, by storing and maintaining the cache state information of the multiple cache lines in the data cache through the state memory, and by the access control circuit accessing based on the cache state information stored in the state memory, redundant access can be reduced and performance improved.

[0012] In one possible implementation of the first aspect, the state memory includes multiple storage repositories; these repositories are used to store the states of cache lines corresponding to different source addresses among the multiple source addresses, respectively. In the above possible implementation, by storing the states of cache lines corresponding to different source addresses among the multiple source addresses in the multiple storage repositories, the efficiency of determining the states of the cache lines corresponding to the multiple source addresses is improved.

[0013] In one possible implementation of the first aspect, the plurality of cache mapping information is used to indicate the cache lines corresponding to the plurality of source addresses; the access control circuit is further used to merge the at least two source addresses when the cache lines corresponding to at least two of the plurality of source addresses are the same. In the above possible implementation, merging at least two source addresses corresponding to the same cache line by the access control circuit can reduce the number of access requests and improve access performance.

[0014] In one possible implementation of the first aspect, the tensor processing unit further includes a first-in-first-out (FIFO) queue; the FIFO queue is used to cache the plurality of cache mapping information; the data buffer is further used to cache the plurality of data according to the plurality of cache mapping information output by the FIFO queue when the first memory returns the plurality of data. In the above possible implementation, caching the plurality of cache mapping information using the FIFO queue and caching the plurality of data according to the order of the plurality of cache mapping information by the data buffer simplifies the management of the data buffer, reduces the overhead of data buffer control, and thus improves performance.

[0015] In one possible implementation of the first aspect, the tensor processing unit further includes: the order-preserving circuit, used to preserve the order of the plurality of data returned from the first memory before outputting them to the data buffer. In the above possible implementation, by preserving the order of the plurality of data returned from the first memory through the order-preserving circuit, the correctness of the states of the cache lines corresponding to the plurality of source addresses in the state memory, as well as the correctness of the plurality of data received by the data buffer, can be guaranteed.

[0016] In one possible implementation of the first aspect, the address generation circuit is further configured to generate multiple batches of address information corresponding to multiple batches of data in the multidimensional tensor based on the second dimension information and the input dimension data interval, wherein each batch of address information includes multiple source addresses of a batch of data. The above possible implementations can improve the efficiency of generating multiple batches of address information.

[0017] Secondly, a tensor processing method is provided, applied in a processor. The method includes: adjusting first-dimensional information of a multidimensional tensor to obtain second-dimensional information, and determining an input dimension data interval based on the second-dimensional information, wherein the product of the size of any dimension in the second-dimensional information and the capacity of a unit access memory space is less than or equal to the capacity of the data cache, and the input dimension data interval is the interval between data of multiple dimensions in the multidimensional tensor; generating multiple source addresses corresponding to multiple data in the multidimensional tensor based on the second-dimensional information and the input dimension data interval, wherein the multiple source addresses are addresses corresponding to the multiple data in a first memory; determining multiple cache mapping information corresponding to the multiple source addresses, and sending multiple access requests based on the multiple source addresses, wherein the multiple access requests are used to retrieve the multiple data from the first memory, and the cache mapping information is used to indicate the cache addresses corresponding to the multiple source addresses; and caching the multiple data in a data cache based on the multiple cache mapping information.

[0018] In one possible implementation of the second aspect, adjusting the first dimension information of the multidimensional tensor to obtain the second dimension information includes: performing at least one of the following processing on the first dimension information of the multidimensional tensor: reordering, splitting axes, borrowing axes, or filling.

[0019] In one possible implementation of the second aspect, the method further includes: determining an output dimension data interval based on the second dimension information, the output dimension data interval being the interval between data of multiple dimensions in the processed multidimensional tensor; generating multiple destination addresses corresponding to the multiple data based on the second dimension interval and the output dimension data interval, the multiple destination addresses being addresses for writing the multiple data into a second memory; and writing the multiple data into the second memory based on the multiple destination addresses.

[0020] In one possible implementation of the second aspect, the data cache includes multiple cache lines, the multiple cache mapping information being used to indicate the cache lines corresponding to the multiple source addresses; the method further includes: querying the cache status information, the cache status information being used to indicate the status of the multiple cache lines, the status being either hit or missing; when the status of the cache line corresponding to any of the multiple source addresses is missing, sending an access request corresponding to the source address, and changing the status of the cache line corresponding to the source address in the cache status information to hit.

[0021] In one possible implementation of the second aspect, the method further includes: storing and querying the status of cache lines corresponding to different source addresses among the multiple source addresses through multiple repositories.

[0022] In one possible implementation of the second aspect, the plurality of cache mapping information is used to indicate the cache lines corresponding to the plurality of source addresses, and the method further includes: merging the at least two source addresses when the cache lines corresponding to at least two of the plurality of source addresses are the same.

[0023] In one possible implementation of the second aspect, the plurality of cache mapping information is stored in a first-in-first-out (FIFO) queue, and the plurality of data are cached in a data cache according to the plurality of cache mapping information, including: when the first memory returns the plurality of data, the plurality of data are cached in a data cache according to the plurality of cache mapping information output by the FIFO queue.

[0024] In one possible implementation of the second aspect, the method further includes: maintaining the order of the plurality of data returned from the first memory.

[0025] In one possible implementation of the second aspect, generating multiple source addresses corresponding to multiple data in the multidimensional tensor based on the second dimension information and the input dimension data interval includes: generating multiple batch address information corresponding to multiple batches of data in the multidimensional tensor based on the second dimension information and the input dimension data interval, wherein each batch address information includes multiple source addresses of a batch of data.

[0026] Thirdly, an electronic device is provided, comprising a circuit board and a processor disposed on the circuit board and provided as in the first aspect or any possible implementation thereof.

[0027] Fourthly, a computer-readable storage medium is provided that stores a computer program or instructions that, when executed, implement the methods provided by the second aspect or any possible implementation thereof.

[0028] Fifthly, a computer program product is provided, comprising: a computer program (also referred to as code or instructions) that, when run, causes a computer to perform the methods provided by the second aspect or any possible implementation thereof.

[0029] Understandably, the beneficial effects that can be achieved by the second to fifth aspects mentioned above can be referred to in the context of the first aspect or any possible implementation of the first aspect, and will not be repeated here. Attached Figure Description

[0030] Figure 1 is a schematic diagram of a shaft disassembly provided in an embodiment of this application;

[0031] Figure 2 is a schematic diagram of a decomposition of multidimensional tensors provided in an embodiment of this application;

[0032] Figure 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0033] Figure 4 is a schematic diagram of the structure of a processor provided in an embodiment of this application;

[0034] Figure 5 is a schematic diagram of a data buffer provided in an embodiment of this application;

[0035] Figure 6 is a schematic diagram of a tensor processing unit provided in an embodiment of this application;

[0036] Figure 7 is a schematic diagram of a multidimensional tensor processing flow provided in an embodiment of this application;

[0037] Figure 8 is a schematic diagram of another multidimensional tensor processing flow provided in an embodiment of this application;

[0038] Figure 9 is a schematic diagram of a multidimensional tensor transpose provided in an embodiment of this application;

[0039] Figure 10 is a schematic diagram of another multidimensional tensor transpose provided in an embodiment of this application;

[0040] Figure 11 is a schematic diagram of a two-level control provided in an embodiment of this application. Detailed Implementation

[0041] The technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings. In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, a, b, and c; where a, b, and c can be single or multiple.

[0042] The embodiments of this application use terms such as "first" and "second" to distinguish objects with similar names, functions, or effects. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or order of execution. The term "coupling" is used to indicate an electrical connection, including direct connection via wires or terminals or indirect connection via other devices. Therefore, "coupling" should be considered as a broad type of electronic communication connection.

[0043] In this application, SRC stands for source and DST stands for destination. For example, SRC memory represents source memory, SRC address represents source address, and SRC direction represents the direction from the source or source memory to the tensor processing unit; DST memory represents destination memory, DST address represents destination address, and DST direction represents the direction from the tensor processing unit to the destination or destination memory.

[0044] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0045] Before introducing the embodiments of this application, the relevant technical concepts and background involved in this application will be explained first.

[0046] 1. A tensor is a multidimensional array used to represent the transformation relationship of physical quantities in different coordinate systems. The rank of a tensor represents its dimension. A scalar can be understood as a 0-dimensional tensor, a vector as a 1-dimensional tensor, and a matrix as a 2-dimensional tensor. In an example, when the rank of a tensor is 0, it can be represented as 1000; when the rank is 1, it can be represented as [2, 4, 100]; and when the rank is 2, it can be represented as {[1, 2, 3], [4, 5, 6], [7, 8, 9]}. Tensors can perform various operations, including but not limited to transpose, inverse, partition, and mathematical operations.

[0047] 2. Axis Splitting and Axis Borrowing. An axis can be understood as the dimension of a tensor. Axis splitting involves dividing one dimension into at least two dimensions, while axis borrowing involves merging at least two dimensions into one. Taking a two-dimensional tensor as an example, as shown in Figure 1, (a) in Figure 1 represents the scenario without axis splitting. In this scenario, the traversal order of the elements is to traverse the elements in the first row from left to right, then the elements in the second row from left to right, and so on, until the elements in the last row are traversed from left to right. (b) in Figure 1 represents the scenario with axis splitting. In this scenario, the traversal order of the elements is to traverse all elements to the left of the axis according to the traversal order in Figure 1(a), and then traverse all elements to the right of the axis according to the traversal order in Figure 1(a). Figure 1(c) illustrates a scenario involving splitting two axes, where axes 1 and 2 are represented. In this scenario, the element traversal order is as follows: traversing all elements to the left of axis 1 and above axis 2 according to the traversal order in Figure 1(a); traversing all elements to the right of axis 1 and above axis 2 according to the traversal order in Figure 1(a); traversing all elements to the left of axis 1 and below axis 2 according to the traversal order in Figure 1(a); and traversing all elements to the right of axis 1 and below axis 2 according to the traversal order in Figure 1(a). Splitting axes changes the element traversal order, making it applicable to various algorithms and improving cache hit rates while reducing access latency. Borrowing axes is the opposite of the splitting scenario described above, and will not be elaborated further in this embodiment. Borrowing axes reduces the number of accesses and improves access efficiency.

[0048] Currently, multidimensional tensor transpose is a common operation in high-performance computing (HPC) and artificial intelligence (AI) applications. The computation method and the hardware architecture used for this operation are key areas for improving computational performance.

[0049] Because chip memory has alignment constraints and there is no universal transpose instruction, it is necessary to convert it to aligned granularity through two-dimensional transpose in the vector unit, then perform small-granular rearrangement, and finally convert it back to unaligned granularity through two-dimensional transpose. However, this algorithm leads to problems such as small bus transport granularity and overly fragmented instructions, which severely limits performance. This vector unit can also be called a vector operation unit, which may include memory, such as random access memory (RAM).

[0050] Specifically, the transpose of a multidimensional tensor is achieved through alignment and rearrangement instructions in the vector operation unit. Typically, the multidimensional tensor is decomposed into multiple sets of 5- to 6-dimensional transposes, which are then executed sequentially.

[0051] For example, as shown in Figure 2, in the transformation scenario of converting the multidimensional tensor (2, 50, 9, 23, 40, 5, 8) to (5, 40, 23, 9, 50, 8, 2), the tensor copied into the borrowed axis is represented as (2, 50, 9, 23, 40, 5, 8), the tensor copied out of the borrowed axis is represented as (5, 40, 23, 9, 50, 8, 2), the tensor copied into the de-axis is represented as (2, 50, 9, 23, (7*5+5), 5, 8), and the tensor copied out of the de-axis is represented as (5, 40, 23, 9, (16*3+2), 8, 2). The memory transfer engine in the vector operation unit... The storage transfer engine (MTE) cycles during the second round of data transfer (e.g., MTE2), where the burst length (burstLen) is as large as possible, and the corresponding tensor is represented as (2, (16*3+2), 9, 23, (7*5+5), 5, 8). The storage transfer engine cycles during the third round of data transfer (e.g., MTE3), where the burst length (burstLen) is as large as possible, and the corresponding tensor is represented as (5, (7*5+5), 23, 23, 9, (16*3+2), 8, 2). After the above-described borrowing and splitting of axes, the multidimensional tensor can be decomposed into multiple 5-6 dimensional transposes, specifically represented as: (2, 16, 7, 5, 8) -> (5, 7, 16, 8, 2), (2, 16, 5, 5, 8) -> (5, 5, 16, 8, 2), (2, 2, 7, 5, 8) -> (5, 7, 2, 8, 2), (2, 2, 5, 5, 8) -> (5, 5, 2, 8, 2). The bolded axes represent borrowing and splitting of axes. For example, (40, 5, 8) in (2, 50, 9, 23, 40, 5, 8) represents borrowing an axis, and (7*5+5) in (2, 50, 9, 23, (7*5+5), 5, 8) represents splitting an axis. The aforementioned multiple dimensions (or axes) are arranged in order from the highest axis to the lowest axis. For example, in the multidimensional tensor (2, 50, 9, 23, 40, 5, 8), 2 represents the dimension of the highest axis and 8 represents the dimension of the lowest axis.

[0052] Furthermore, the transpose of 5- to 6-dimensional elements can be achieved through the following three steps (i.e., steps 1 to 3). In the implementation process described below, it may be necessary to iterate through these three steps multiple times to complete a 5- to 6-dimensional transpose.

[0053] Step 1: Move the data from external storage into the RAM of the vector operation unit.

[0054] Step 2: Since non-32-byte (B) aligned transpose execution is not provided, the data is moved to 32-byte alignment within the vector unit using a transpose instruction within a 32-byte aligned block. Then, the data is rearranged at the block level using 32-byte blocks as a unit, and finally, redundant data is removed using a transpose instruction to become unaligned. For example, (a, b, c) -> (c, b, a) (assuming the data type is float 16, and a, b, and c are all non-32-byte aligned numbers) is operated on in the vector unit as follows: (a, b, c) -> (a, b, c, 16) (using the same 32-byte block on the input side, increasing the space by 16 times through the intra-block transpose instruction); (a, b, c, 16) -> (c, b, a, 16) (using inter-block rearrangement instructions); (c, b, a, 16) -> (16, c, b, a) (using the intra-block transpose instruction, only 1 / 16 of the output is valid).

[0055] Step 3: Remove it from the RAM of the vector operation unit.

[0056] In some implementations, a general tensor transpose can be achieved by using continuous outputs. Software optimizations can make input and output memory access continuous; however, this approach is complex and suffers from bottlenecks in address calculation. In other implementations, the overhead of address calculation can be eliminated by adding a direct memory access (DMA) unit. However, this approach has limitations, such as the inability to implement some flexible transpose operations.

[0057] Based on this, embodiments of this application provide a processor with a tensor processing unit to solve the problem of transposing or rearranging multidimensional tensors, reduce programming complexity, improve cache hit rate, and thus significantly improve performance. This processor can be applied to electronic devices, including but not limited to: mobile phones, tablets, laptops, PDAs, mobile internet devices (MIDs), cameras, wearable devices (such as smartwatches, smart bracelets, pedometers, etc.), audio equipment, audio and video players, set-top boxes, game consoles, printers, mice, keyboards, in-vehicle equipment (such as equipment on vehicles like cars, airplanes, ships, trains, and high-speed trains), virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, smart home devices (such as refrigerators, televisions, air conditioners, electricity meters, etc.), intelligent robots, workshop equipment, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or wireless terminals in smart homes, and flying equipment (such as intelligent robots, hot air balloons, drones, airplanes), etc.

[0058] The following example uses a mobile phone as an example to illustrate the structure of the electronic device. As shown in Figure 3, the electronic device may include: a radio frequency (RF) circuit 110, a memory 120, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a processor 170, and a power supply 180, etc.

[0059] RF circuit 110 can be used to send and receive information, or to receive or send signals during a call. Specifically, it receives downlink information from the base station and processes it in processor 170; additionally, it sends uplink data to the base station. Typically, RF circuit 110 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 110 can also communicate wirelessly with networks and other devices.

[0060] The memory 120 can be used to store data, software programs, and modules; it mainly includes a program storage area and a data storage area. The program storage area can store the operating system and at least one application program required for a function, such as sound playback or image playback. The data storage area can store data created based on the use of the electronic device, such as audio data, image data, and a phone book. Furthermore, the electronic device may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. In this embodiment, the memory may include multiple memories, including a first memory and a second memory.

[0061] The input unit 130 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the electronic device. Specifically, the input unit 130 may include a touch screen 131 and other input devices 132. The touch screen 131 can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch screen), and drive the corresponding connected devices according to a pre-set program. Optionally, other input devices 132 may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power switch buttons, etc.), a trackball, a mouse, a joystick, etc.

[0062] Display unit 140 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device. In one example, display unit 140 may include display screen 141, which may be configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar device. Furthermore, touchscreen 131 may cover display screen 141. When touchscreen 131 detects a touch operation on or near it, it transmits the information to processor 170 to determine the type of touch event. Subsequently, processor 170 provides corresponding visual output on display screen 141 based on the type of touch event. Although in the figures, touchscreen 131 and display screen 141 are shown as two separate components to implement the input and output functions of the electronic device, in some embodiments, touchscreen 131 and display screen 141 can be integrated to achieve the input and output functions of the electronic device.

[0063] Sensor 150 may include one or more sensors for providing status assessments of various aspects of the electronic device. Sensor 150 may include a light sensor, which can be used in imaging applications, i.e., as a component of a camera or video camera. Furthermore, sensor 150 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor. Sensor 150 can detect acceleration / deceleration, orientation, on / off state, relative positioning of components, or temperature changes of the electronic device.

[0064] Audio circuitry 160, a speaker, and a microphone provide an audio interface between the user and the electronic device. Audio circuitry 160 converts received audio data into electrical signals and transmits them to the speaker, where the speaker converts them into sound signals for output. On the other hand, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 160, converted into audio data, and output to RF circuitry 110 for transmission to, for example, another mobile phone, or to memory 120 for further processing.

[0065] The processor 170 is the control center of the electronic device, connecting various parts of the device through various interfaces and lines. It executes software programs and / or modules stored in the memory 120, and calls data stored in the memory 120, to perform various functions and process data, thereby providing overall monitoring of the electronic device. Optionally, the processor 170 may include one or more processing units, which may include, but are not limited to: a central processing unit (CPU), a network processing unit (NPU), a graphics processing unit (GPU), an image signal processor (ISP), a tensor processing unit (TPU), a data processing unit (DPU), a digital signal processor (DSP), a microcontroller, or a microprocessor. Furthermore, the processor 170 may also include other hardware circuits or accelerators, such as application-specific integrated circuits (ASICs), complex programmable logic devices (CPLDs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. Optionally, the processor 170 may also be a combination of functions that implement computing, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc.

[0066] The electronic device may also include a power supply 180 (e.g., a battery) to power various components. The power supply 180 can be logically connected to the processor 170 via a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Optionally, the power management system can simultaneously support fast charging and non-fast charging technologies. In practical applications, the power management system can charge the battery in the power supply 180 using either fast charging or non-fast charging technologies.

[0067] Although not shown, the electronic device may also include a wireless fidelity (WiFi) module, a Bluetooth module, etc., which will not be described in detail in the embodiments of this application. Those skilled in the art will understand that the structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0068] Figure 4 is a schematic diagram of a processor according to an embodiment of this application. The processor includes a tensor processing unit and can be applied to the electronic device described above. The tensor processing unit can be coupled to a first memory and a second memory. The first memory stores multidimensional tensors, and the second memory stores processed multidimensional tensors. The first and second memories can be on-chip memories of the processor, or off-chip memories, or one of the first and second memories can be on-chip memories and the other off-chip memories. Figure 4 illustrates an example where both the first and second memories are on-chip memories of the processor.

[0069] The tensor processing unit includes a preprocessing circuit 10, an address generation circuit 20, an access control circuit 30, and a data buffer 40; wherein the preprocessing circuit 10 is coupled to the address generation circuit 20, the address generation circuit 20 is coupled to the access control circuit 30 and the data buffer 40, and the access control circuit 30 is also coupled to the data buffer 40.

[0070] The preprocessing circuit 10 is used to: determine address generation parameters based on the first dimension information of the multidimensional tensor; wherein the address generation parameters may include second dimension information, input dimension data interval, and output dimension data interval. For example, the preprocessing circuit adjusts the adjustment dimension information to obtain the second dimension information, and determines the input dimension data interval and output dimension data interval based on the second dimension information. Optionally, the preprocessing circuit 10 may also be referred to as a split-axis circuit, a split-axis unit, or a split-axis module.

[0071] The first dimension information mentioned above refers to the initial dimension information of the multidimensional tensor. The second dimension information can be the dimension information after processing or adjusting the first dimension information. The product of the size of any dimension in the second dimension information and the capacity of the unit access storage space is less than or equal to the capacity of the data cache. The capacity of the unit access storage space can refer to the capacity of the storage space that can be accessed or read at one time; for example, the unit access storage space can be a cache line, and the capacity of the cache line can be 128 bytes (B). The input dimension data interval can include the intervals between data in multiple dimensions of the multidimensional tensor, such as the interval between two adjacent data in any dimension in the first memory (or input storage space). The output dimension data interval can refer to the intervals between data in multiple dimensions of the processed multidimensional tensor, such as the interval between two adjacent data in any dimension in the second memory (or output storage space).

[0072] In one possible embodiment, the preprocessing circuit 10 is further configured to: perform at least one of the following processes on the first dimension information: reordering, splitting axis, borrowing axis, or padding, to obtain second dimension information; determine the input dimension data interval based on the second dimension information, for example, according to the order of dimensions from smallest to largest, the storage interval between two adjacent data in any dimension of the second dimension information in the first memory is equal to the product of the dimensions of the other dimensions preceding that dimension, and the interval between two adjacent data in the smallest dimension can be equal to 1. Optionally, when the first dimension information is subjected to at least one of the following processes: splitting axis, borrowing axis, or padding, the address generation information may further include at least one of the following: splitting axis point information, borrowing axis point information, and padding data (pad) size of at least one dimension. The above padding process may be padding of at least one dimension, and the corresponding data padding size includes left padding size and / or right padding size. Further, the preprocessing circuit 10 is also configured to determine the output dimension data interval based on the second dimension information.

[0073] In this application, the starting address of the tensor in the first memory can be represented as src_addr (or the starting address of the original tensor in the input memory space), the starting address of the transposed or rearranged tensor in the second memory can be represented as dst_addr (or the starting address of the result tensor in the output memory space), the first dimension information of the tensor can be represented as loop_size (i.e., the size of each dimension of the tensor, arranged from high to low), the input dimension data interval can be represented as loop_src_stride (i.e., the interval of each dimension in the input memory space, corresponding one-to-one with each dimension), and the output dimension data interval can be represented as loop_dst_stride (i.e., the interval of each dimension in the output memory space, corresponding one-to-one with each dimension). Moreover, each dimension may have a left padding size and a right padding size, which can be represented as loop_lp_size and the right padding size as loop_rp_size.

[0074] In one example, taking the transpose or rearrangement of a 5-dimensional tensor as an example, the transpose or rearrangement operation of a 5-dimensional tensor that supports left and right padding of each axis can be expressed by the following formula, where Xn represents the base address and Xd represents the starting address.

[0075] In another example, if a 3D tensor transpose is to be expressed, such as [2,3,4]->[4,3,2] (axis from high to low), it can be expressed with the following parameters: loop size = [2,3,4]; src_stride = [12,4,1]; dst_stride = [1,3,6]; lp_size = [0,0,0]; rp_size = [0,0,0].

[0076] In another example, if we want to express the reshape, broadcast, and transpose of a tensor, such as [5,2]->reshape->[5,1,2]->broadcast->[5,3,2]->transpose->[2,5,3]->right pad lowest axis to 4->[2,5,4], we can use the following parameters: loop size = [2,5,3]; src_stride = [1,2,0]; dst_stride = [20,4,1]; lp_size = [0,0,0]; rp_size = [0,0,1].

[0077] In this tensor processing unit, the address generation circuit 20 is used to: generate address information for multiple data in the multidimensional tensor according to the address generation parameters, wherein the address information includes multiple source addresses and multiple destination addresses. For example, the address generation circuit 20 generates multiple source addresses corresponding to multiple data in the multidimensional tensor based on the second dimension information and the input dimension data interval, such as generating the source address of each data or element in the multidimensional tensor at the granularity, thus generating multiple source addresses corresponding to the multiple data in the multidimensional tensor; the address generation circuit 20 can also generate multiple destination addresses corresponding to the multiple data based on the second dimension information and the output dimension data interval, such as generating the destination address of each data or element at the granularity, thus generating multiple destination addresses corresponding to the multiple data in the multidimensional tensor. Here, the multiple source addresses are the addresses corresponding to the multiple data in the multidimensional tensor in the first memory, and the multiple destination addresses are the addresses corresponding to the multiple data in the second memory, and the multiple source addresses and multiple destination addresses can correspond one-to-one. In one possible example, the address generation circuit 20 may include an SRC address generation circuit for generating multiple source addresses and a DST address generation circuit for generating multiple destination addresses.

[0078] Optionally, for data in multiple dimensions of the multidimensional tensor, or different data in the same dimension, the address generation circuit 20 can generate source and destination addresses corresponding to all data in the multidimensional tensor through multiple operations. In one possible embodiment, the address generation circuit 20 is further configured to: generate multiple batches of address information according to the address generation parameters, each batch of address information including multiple source addresses and multiple destination addresses. Further, for the multiple source addresses and multiple destination addresses in each batch of address information, the address generation circuit 20 can also send the multiple source addresses to the access control circuit 30 and the multiple destination addresses to the data buffer 40.

[0079] In this tensor processing unit, the access control circuit 30 is used to: determine multiple cache mapping information corresponding to the multiple source addresses, and send multiple access requests based on the multiple source addresses. These multiple access requests are used to retrieve multiple data from the first memory. The multiple cache mapping information is also used to indicate the cache addresses corresponding to the multiple source addresses. Optionally, the data cache 40 includes multiple cache lines, and the multiple cache mapping information is used to indicate the cache lines corresponding to the multiple source addresses. Further, the multiple cache mapping information is also used to indicate the cache location (or data mapping address) in the cache lines corresponding to the multiple source addresses, that is, the cache location of the data corresponding to any source address in the cache line. For example, the cache mapping information corresponding to a source address may include the cache line identifier corresponding to the source address.

[0080] In one possible embodiment, the access control circuit 30 can be used to: assign a corresponding cache line identifier to each of the plurality of source addresses, and send a plurality of access requests to the first memory, wherein the plurality of access requests correspond one-to-one with the plurality of source addresses, and any access request may carry the corresponding source address and be used to read the data corresponding to the source address from the first memory.

[0081] Furthermore, the access control circuit 30 is also used to merge the at least two source addresses when the cache lines corresponding to at least two of the plurality of source addresses are the same. For example, the access control circuit 30 can compare the cache line identifiers corresponding to the plurality of source addresses. When at least two of the plurality of source addresses correspond to the same cache line identifier, the access control circuit 30 can merge the at least two source addresses into one source address and send an access request to the first memory based on the merged source address. This access request can be used to read the data corresponding to the at least two source addresses from the first memory. Wherein, the access control circuit 30, when used to merge at least two source addresses, can also be referred to as an address comparison merging circuit.

[0082] Optionally, the functions of the access control circuit 30 and the address generation circuit 20 can be integrated together, for example, the address generation circuit 20 and the access control circuit 30 can be integrated together. In this application, the access control circuit 30 and the address generation circuit 20 are described as two independent circuits.

[0083] In this tensor processing unit, the data buffer 40 is used to: buffer the multiple data according to the multiple cache mapping information, and write the multiple data into the second memory according to the multiple destination addresses. Optionally, the data buffer 40 can also perform data rearrangement on the multiple data.

[0084] In one possible embodiment, the data buffer 40 can receive multiple data corresponding to multiple access requests returned by the first memory, and receive multiple cache mapping information sent by the access control circuit 30, and cache the multiple data according to the multiple cache mapping information; the data buffer 40 can also receive multiple destination addresses sent by the address generation circuit 20, and write the cached multiple data into the second memory according to the multiple destination addresses. For example, the data buffer 40 can send multiple write requests to the second memory through a first interface according to the multiple destination addresses. The first interface can be used to couple with the second memory. Each write request carries one of the multiple data, so that the multiple data can be written into the second memory through the multiple write requests.

[0085] Optionally, as shown in Figure 5, the data cache 40 can include a single-way cache, which comprises K cache lines. These K cache lines can be divided into P banks. Each write to the data cache 40 (i.e., cached data) can be done at the cache line level, thus avoiding bank conflicts. Each read from the data cache 40 (i.e., output data) may result in bank conflicts due to data rearrangement. These conflicts can be eliminated using a multinomial hash-based rearrangement method or a special address mapping scheme. Figure 5 illustrates an example where the data cache 40 includes 128B cache lines (4*32) and 32 banks, each bank containing 4B data.

[0086] Furthermore, as shown in FIG6, the tensor processing unit further includes a state memory 50, which can be coupled between the access control circuit 30 and the data buffer 40. For example, the state memory 50 can be a random access memory (RAM), which may also be referred to as TAG RAM in this application.

[0087] The state memory 50 is used to store cache state information, which indicates the status of the plurality of cache lines, either hit or miss. Furthermore, the access control circuit 30 is also used to query the cache state information, and when the cache line corresponding to any one of the plurality of source addresses is missing, send an access request for that source address, and update the cache line status of that source address in the cache state information to hit. The state memory 50 is also used to output cache mapping information corresponding to that source address when the cache line corresponding to any one of the plurality of source addresses is hit.

[0088] Optionally, the state memory 50 includes multiple repositories; these repositories are used to store and query the status of cache lines corresponding to different source addresses among the multiple source addresses. The multiple repositories can store or determine the status of cache lines corresponding to the multiple source addresses in one or more cycles. For example, the state memory 50 is a TAG RAM, which includes M banks, each bank corresponding to (1 / M)*K cache lines, so that each bank can determine one source address among the multiple source addresses in each cycle. In some scenarios, multiple addresses may map to the same bank, causing congestion. Therefore, a polynomial hash rearrangement method or some special address mapping method can be used to ensure that a batch of addresses is mapped as evenly as possible to these banks.

[0089] Furthermore, as shown in Figure 6, the tensor processing unit also includes a first-in, first-out (FIFO) queue 60, which can be coupled between the state memory 50 and the data buffer 40. In one possible embodiment, the FIFO queue 60 is used to cache the plurality of cache mapping information, for example, to cache the cache mapping information output by the state memory 50 when the state of the cache line corresponding to any one of the plurality of source addresses is a hit. For example, the FIFO queue 60 can be used to record the identifiers of the plurality of cache lines corresponding to the plurality of source addresses, and the mapping position of the data corresponding to each source address in the cache line.

[0090] Optionally, the data buffer 40 is further configured to: when receiving multiple data returned from the first memory, buffer the multiple data according to the multiple buffer mapping information output at the exit of the FIFO queue 60; when the buffer line status corresponding to the multiple source addresses is all commands, or when the multiple data corresponding to the multiple source addresses has been returned, write the multiple data into the second memory according to the multiple destination addresses sent by the address generation circuit 20. The FIFO queue 60 can be used to: remove the multiple buffer mapping information corresponding to the multiple source addresses.

[0091] Furthermore, as shown in Figure 6, the tensor processing unit also includes a sequence-preserving circuit 70. This sequence-preserving circuit 70 can be coupled to the data buffer 40, for example, between a second interface and the data buffer 40. The second interface can be an interface for coupling with a second memory. For example, the sequence-preserving circuit 70 can be a reorder buffer (ROB). In one possible embodiment, the sequence-preserving circuit 70 is used to preserve the order of the plurality of data returned from the first memory before outputting it to the data buffer 40. The plurality of data returned from the first memory may be out of order. In this embodiment, by preserving the order of the plurality of data through the sequence-preserving circuit 70, the correctness of the order of the plurality of data can be guaranteed, thereby ensuring the accuracy of the cache line states stored in the state memory 50.

[0092] In this embodiment, the tensor processing unit in the processor can accelerate various data transfer operations through efficient address generation and automatic packet merging. The address generation circuit 20 can generate multiple (e.g., 32) addresses per frame based on the axis parameters of the multidimensional tensor. The access control circuit 30 can perform real-time address merging to reduce the number of requests. Through the data buffer 40, a complete burst (e.g., 128B) is retrieved each time to achieve large burst source memory access, minimizing small packet access requests, and utilizing the space of the data buffer 40 for data rearrangement and reuse. Data rearrangement is achieved by simultaneously reading data from different banks in multiple cachelines, and large bursts are written to internal memory. The reuse rate of the data buffer 40 is increased and performance is improved by hardware For loop reordering and address scrambling. A FIFO queue scheme similar to a streaming cache is used to simplify the management of the data buffer 40 and reduce the control overhead of the data buffer 40.

[0093] For ease of understanding, the structure and corresponding processing flow of this tensor processing unit are illustrated below using Figure 7. For example, this tensor processing unit may include: a split-axis circuit, an SRC address generation circuit, a DST address generation circuit, a TAG RAM, a FIFO queue, a data cache (dcache), a ROB, an SRC MEM interface, and a DST MEM interface. The processing flow corresponding to this tensor processing unit may include the following steps.

[0094] (1) After the instruction to process the multidimensional tensor is issued, the split axis circuit can perform at least one of the following processes on the multidimensional tensor: split axis, borrow axis, reorder and fill, so as to obtain a set of reordered information of each dimension, such as N loop size, input dimension data interval src_stride, output dimension data interval dst_stride, fill size pad_size, and split axis points of each axis.

[0095] (2) The SRC address generation circuit and the DST address generation circuit can generate source addresses (which can be represented as SRC addresses) and destination addresses (which can be represented as DST addresses) in batches according to the parameters output by the split axis circuit. The above address generation can be performed in parallel (>1 element / cycle).

[0096] (3) TAG RAM can be used to cache the status of cache lines in dcache, recording the hit / miss status of all cache lines in dcache. If there are K cache lines in dcache, there are K hit / miss record entries in TAG RAM, which correspond one-to-one with the cache line identifier (cacheline ID). SRC addresses are entered into TAG RAM in batches for cache miss / hit checks. Before SRC addresses are entered into TAG RAM in batches, a merging operation is first performed on each batch of addresses (e.g., 32 addresses). If some addresses are detected to be in the same cache line (e.g., 128B), these addresses in the same cache line will be merged into one address and entered into TAG RAM for hit / miss checks. The same batch of addresses will be checked in TAG RAM at least once, and if there are 32 addresses in a batch, the TAG RAM will be checked at most 32 times. If the check finds that the address is in a miss state in the TAG RAM, an access request is sent to the first memory (which can be represented as SRC memory). Each access is granular enough for one cacheline (128B), and the cacheline ID in the TAG RAM is marked as hit. If the check finds that the address is already hit in the TAG RAM, no access request is sent to the SRC memory. Regardless of whether it's a hit or a miss, all requests in this batch are marked and output to the next-level request FIFO. Note that since the data may not necessarily be returned from the SRC memory, it could be a fake hit.

[0097] (4) The TAG RAM is divided into M banks, each bank corresponding to (1 / M)*K cache lines. In each cycle, each bank checks one address within the same batch of addresses. In some scenarios, multiple addresses may be mapped to the same bank, causing congestion. Therefore, some polynomial-based hash rearrangement methods or some special address mapping methods can be selected to make a batch of addresses as evenly as possible mapped to these banks.

[0098] (5) The FIFO queue records the cacheline ID and dcache mapping address corresponding to all generated SRC addresses (but does not need to record the actual address). This is because data may not actually be returned from SRC memory, and a buffering mechanism is needed to offset the delay of the data returning from SRC memory. When data is actually returned from SRC memory, the data can be written to dcache according to the dcache mapping address given at the exit of the FIFO queue; if the cache line status corresponding to the current batch address at the exit of the FIFO queue is all hit (or partially hit and all missed requests are actually returned), then dcache gathers the current batch address, writes it to the second memory (which can be represented as DST memory) through the DST MEM interface according to the DST address generated by the DST address generation circuit, and at the same time, the FIFO queue removes this batch of requests.

[0099] (6) dcache is a one-way data cache with K cachelines. To enable efficient data rearrangement, it is divided into P banks (P can be 32). Each write to dcache is at the cacheline level, so there will be no bank conflicts. However, reading from dcache may result in bank conflicts due to data rearrangement. To avoid bank conflicts when reading from dcache, some multinomial hash-based rearrangement methods or some special address mapping schemes can be selected to eliminate them.

[0100] (7) If the SRC memory is returned out of order, the data received from the SRC MEM interface needs to be rearranged into order through ROB before being sent to dcache to ensure data correctness. Otherwise, the fake hit judgment of TAG RAM will be incorrect.

[0101] Optionally, referring to Figure 7 and as shown in Figure 8, when multiple processing cores in an electronic device have parallel multidimensional tensor processing requirements, these multiple processing cores can also reuse the tensor processing unit. For example, these multiple processing cores can time-division multiplex multiple circuits of the tensor processing unit. Address differentiation can be performed in the data cache dcache and FIFO queue. For example, these multiple processing cores can use multiple different cache regions in the dcache (e.g., represented as dcache0 to dcache3). Correspondingly, the multiple DST MEM interfaces corresponding to these multiple processing cores can be represented as DST 0MEM interface to DST 3MEM interface. Different cache regions and DST MEM interfaces can be selected by a selector.

[0102] Furthermore, the core idea of ​​the preprocessing circuit 10 is to automatically divide each dimension into blocks and change the order of data reading to improve the cache hit rate and reduce capacity misses within the limited data cache 40. In this application, the preprocessing circuit 10 can be hardware (e.g., a hardened axis splitting circuit), which uses a fixed, hardened algorithm to provide the axes to be split and their splitting values, and arranges all axes in the optimal order. Alternatively, the preprocessing circuit 10 (e.g., the axis splitting circuit) can be implemented in software, although this method is less efficient than hardware implementation.

[0103] For example, as shown in Figure 9, when the multidimensional tensor is a matrix, if we want to transpose the SRC matrix (256, 128) to the DST matrix (128, 256) (i.e., (256, 128) -> (128, 256)), then the dimension information of the matrix is ​​num = [128, 256], src_stride = [1, 128], and dst_stride = [256, 1]. If we directly calculate the input address based on the output address and access the SRC matrix at the granularity of cache lines, that is, access one cache line at a time, and this cache line includes one column of data in the first column, then when accessing based on the address of the second column of (256, 128), the access volume has reached 128B * 256 = 32KB, taking each cache line as an example of 128B. If the cache size is less than 32KB (i.e., cache size < 32KB), then a capacity failure has occurred. Every request in that column will be missing, and in fact, all requests except for the first column will be missing, resulting in a lot of redundant access.

[0104] If the axes of the above SRC matrix are adjusted, as shown in Figure 10, for example, the axes 128 and 256 are decomposed into 4*32 and 8*32 respectively, that is, the above num=[128,256],src_stride=[1,128],dst_stride=[256,1] is converted into num=[4,32,8,32],src_stride=[32,1,128*32,128],dst_stride=[256*32,256,32,1],and then the decomposed axes are rearranged to get num=[4,8,32,32],src_stride=[32,128*32,1,128],dst_stride=[256*32,32,256,1]. Thus, when retrieving data, a [32*32] sub-region is executed first, resulting in a cache size > 128B * 32 = 4KB, preventing capacity invalidation. Essentially, the above adjustment involves splitting and reordering the two nested loops. This demonstrates that loop reordering plays a crucial role in improving data locality for transpose and rearrangement operations.

[0105] In this embodiment, automatic shaft removal can be achieved through a hardening process. In one possible embodiment, the operation may include the following steps.

[0106] (1) Cacheline borrowing axes is performed in both the SRC and DST directions. Because when the size of the lower axis is relatively small, if the axis is directly shifted and reordered, the effective data at the input and output of the tensor processing unit will be smaller, thus reducing performance. In this case, it is necessary to merge the higher axis into the cacheline, or to decompose the higher axis and borrow the axis, thereby increasing utilization.

[0107] (2) Axis shifting and reordering: Analyze the parameters from the input and output ends of the tensor processing unit, move the axes that did not participate in the axis borrowing and splitting (irrelevant axes) to the outer loop, and put the small tiles in the inner loop to improve the utilization rate.

[0108] (3) Pack the data cache size (dcache size) in the SRC direction and DST direction respectively to increase cache utilization.

[0109] After the above calculations, the output can include: the loop size, src stride, dst stride, lp size, and rp size after the order is adjusted; and the split axis value (div size) for each axis (if there is a split axis, then div size < the corresponding loop size; if there is no split axis, then div size = 0). Figure 1 above shows the generation order of relative addresses under different split axis conditions, including no split axis, splitting one axis, and splitting two axes. Since this split axis operation can be executed in parallel with the actual handling process, a longer execution time can be reserved without significant hardware overhead.

[0110] Furthermore, during the address generation process of the address generation circuit 20, if the borrowing axis is not considered, a batch of source addresses and destination addresses can be generated in parallel according to the address generation parameters using parallel multiplication-addition or modulo operations. If the borrowing axis is considered, when calculating the address of a certain dimension, if a borrowing axis point is encountered, the borrowing axis point can be automatically detected and the user can jump to a new address.

[0111] In one possible embodiment, as shown in Figure 11, considering axis splitting and borrowing, the unit used for splitting, borrowing, reordering, and filling processes on each dimension may include two levels of control: an overall control unit for selecting the axis to split; and a control unit for each axis to determine calculation parameters such as the size (i.e., axis_size) of each axis. The overall control unit can be controlled via carry-over from the highest axis.

[0112] In this embodiment, the preprocessing circuit 10 determines address generation parameters, including second dimension information, input dimension data interval, and output dimension data interval, based on the first dimension information of the multidimensional tensor. The address generation circuit 20 generates multiple source addresses and multiple destination addresses corresponding to multiple data in the multidimensional tensor based on the address generation parameters. The access control circuit determines multiple cache mapping information corresponding to the multiple source addresses and sends multiple access requests for retrieving multiple data from the first memory based on the multiple source addresses. The data cache caches the multiple data based on the multiple cache mapping information and writes the multiple data into the second memory based on the multiple destination addresses, thereby achieving efficient processing of the multidimensional tensor and improving cache hit rate, reducing capacity misses, and improving performance in the limited data cache 40.

[0113] Based on this, embodiments of this application also provide a tensor processing method, which can be applied to the processor with tensor processing unit provided above. The method includes: adjusting the first dimension information of the multidimensional tensor to obtain second dimension information, and determining an input dimension data interval based on the second dimension information, wherein the product between the size of any dimension in the second dimension information and the capacity of a unit access storage space is less than or equal to the capacity of the data cache, and the input dimension data interval is the interval between data of multiple dimensions in the multidimensional tensor; generating multiple source addresses corresponding to multiple data in the multidimensional tensor based on the second dimension information and the input dimension data interval, wherein the multiple source addresses are addresses corresponding to the multiple data in a first memory; determining multiple cache mapping information corresponding to the multiple source addresses, and sending multiple access requests based on the multiple source addresses, wherein the multiple access requests are used to retrieve the multiple data from the first memory, and the cache mapping information is used to indicate the cache address corresponding to the multiple source addresses; and caching the multiple data in a data cache based on the multiple cache mapping information.

[0114] Furthermore, the method also includes: determining the output dimension data interval based on the second dimension information, wherein the output dimension data interval is the interval between data of multiple dimensions in the processed multidimensional tensor; generating multiple destination addresses corresponding to the multiple data based on the second dimension interval and the output dimension data interval, wherein the multiple destination addresses are addresses for writing the multiple data into the second memory; and writing the multiple data into the second memory based on the multiple destination addresses.

[0115] Optionally, adjusting the first dimension information of the multidimensional tensor to obtain the second dimension information includes: performing at least one of the following processes on the first dimension information of the multidimensional tensor: reordering, splitting axes, borrowing axes, or filling, to obtain address generation parameters. The address generation parameters further include the second dimension information, and at least one of the following: splitting axis point information, borrowing axis point information, and the size of the fill data for at least one dimension.

[0116] Optionally, generating address information based on the address generation parameters includes: generating multiple batches of address information based on the address generation parameters, wherein each batch of address information includes multiple source addresses and multiple destination addresses.

[0117] In one possible embodiment, the data cache includes multiple cache lines, the multiple cache mapping information used to indicate the cache lines corresponding to the multiple source addresses. The method further includes: querying cache status information, the cache status information used to indicate the status of the multiple cache lines, the status being either hit or missing; when the status of the cache line corresponding to any of the multiple source addresses is missing, sending an access request corresponding to that source address, and updating the status of the cache line corresponding to that source address in the cache status information to hit.

[0118] Furthermore, in one possible embodiment, the method further includes: storing and querying the status of cache lines corresponding to different source addresses among the multiple source addresses through multiple repositories.

[0119] In one possible embodiment, the plurality of cache mapping information is used to indicate the cache lines corresponding to the plurality of source addresses, and the method further includes: merging the at least two source addresses when the cache lines corresponding to at least two of the plurality of source addresses are the same.

[0120] In one possible embodiment, the plurality of cache mapping information is cached in a FIFO queue, and the plurality of data are cached in a data cache according to the plurality of cache mapping information, including: when the first memory returns the plurality of data, the plurality of data are cached according to the plurality of cache mapping information output by the FIFO queue.

[0121] In one possible embodiment, the method further includes: maintaining the order of the plurality of data returned from the first memory.

[0122] In this embodiment, the preprocessing circuit adjusts the first dimension information of the multidimensional tensor to obtain the second dimension information, and determines the input dimension data interval based on the second dimension information. The product between the size of any dimension in the second dimension information and the capacity of the unit access storage space is less than or equal to the capacity of the data cache. The address generation circuit generates multiple source addresses corresponding to multiple data in the multidimensional tensor based on the second dimension information and the input dimension data interval. The access control circuit determines multiple cache mapping information corresponding to the multiple source addresses, and sends multiple access requests for retrieving multiple data from the first memory based on the multiple source addresses. The data cache caches the multiple data based on the multiple cache mapping information, thereby achieving general and efficient processing of multidimensional tensors, improving cache hit rate, reducing capacity misses, and improving performance in a limited data cache.

[0123] In another embodiment of this application, an electronic device is provided, comprising a circuit board and a processor disposed on the circuit board. It is understood that all relevant content related to the above-described processor embodiments can be referenced in the embodiments of the method and the electronic device, and will not be repeated here.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed.

[0125] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. This readable storage medium may include various media capable of storing program code, such as a USB flash drive, external hard drive, read-only memory, random access memory, magnetic disk, or optical disk. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product.

[0127] In another embodiment of this application, a readable storage medium is also provided, which stores computer-executable instructions that are executed by a device (which may be a microcontroller, chip, etc.) or a processor when executing the steps in the above method embodiments.

[0128] In another embodiment of this application, a computer program product is also provided, the computer program product including computer instructions stored in a readable storage medium; at least one processor of the device can read the computer instructions from the readable storage medium, and the at least one processor executes the computer instructions to cause the device to perform the steps in the above method embodiments.

[0129] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A processor, characterized in that, include: A tensor processing unit for implementing multidimensional tensor processing, the tensor processing unit includes: a preprocessing circuit, an address generation circuit, an access control circuit, and a data buffer; The preprocessing circuit is used to adjust the first dimension information of the multidimensional tensor to obtain the second dimension information, and determine the input dimension data interval based on the second dimension information. The product between the size of any dimension in the second dimension information and the capacity of the unit access storage space is less than or equal to the capacity of the data buffer. The input dimension data interval is the interval between the data of multiple dimensions in the multidimensional tensor. The address generation circuit is used to generate multiple source addresses corresponding to multiple data in the multidimensional tensor based on the second dimension information and the input dimension data interval, wherein the multiple source addresses are the addresses corresponding to the multiple data in the first memory; The access control circuit is used to determine multiple cache mapping information corresponding to the multiple source addresses, and send multiple access requests according to the multiple source addresses. The multiple access requests are used to obtain the multiple data from the first memory, and the cache mapping information is used to indicate the cache address corresponding to the multiple source addresses in the data cache. The data cache is used to cache the multiple data according to the multiple cache mapping information.

2. The processor according to claim 1, characterized in that, The preprocessing circuit is also used for: The second dimension information is obtained by performing at least one of the following processes on the first dimension information of the multidimensional tensor: reordering, splitting axes, borrowing axes, or filling.

3. The processor according to claim 1 or 2, characterized in that, The preprocessing circuit is further configured to determine the output dimension data interval based on the second dimension information, wherein the output dimension data interval is the interval between the data of multiple dimensions in the processed multidimensional tensor; The address generation circuit is further configured to generate multiple destination addresses corresponding to the multiple data based on the second dimension interval and the output dimension data interval, wherein the multiple destination addresses are the addresses where the multiple data are written into the second memory; The data buffer is further configured to write the plurality of data into the second memory according to the plurality of destination addresses.

4. The processor according to any one of claims 1-3, characterized in that, The data buffer includes multiple cache lines, and the multiple cache mapping information is used to indicate the cache lines corresponding to the multiple source addresses. The tensor processing unit also includes a state memory. The state memory is used to store cache state information, which is used to indicate the state of the plurality of cache lines, wherein the state is a hit or a miss. The access control circuit is also used to query the cache status information, and when the status of the cache line corresponding to any one of the multiple source addresses is missing, send an access request corresponding to the source address, and make the status of the cache line corresponding to the source address in the cache status information more accurate.

5. The processor according to claim 4, characterized in that, The state memory includes multiple storage stores; The multiple repositories are used to store and query the status of cache lines corresponding to different source addresses among the multiple source addresses.

6. The processor according to any one of claims 1-5, characterized in that, The multiple cache mapping information is also used to indicate the cache lines corresponding to the multiple source addresses; The access control circuit is further configured to merge the at least two source addresses when at least two of the plurality of source addresses correspond to the same cache line.

7. The processor according to any one of claims 1-6, characterized in that, The tensor processing unit also includes a first-in-first-out (FIFO) queue; The FIFO queue is used to cache the multiple cache mapping information; The data buffer is further configured to cache the plurality of data according to the plurality of cache mapping information output by the FIFO queue when the first memory returns the plurality of data.

8. The processor according to any one of claims 1-7, characterized in that, The tensor processing unit also includes a sequence-preserving circuit. The order-preserving circuit is used to preserve the order of the plurality of data returned by the first memory and then output them to the data buffer.

9. The processor according to any one of claims 1-8, characterized in that, The address generation circuit is further configured to generate multiple batches of address information corresponding to multiple batches of data in the multidimensional tensor based on the second dimension information and the input dimension data interval, wherein each batch of address information includes multiple source addresses of a batch of data.

10. A tensor processing method, characterized in that, Applied in a processor, the method includes: The first dimension information of the multidimensional tensor is adjusted to obtain the second dimension information, and the input dimension data interval is determined according to the second dimension information. The product between the size of any dimension in the second dimension information and the capacity of the unit access storage space is less than or equal to the capacity of the data cache. The input dimension data interval is the interval between the data of multiple dimensions in the multidimensional tensor. Based on the second dimension information and the input dimension data interval, multiple source addresses corresponding to multiple data in the multidimensional tensor are generated, and the multiple source addresses are the addresses corresponding to the multiple data in the first memory; Determine multiple cache mapping information corresponding to the multiple source addresses, and send multiple access requests according to the multiple source addresses. The multiple access requests are used to obtain the multiple data from the first memory, and the cache mapping information is used to indicate the cache address corresponding to the multiple source addresses. The multiple data are cached in the data cache according to the multiple cache mapping information.

11. The method according to claim 10, characterized in that, The process of adjusting the first dimension information of the multidimensional tensor to obtain the second dimension information includes: The second dimension information is obtained by performing at least one of the following processes on the first dimension information of the multidimensional tensor: reordering, splitting axes, borrowing axes, or filling.

12. The method according to claim 10 or 11, characterized in that, The method further includes: The output dimension data interval is determined based on the second dimension information, and the output dimension data interval is the interval between the data of multiple dimensions in the processed multidimensional tensor. Based on the second dimension interval and the output dimension data interval, multiple destination addresses corresponding to the multiple data are generated, and the multiple destination addresses are the addresses where the multiple data are written into the second memory; The multiple data are written into the second memory according to the multiple destination addresses.

13. The method according to any one of claims 10-12, characterized in that, The data cache includes multiple cache lines, and the multiple cache mapping information is used to indicate the cache lines corresponding to the multiple source addresses. The method further includes: Query cache status information, which indicates the status of the plurality of cache lines, wherein the status is hit or missing. When the cache line corresponding to any of the multiple source addresses is missing, an access request corresponding to the source address is sent, and the cache line corresponding to the source address in the cache status information is updated to a hit.

14. The method according to claim 13, characterized in that, The method further includes: The status of cache lines corresponding to different source addresses among the multiple source addresses is stored and queried through multiple repositories.

15. The method according to any one of claims 10-14, characterized in that, The plurality of cache mapping information is used to indicate the cache lines corresponding to the plurality of source addresses, and the method further includes: When at least two of the multiple source addresses correspond to the same cache line, the at least two source addresses are merged.

16. The method according to any one of claims 10-15, characterized in that, The multiple cache mapping information is cached in a first-in-first-out (FIFO) queue, and the step of caching the multiple data in the data cache according to the multiple cache mapping information includes: When the first memory returns the plurality of data, the plurality of data are cached in the data cache according to the plurality of cache mapping information output by the FIFO queue.

17. The method according to any one of claims 10-16, characterized in that, The method further includes: The multiple data returned by the first memory are kept in order.

18. The method according to any one of claims 10-17, characterized in that, The step of generating multiple source addresses corresponding to multiple data in the multidimensional tensor based on the second dimension information and the input dimension data interval includes: Based on the second dimension information and the input dimension data interval, multiple batch address information corresponding to multiple batches of data in the multidimensional tensor is generated respectively. Each batch address information includes multiple source addresses of a batch of data.

19. An electronic device, characterized in that, The electronic device includes a circuit board and a processor as described in any one of claims 1 to 9 disposed on the circuit board.

20. A readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on the device, cause the device to perform the method as described in any one of claims 10-18.

21. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed on a device, cause the device to perform the method as described in any one of claims 10-18.

Images