Data processing apparatus, data processing method and related product

Abstract

The present disclosure discloses a data processing apparatus, a data processing method and related products. The data processing apparatus can be implemented as a computing apparatus included in a combined processing apparatus, which can further include an interface apparatus and other processing apparatuses. The computing apparatus interacts with the other processing apparatuses to jointly complete a user-specified computing operation. The combined processing apparatus can further include a storage apparatus connected with the computing apparatus and the other processing apparatuses respectively, for storing data of the computing apparatus and the other processing apparatuses. The scheme of the present disclosure provides a special instruction for data fusion related operations, which can simplify processing and improve the processing efficiency of the machine.

Description

Technical Field

[0001] This disclosure generally relates to the field of data processing. More specifically, this disclosure relates to data processing apparatus, data processing methods, chips, and circuit boards. Background Technology

[0002] In recent years, the rapid development of deep learning has led to leaps in the performance of algorithms in fields such as computer vision and natural language processing. However, deep learning algorithms are computationally and storage-intensive tools. As information processing tasks become increasingly complex and the requirements for real-time performance and accuracy of algorithms continue to rise, neural networks are often designed to be deeper and deeper, resulting in ever-increasing computational and storage demands. This makes it difficult to directly apply existing deep learning-based artificial intelligence technologies to devices with limited hardware resources, such as mobile phones, satellites, or embedded devices.

[0003] Therefore, the compression, acceleration, and optimization of deep neural network models have become extremely important. Numerous studies have attempted to reduce the computational and storage requirements of neural networks without compromising model accuracy, which is of great significance for the engineering application of deep learning technology in embedded and mobile devices. Sparsity is one such method for lightweighting models.

[0004] Network parameter sparsification reduces redundant components in large networks through appropriate methods, thereby lowering the network's computational and storage requirements. Existing hardware and / or instruction sets cannot effectively support sparsification processing and / or related processing. Summary of the Invention

[0005] In order to at least partially solve one or more of the technical problems mentioned in the background art, the present disclosure provides a data processing apparatus, a data processing method, a chip, and a board.

[0006] In a first aspect, this disclosure discloses a data processing apparatus, comprising: a control circuit configured to parse a fusion instruction, the fusion instruction instructing the performance of fusion processing on multiple streams of data to be fused, wherein at least one operand of the fusion instruction includes at least one descriptor indicating at least one of the following: shape information and spatial information of tensor data; a tensor interface circuit configured to parse the descriptor; a storage circuit configured to store information before and / or after the fusion processing; and an arithmetic circuit configured to, based on the parsed descriptor and according to the fusion instruction, group data elements in the multiple streams of data to be fused into a single ordered fused data stream according to their corresponding indices, wherein each fused data element is characterized using an arithmetic structure element, the data element including any of scalar, vector, or higher-dimensional data.

[0007] In a second aspect, this disclosure provides a chip that includes the data processing apparatus of any of the embodiments of the first aspect.

[0008] In a third aspect, this disclosure provides a board including the chip of any of the embodiments of the second aspect above.

[0009] In a fourth aspect, this disclosure provides a data processing method, comprising: parsing a fusion instruction, the fusion instruction instructing the performance of fusion processing on multiple streams of data to be fused, wherein at least one operand of the fusion instruction includes at least one descriptor, the descriptor indicating at least one of the following: shape information and spatial information of tensor data; parsing the descriptor; based on the parsed descriptor, according to the fusion instruction, grouping data elements in the multiple streams of data to be fused into a single ordered fused data stream according to their corresponding indices, wherein each fused data element is represented using an operational structure element, the data element including any of scalar, vector, or higher-dimensional data; and outputting the fused data.

[0010] Using the data processing apparatus, data processing method, chip, and board provided above, this disclosure provides a fusion instruction for performing operations related to the merge sorting and fusion of multiple data streams, wherein at least one operand of the fusion instruction is described by a descriptor. In some embodiments, the fusion instruction is a hardware instruction, implemented through dedicated hardware circuitry for data fusion processing. In some embodiments, the data processing apparatus can, according to the fusion instruction, merge multiple ordered data streams into a single ordered fused data stream in indexed order, and the data elements in the fused data stream can be represented in the form of an operational structure, thereby facilitating subsequent computational processing.

[0011] In some embodiments, the merging instruction may include an operation mode bit to indicate that the merging instruction is a merge sort merging process, or the merging instruction itself may indicate a merge sort merging process operation.

[0012] In some embodiments, the data elements to be fused can be vectors or higher-dimensional data. For example, the data elements in the data to be fused can be sparsed effective data elements in radar-based object detection, so that the data fusion instructions and fusion operations provided in this disclosure embodiment can support related processing in radar algorithms.

[0013] Processing can be simplified by providing dedicated fusion instructions to perform operations related to the fusion of multi-channel data. Furthermore, processing can be accelerated by providing dedicated hardware implementations for data fusion-related operations, thereby improving machine processing efficiency. Attached Figure Description

[0014] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0015] Figure 1 This is a structural diagram of the board card shown in the embodiment of this disclosure;

[0016] Figure 2 This is a structural diagram illustrating the combined processing apparatus according to an embodiment of the present disclosure;

[0017] Figure 3 This is a schematic diagram illustrating the internal structure of a processor core in a single-core or multi-core computing device according to embodiments of this disclosure;

[0018] Figure 4 A schematic diagram of a data storage space according to an embodiment of this disclosure is shown;

[0019] Figure 5 A schematic diagram of data blocks in a data storage space according to an embodiment of this disclosure is shown;

[0020] Figure 6 An exemplary schematic diagram of data fusion processing according to embodiments of this disclosure is shown;

[0021] Figure 7 This diagram illustrates the structure of a data processing apparatus according to an embodiment of the present disclosure.

[0022] Figure 8 This disclosure shows an exemplary circuit diagram for data fusion processing according to one embodiment;

[0023] Figure 9 The instructions of each descriptor in the fusion command are illustrated exemplarily; and

[0024] Figure 10 An exemplary flowchart of a data processing method according to an embodiment of this disclosure is shown. Detailed Implementation

[0025] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0026] It should be understood that the terms "first," "second," "third," and "fourth," etc., that may appear in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0027] It should also be understood that the terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0028] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0029] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

[0030] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0031] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0032] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0033] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a storage device 204.

[0034] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0035] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0036] Processing device 203, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0037] Storage device 204 is used to store data to be processed. It may be DRAM or DDR memory, typically 16G or larger in size, and is used to store data of computing device 201 and / or processing device 203.

[0038] Figure 3 The diagram shows the internal structure of the processor core when the computing device 201 is a single-core or multi-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three main modules: a control module 31, an arithmetic module 32, and a storage module 33.

[0039] The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 fetches instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.

[0040] The computation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 322 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0041] Storage module 33 is used to store or move relevant data, including neuron RAM (NRAM) 331, weight RAM (WRAM) 332, and direct memory access (DMA) module 333. NRAM 331 is used to store input neurons, output neurons, and intermediate results after computation; WRAM 332 is used to store the convolution kernels of the deep learning network, i.e., the weights; DMA 333 is connected to DRAM 204 through bus 34 and is responsible for data transfer between computing device 301 and DRAM 204.

[0042] Traditional processor instructions are designed to perform basic single-data scalar operations. Here, a single-data scalar operation means that each operand of the instruction is a scalar data. However, with the development of artificial intelligence technology, in tasks such as image processing and pattern recognition, the operands are often multi-dimensional vector data (i.e., tensor data). Using only scalar or vector operations cannot enable the hardware to efficiently complete the computational tasks. Therefore, in the embodiments disclosed herein, a fusion instruction involving tensor data is provided. At least one operand of the fusion instruction includes tensor data, which is indicated by at least one descriptor. Specifically, the descriptor may indicate at least one of the following: shape information of the tensor data, and spatial information of the tensor data. The shape information of the tensor data can be used to determine the data address of the tensor data corresponding to the operand in the data storage space. The spatial information of the tensor data can be used to determine the dependencies between instructions, and thus determine, for example, the execution order of the instructions.

[0043] In one possible implementation, the spatial information of tensor data can be indicated by a spatial identifier (ID). A spatial ID, also known as a spatial alias, refers to a spatial region used to store the corresponding tensor data. This spatial region can be a continuous space or multiple segments; this disclosure does not restrict the specific composition of the spatial region. Different spatial IDs indicate that the spatial regions they point to are independent of each other.

[0044] The following section will describe in detail, with reference to the accompanying drawings, various possible ways to implement the shape information of tensor data.

[0045] Tensors can contain various forms of data composition. Tensors can be of different dimensions; for example, a scalar can be considered a 0-dimensional tensor, a vector a 1-dimensional tensor, and a matrix a 2-dimensional or higher tensor. The shape of a tensor includes information such as its dimensions and the size of each dimension. For example, for a 3D tensor:

[0046] x3=[[[1,2,3,[4,5,6]];[[7,8,9],[10,11,12]]]

[0047] The shape or dimensions of this tensor can be represented as X3 = (2, 2, 3), meaning that the tensor is a three-dimensional tensor indicated by three parameters, with the first dimension having a size of 2, the second dimension having a size of 2, and the third dimension having a size of 3. When storing tensor data in memory, the shape of the tensor data cannot be determined based on its data address (or storage area), and consequently, the relationships between multiple tensor data cannot be determined, resulting in low processor efficiency in accessing tensor data.

[0048] In one possible implementation, a descriptor can be used to indicate the shape of N-dimensional tensor data, where N is a positive integer, such as N = 1, 2, or 3, or zero. The three-dimensional tensor in the example above can be represented by the descriptor (2, 2, 3). It should be noted that this disclosure does not impose any restrictions on how the descriptor indicates the shape of the tensor.

[0049] In one possible implementation, the value of N can be determined based on the dimension (also known as the order) of the tensor data, or it can be set according to the needs of using the tensor data. For example, when N is 3, the tensor data is three-dimensional, and the descriptor can be used to indicate the shape (e.g., offset, size, etc.) of the three-dimensional tensor data in the three dimensions. It should be understood that those skilled in the art can set the value of N according to actual needs, and this disclosure does not limit this.

[0050] Although tensor data can be multidimensional, because the layout of memory is always one-dimensional, there is a correspondence between tensors and their storage in memory. Tensor data is typically allocated in contiguous storage space, meaning that tensor data can be expanded in one dimension (e.g., row-major order) and stored in memory.

[0051] The relationship between a tensor and its underlying storage can be represented by the dimension offset, dimension size, and dimension stride. The dimension offset refers to the offset relative to a reference position within that dimension. The dimension size refers to the number of elements in that dimension. The dimension stride refers to the interval between adjacent elements within that dimension. For example, the stride of the three-dimensional tensor above is (6, 3, 1), meaning the stride of the first dimension is 6, the stride of the second dimension is 3, and the stride of the third dimension is 1.

[0052] Figure 4 A schematic diagram of a data storage space according to an embodiment of this disclosure is shown. Figure 4 As shown, data storage space 41 stores two-dimensional data in row-major order, which can be represented by (x, y) (where the X-axis is horizontal to the right and the Y-axis is vertical downwards). The size in the X-axis direction (the size of each row, or the total number of columns) is ori_x (not shown in the figure), and the size in the Y-axis direction (the total number of rows) is ori_y (not shown in the figure). The starting address PA_start (base address) of data storage space 41 is the physical address of the first data block 42. Data block 43 is a portion of the data in data storage space 41. Its offset 45 in the X-axis direction is denoted as offset_x, its offset 44 in the Y-axis direction is denoted as offset_y, its size in the X-axis direction is denoted as size_x, and its size in the Y-axis direction is denoted as size_y.

[0053] In one possible implementation, when using a descriptor to define data block 43, the data reference point of the descriptor can be the first data block of data storage space 41, and the reference address of the descriptor can be agreed to be the starting address PA_start of data storage space 41. Then, the content of the descriptor of data block 43 can be determined by combining the size of data storage space 41 on the X-axis (ori_x), the size of data storage space 41 on the Y-axis (ori_y), and the offsets of data block 103 on the Y-axis (offset_y), X-axis (offset_x), X-axis (size_x), and Y-axis (size_y).

[0054] In one possible implementation, the content of the descriptor can be represented using the following formula (1):

[0055]

[0056] It should be understood that although the content of the descriptor in the above example represents a two-dimensional space, those skilled in the art can set the specific dimension represented by the content of the descriptor according to the actual situation, and this disclosure does not limit this.

[0057] In one possible implementation, the reference address of the data reference point of the descriptor in the data storage space can be agreed upon. Based on the reference address, the content of the tensor data descriptor is determined according to the positions of at least two vertices at diagonal positions in N dimensional directions relative to the data reference point.

[0058] For example, the base address PA_base of the data reference point in the data storage space can be agreed upon. For instance, a piece of data (e.g., data at position (2, 2)) can be selected in data storage space 41 as the data reference point, and the physical address of that data in the data storage space can be used as the base address PA_base. The location of the data reference point can be determined based on the positions of the two diagonally opposite vertices relative to the data reference point. Figure 4 The contents of the descriptor for data block 43 are determined first. This involves identifying the positions of at least two diagonal vertices of data block 43 relative to the data reference point. For example, the positions of the diagonal vertices from the top left to the bottom right relative to the data reference point are used, where the relative positions of the top left vertex are (x_min, y_min) and the bottom right vertex are (x_max, y_max). Then, the contents of the descriptor for data block 43 can be determined based on the reference address PA_base, the relative positions of the top left and bottom right vertices (x_min, y_min), and (x_max, y_max).

[0059] In one possible implementation, the contents of the descriptor (based on the base address PA_base) can be represented using the following formula (2):

[0060]

[0061] It should be understood that although the above example uses the top left and bottom right corners as the two diagonal vertices to determine the content of the descriptor, those skilled in the art can set the specific vertices of at least two diagonal vertices according to actual needs, and this disclosure does not limit this.

[0062] In one possible implementation, the content of the tensor data descriptor can be determined based on the reference address of the descriptor's data reference point in the data storage space, and the mapping relationship between the data description location and the data address of the tensor data indicated by the descriptor. The mapping relationship between the data description location and the data address can be set according to actual needs. For example, when the tensor data indicated by the descriptor is three-dimensional spatial data, the function f(x, y, z) can be used to define the mapping relationship between the data description location and the data address.

[0063] In one possible implementation, the content of the descriptor can be represented using the following formula (3):

[0064]

[0065] In one possible implementation, the descriptor is also used to indicate the address of N-dimensional tensor data, wherein the content of the descriptor also includes at least one address parameter representing the address of the tensor data, for example, the content of the descriptor can be the following equation (4):

[0066]

[0067] PA is the address parameter. The address parameter can be a logical address or a physical address. When resolving the descriptor, PA can be any one of the vertices, midpoints, or preset points of the vector shape, combined with the shape parameters in the X and Y directions to obtain the corresponding data address.

[0068] In one possible implementation, the address parameter of the tensor data includes the reference address of the data reference point of the descriptor in the data storage space of the tensor data, and the reference address includes the starting address of the data storage space.

[0069] In one possible implementation, the descriptor may also include at least one address parameter representing the address of the tensor data, for example, the content of the descriptor may be the following equation (5):

[0070]

[0071] PA_start is the base address parameter, which will not be elaborated further.

[0072] It should be understood that those skilled in the art can set the mapping relationship between data description location and data address according to the actual situation, and this disclosure does not impose any restrictions on this.

[0073] In one possible implementation, a pre-defined base address can be set within a task. All descriptors in instructions within this task use this base address, and the descriptor content can include shape parameters based on this base address. This base address can be determined by setting environment parameters for this task. For a description and usage of the base address, please refer to the above embodiments. In this implementation, the descriptor content can be mapped to data addresses more quickly.

[0074] In one possible implementation, the base address can be included in the content of each descriptor, allowing each descriptor to have a different base address. Compared to using environment parameters to set a common base address, this approach allows descriptors to describe data more flexibly and utilize a larger data address space.

[0075] In one possible implementation, the data address in the data storage space corresponding to the operand of the processing instruction can be determined based on the content of the descriptor. The calculation of the data address is automatically performed by the hardware, and the calculation method will differ depending on the representation of the descriptor content. This disclosure does not limit the specific calculation method of the data address.

[0076] For example, if the content of the descriptor in the operand is represented using formula (1), and the offsets of the tensor data indicated by the descriptor in the data storage space are offset_x and offset_y respectively, and the size is size_x*size_y, then the starting data address PA1 of the tensor data indicated by the descriptor in the data storage space is... (x,y) The following formula (6) can be used to determine it:

[0077] PA1 (x,y) =PA_start+(offset_y-1)*ori_x+offset_x (6)

[0078] The data starting address PA1 is determined according to the above formula (6). (x,y) By combining the offsets offset_x and offset_y, as well as the storage area sizes size_x and size_y, the storage area of ​​the tensor data indicated by the descriptor in the data storage space can be determined.

[0079] In one possible implementation, when the operand also includes a data description location for the descriptor, the data address of the corresponding data in the data storage space can be determined based on the content of the descriptor and the data description location. In this way, a portion (e.g., one or more data items) of the tensor data indicated by the descriptor can be processed.

[0080] For example, the content of the descriptor in the operand is represented using formula (2), the tensor data indicated by the descriptor has offsets of offset_x and offset_y in the data storage space, and the size is size_x*size_y. The data description position of the descriptor included in the operand is (x q y q If the tensor data indicated by the descriptor is located at the data storage address PA2, then... (x,y) The following formula (7) can be used to determine it:

[0081] PA2 (x,y) =PA_start+(offset_y+y q -1)*ori_x+(offset_x+x q (7)

[0082] In one possible implementation, a descriptor can indicate chunks of data. Data chunking can effectively speed up computation and improve processing efficiency in many applications. For example, in graphics processing, convolution operations often use data chunking for fast processing.

[0083] Figure 5 This diagram illustrates data blocks in a data storage space according to embodiments of this disclosure. Figure 5 As shown, data storage space 500 also uses a row-major approach to store two-dimensional data, which can be represented by (x, y) (where the X-axis is horizontal to the right and the Y-axis is vertical downwards). The dimension along the X-axis (the dimension of each row, or the total number of columns) is ori_x (not shown in the figure), and the dimension along the Y-axis (the total number of rows) is ori_y (not shown in the figure). Unlike Figure 4 tensor data, Figure 5 The tensor data stored in it consists of multiple data blocks.

[0084] In this case, the descriptor needs more parameters to represent these data tiles. Taking the X-axis (X dimension) as an example, it can involve parameters such as: ori_x, x.tile.size (size of the tile, 502), x.tile.stride (stride of the tile, 504, i.e., the distance between the first point of the first tile and the first point of the second tile), x.tile.num (number of tiles, shown as 3 tiles in the figure), x.stride (overall stride, i.e., the distance between the first point of the first row and the first point of the second row), etc. Other dimensions can similarly include corresponding parameters.

[0085] In one possible implementation, the descriptor may include an identifier and / or content. The identifier is used to distinguish descriptors; for example, the identifier may be a number. The content may include at least one shape parameter representing the shape of the tensor data. For example, if the tensor data is 3-dimensional, and the shape parameters of two of its three dimensions are fixed, the content of its descriptor may include the shape parameter representing the other dimension of the tensor data.

[0086] In one possible implementation, the identifier and / or content of the descriptor can be stored in descriptor storage space (internal memory), such as registers, on-chip SRAM, or other media caches. The tensor data indicated by the descriptor can be stored in data storage space (internal or external memory), such as on-chip cache or under-chip memory. This disclosure does not limit the specific location of the descriptor storage space or the data storage space.

[0087] In one possible implementation, the descriptor's identifier, content, and the tensor data indicated by the descriptor can be stored in the same area of ​​internal memory. For example, a contiguous area of ​​on-chip cache, with addresses ADDR0-ADDR1023, can be used to store the descriptor's content. Addresses ADDR0-ADDR63 can be used as the descriptor storage space to store the descriptor's identifier and content, while addresses ADDR64-ADDR1023 can be used as the data storage space to store the tensor data indicated by the descriptor. Within the descriptor storage space, addresses ADDR0-ADDR31 can be used to store the descriptor's identifier, and addresses ADDR32-ADDR63 can be used to store the descriptor's content. It should be understood that address ADDR is not limited to one bit or one byte; it is used here to represent an address, a unit of address. Those skilled in the art can determine the descriptor storage space, data storage space, and their specific addresses based on actual circumstances, and this disclosure does not impose any limitations in this regard.

[0088] In one possible implementation, the descriptor's identifier, content, and the tensor data it points to can be stored in different areas of internal memory. For example, registers can be used as descriptor storage space, storing the descriptor's identifier and content, while on-chip caches can be used as data storage space, storing the tensor data pointed to by the descriptor.

[0089] In one possible implementation, when using registers to store the identifier and content of descriptors, the register number can be used to represent the descriptor's identifier. For example, when the register number is 0, the identifier of the descriptor it stores is set to 0. When the descriptor in the register is valid, a region in the cache space can be allocated to store the tensor data according to the size of the tensor data indicated by the descriptor.

[0090] In one possible implementation, the identifier and content of the descriptor can be stored in internal memory, while the tensor data pointed to by the descriptor can be stored in external memory. For example, the identifier and content of the descriptor can be stored on-chip, and the tensor data pointed to by the descriptor can be stored off-chip.

[0091] In one possible implementation, the data address of the data storage space corresponding to each descriptor can be a fixed address. For example, a separate data storage space can be allocated for tensor data, and the starting address of each tensor data in the data storage space corresponds one-to-one with a descriptor. In this case, the circuit or module responsible for parsing computation instructions (e.g., an entity outside the computing device disclosed herein) can determine the data address of the data corresponding to the operand in the data storage space based on the descriptor.

[0092] In one possible implementation, when the data address of the data storage space corresponding to the descriptor is a variable address, the descriptor can also be used to indicate the address of N-dimensional tensor data. The content of the descriptor may further include at least one address parameter representing the address of the tensor data. For example, if the tensor data is 3-dimensional, when the descriptor points to the address of the tensor data, the content of the descriptor may include one address parameter representing the address of the tensor data, such as the starting physical address of the tensor data, or it may include multiple address parameters representing the address of the tensor data, such as the starting address of the tensor data plus an address offset, or address parameters based on each dimension of the tensor data. Those skilled in the art can set the address parameters according to actual needs, and this disclosure does not impose any limitations on this.

[0093] In one possible implementation, the address parameter of the tensor data may include the reference address of the data reference point of the descriptor within the data storage space of the tensor data. The reference address may vary depending on the data reference point. This disclosure does not impose any restrictions on the selection of the data reference point.

[0094] In one possible implementation, the base address may include the starting address of the data storage space. When the data base point of the descriptor is the first data block in the data storage space, the base address of the descriptor is the starting address of the data storage space. When the data base point of the descriptor is data other than the first data block in the data storage space, the base address of the descriptor is the address of that data block in the data storage space.

[0095] In one possible implementation, the shape parameters of the tensor data include at least one of the following: the size of the data storage space in at least one of the N-dimensional directions, the size of the storage region in at least one of the N-dimensional directions, the offset of the storage region in at least one of the N-dimensional directions, the positions of at least two vertices at diagonal positions in the N-dimensional directions relative to the data reference point, and the mapping relationship between the data description location of the tensor data indicated by the descriptor and the data address. Here, the data description location is the mapped position of a point or region in the tensor data indicated by the descriptor. For example, when the tensor data is 3D data, the descriptor can use three-dimensional spatial coordinates (x, y, z) to represent the shape of the tensor data, and the data description location of the tensor data can be the position of a point or region mapped to the tensor data in three-dimensional space, represented by three-dimensional spatial coordinates (x, y, z).

[0096] It should be understood that those skilled in the art can choose the shape parameters representing tensor data according to the actual situation, and this disclosure does not impose any restrictions on this. By using descriptors during data access, associations between data can be established, thereby reducing the complexity of data access and improving instruction processing efficiency.

[0097] Based on the aforementioned hardware environment, the embodiments disclosed herein provide a data processing apparatus that performs data fusion-related operations with tensor data according to specialized fusion instructions. As mentioned in the background art, network parameter sparsification can effectively reduce the network's computational and storage requirements. However, network parameter sparsification also brings a series of impacts to subsequent processing. For example, in sparse matrix multiplication, it may be necessary to sort and accumulate the vectors obtained in the middle of the operation to obtain the expected operation result. As another example, in radar algorithms, it is necessary to fuse the sparsified data in radar-based object detection. Therefore, the embodiments disclosed herein provide a hardware solution for data fusion processing to simplify and accelerate such processing.

[0098] Figure 6 This diagram illustrates an exemplary principle of data fusion processing according to embodiments of this disclosure. The diagram exemplarily shows four streams of data to be fused, each stream comprising six data elements. Data elements can be scalars, vectors, or higher-dimensional tensors. Data elements are exemplarily shown as vectors, such as D11, D12, ..., D46. These vectors have a uniform length; for example, D11 is (d1, d2, d3, ..., dn) with a length of n. Each data element has an associated index indicating its position within the corresponding stream of data. For example, the original stream of data may contain 1000 data elements, but only some of these elements are valid. In this case, these valid elements can be extracted to form the aforementioned data to be fused, and their corresponding indices can be extracted to indicate their positions in the original data; these indices form the aforementioned fusion index.

[0099] The diagram schematically illustrates the four indices to be merged, with each index corresponding to one channel of data to be merged. The first index identifies the position of each data element in the first channel, the second index identifies the position of each data element in the second channel, and so on. Furthermore, the index elements in each channel are stored in an ordered manner and correspond one-to-one with the data elements in the corresponding channel. In the example shown, the index elements in each channel are arranged in a first order (e.g., ascending order), and the data elements in each channel are also arranged in the order of their corresponding indices. For example, the first index element in the first channel indicates that the index of the first data element in the first channel is 0, i.e., the first element; the second index element in the first channel indicates that the index of the second data element in the first channel is 2, i.e., the third element; and so on.

[0100] After data fusion, the four data streams are merged into a single ordered fused data stream according to their corresponding indices, with data elements having the same index being combined into a single fused data element. As shown in the figure, the fused index comprises 16 index elements, arranged in a second order (e.g., ascending order), with duplicate index elements removed from the indices to be fused, as indicated by the dark squares in the figure. Correspondingly, the fused data also comprises 16 data elements, arranged in the order of their corresponding indices, with data elements having the same index being combined, as indicated by the dark squares in the figure. Since data elements may be vectors or higher-dimensional tensors, in some embodiments disclosed herein, at least for the merging of data elements with the same index, the form of operational structure elements can be used.

[0101] In this example, the merging of data elements with the same index is illustrated using an addition formula. For example, for the merged index element "0", its corresponding merged data element is the sum of the first data elements of each channel (D11+D21+D31+D41). As another example, for the merged index element "9", its corresponding merged data element is the sum of the 5th data element of channel 1 and the 3rd data element of channel 4 (D15+D43). When the data elements are vectors, the merged data element is the sum of the corresponding vectors. The specific representation of the merged data element will be described later.

[0102] Those skilled in the art will understand that the first order and the second order mentioned above may be the same or different, and both may be selected from either: an ascending order or a descending order. Those skilled in the art will also understand that although the data streams shown in the diagram have an equal number of data elements, the number of data elements in each data stream may be the same or different, and this disclosure is not limited in this respect.

[0103] Figure 7 A structural block diagram of a data processing apparatus 700 according to an embodiment of this disclosure is shown. The data processing apparatus 700 can, for example, be implemented in... Figure 2 In the computing device 201. As shown in the figure, the data processing device 700 may include a control circuit 710, a storage circuit 720, an arithmetic circuit 730, and a tensor interface circuit 712.

[0104] The function of the control circuit 710 can be similar to Figure 3 The control module 31 may include, for example, an instruction fetching unit for fetching instructions from, for example... Figure 2 The processing device 203 has instructions and an instruction decoding unit for decoding the acquired instructions and sending the decoding results as control information to the arithmetic circuit 730 and the storage circuit 720.

[0105] In one embodiment, the control circuit 710 may be configured to parse fusion instructions, wherein the fusion instructions instruct the fusion processing to be performed on multiple streams of data to be fused.

[0106] The storage circuit 720 can be configured to store various types of information, including at least information before and / or after the fusion process. The storage circuit could be, for example, […]. Figure 3 WRAM 332.

[0107] The arithmetic circuit 730 can be configured to perform corresponding operations according to fusion instructions. Specifically, the arithmetic circuit 730 can group data elements from multiple data streams to be fused into a single ordered fused data stream according to their corresponding indices, wherein data elements with at least the same index are merged into an arithmetic structure element. The data elements can be any of scalars, vectors, or higher-dimensional data.

[0108] In one embodiment, the arithmetic circuit 730 may further include an arithmetic processing circuit (not shown), which can be configured to preprocess the data before the arithmetic circuit performs the operation or postprocess the data after the operation according to the arithmetic instructions. In some application scenarios, the aforementioned preprocessing and postprocessing may include, for example, data splitting and / or data concatenation operations.

[0109] The Tensor Interface Unit (TIU) 712 can be configured to perform operations associated with descriptors under the control of the control circuit 710. These operations may include, but are not limited to, registering, modifying, deregistering, and resolving descriptors; reading and writing descriptor contents, etc. This disclosure does not limit the specific hardware type of the Tensor Interface Unit. In this way, operations associated with descriptors can be implemented through dedicated hardware, further improving the efficiency of tensor data access.

[0110] In some embodiments, the tensor interface circuit 712 may be configured to parse the shape information of the tensor data included in the operand of the instruction in order to determine the data address of the data corresponding to the operand in the data storage space.

[0111] Optionally or additionally, in some other embodiments, the tensor interface circuit 712 may be configured to compare the spatial information (e.g., spatial ID) of the tensor data included in the operands of two instructions to determine the dependency relationship between the two instructions, and thereby determine out-of-order execution, synchronization, or other operations of the instructions.

[0112] Despite Figure 7 The control circuit 710 and the tensor interface circuit 712 are shown as two separate modules, but those skilled in the art will understand that these two circuits can also be implemented as one or more modules, and this disclosure is not limited in this respect.

[0113] There are many ways to implement arithmetic circuits. Figure 8 An exemplary circuit diagram for data fusion processing according to one embodiment of this disclosure is shown.

[0114] As shown in the figure, in one embodiment, the storage circuit can be exemplarily divided into two parts: a first storage circuit 822 and a second storage circuit 824.

[0115] The first storage circuit 822 can be configured to store K-channel data to be fused and K-channel indices corresponding to these K-channel data, where K>1. The index elements in these K-channel indices indicate the index information of the corresponding data elements in the K-channel data, meaning there is a one-to-one correspondence between index elements and data elements. Furthermore, the index elements of each of these K-channel indices are arranged in a first ordered order, and the data elements of each of the K-channel data are arranged in the order of their corresponding indices.

[0116] The figure illustrates an example. Figure 6 The diagram shows four indices and their corresponding four data paths. Since data elements may be vectors or higher-dimensional data, for simplicity, their storage addresses can be used instead. Each data element in the diagram is identified by the symbol Pt, representing a pointer to the storage address of the corresponding data element (e.g., a vector, a 3D tensor, etc.). It is understood that the actual data elements are also stored in the storage circuitry, but for clarity, they are not shown in the diagram. In some embodiments, each index can be stored contiguously, for example, as a vector, so that the index / index vector can be accessed based on the starting address of each index or the starting address of the vector. Correspondingly, each data path can also be stored contiguously, for example, as a vector, so that the data / data vector can be accessed based on the starting address of each data path or the starting address of the vector. However, in this case, the vector elements in the data vector are pointers to the final data elements.

[0117] The second storage circuit 824 can be configured to store the fused data output by the arithmetic circuit. This data may include: a fused index after sorting and fusion of K-way indices, and fused data. The fused index elements in the fused index are arranged in a second order, and the fused data elements in the fused data correspond one-to-one with the fused index elements and are arranged in the order of the fused index. Each fused data element is associated with one or more arithmetic structure elements.

[0118] As shown in the example in the diagram, the four data paths to be merged are transformed into one merged data path, and the corresponding four indices are also transformed into one merged index. The merged index elements are arranged in ascending order, and index elements of the same size are removed. The corresponding merged data elements are arranged in the order of their merged indices. Each merged data element can be an address pointer pointing to the corresponding final data element. In some embodiments, the storage space for each final data element can be of a fixed size. Therefore, when these data elements are stored contiguously, the subsequent addresses can be determined by a fixed offset from the starting address. For example, the diagram exemplifies that the first merged data element can be the address `base_addr`. Assuming that the address size occupied by each final data element is `offset`, then the second merged data element can be the address `base_addr + offset`, the third merged data element can be the address `base_addr + 2 * offset`, and so on.

[0119] Among the final data elements pointed to by these addresses, some are original data elements because their indices are unique within the fusion index and do not require accumulation; others, however, require accumulation because multiple original data elements with the same index exist. In the embodiments disclosed herein, at least for fused data elements composed of data elements with the same index, the final data element they point to can be obtained without immediate computation, but rather represented by an associated operational structure element. The operational structure element will be described in detail below with reference to specific circuits.

[0120] In some embodiments, the arithmetic circuit may include a sorting circuit 832 and an output circuit 836 to collaboratively implement the sorting, accumulation, and fusion function. Specifically, the sorting circuit 832 is used to sort the K-way indices according to the size of the index elements and output them in an orderly manner to the output circuit 836. The output circuit 836 can generate an arithmetic structure element representing the accumulation operation of the data elements corresponding to the same index element, at least when receiving the same index element from the sorting circuit, and remove duplicate index elements.

[0121] In some embodiments, the sorting circuit 832 may include a comparison circuit 831 and a buffer circuit 833. The comparison circuit 831 performs a comparison function, comparing the sizes of index elements in multiple indices to be merged and submitting the comparison results to the control circuit 810 for sorting. The control circuit 810 determines the insertion position of the index element in the buffer circuit 833 based on the comparison results. The buffer circuit 833 is used to cache the compared index elements and their corresponding data elements, caching them in order of index element size.

[0122] Specifically, the comparison circuit 831 can be configured to compare the index elements in the index to be merged with the index elements not yet output in the buffer circuit 833, and output the comparison result to the control circuit 810. The buffer circuit 833 can be configured to, according to the control of the control circuit 810, orderly store the information of the compared index elements and their corresponding data elements, and orderly output the information of the compared index elements and their corresponding data elements.

[0123] In some embodiments, buffer circuit 833 can be configured to cache K index elements, which are sorted by size. Those skilled in the art will understand that the buffer circuit can also be configured to cache more index elements, and the embodiments disclosed herein are not limited in this respect. Depending on the sorting method in buffer circuit 833 and the desired output sorting method, such as ascending or descending, the first or last index element in the current sequence can be output in a specified order each time. For example, in the example in the figure, buffer circuit 833 caches index elements from left to right in descending order, outputting the rightmost index element each time, which is the smallest index element in the current sequence, such as "7".

[0124] In these embodiments, the comparison circuit 831 may include a K-1 comparator configured to compare the index element to be fused with the index elements not yet output in the buffer circuit 833, that is, to compare with the K-1 index elements remaining after the first or last index element of the current sequence is output, generate a comparison result and output it to the control circuit 810.

[0125] For example, for four data streams to be merged, a three-way comparator is shown in the figure, which compares the specified index element (9 in this case) received from the first storage circuit 822 with the three index elements that are not currently output from the buffer circuit 833. The three index elements on the left in the figure are 100, 10 and 9.

[0126] In some embodiments, the comparison result of the comparator can be represented using a bitmap. For example, if the index element to be merged (e.g., 9) is greater than or equal to the index element in the buffer circuit, the comparator can output "1"; otherwise, it outputs "0"; and vice versa. In the example in the figure, the comparison result of the index element to be merged (9) with the respective index elements (100, 10, and 9) in the buffer circuit is "001", which is output to the control circuit 810.

[0127] The control circuit 810 can be configured to determine the insertion position of the index element to be merged in the current sequence of the buffer circuit 833 based on the received comparison result. Specifically, the control circuit 810 can be further configured to determine the insertion position based on the change position of the bits in the bitmap. In the example shown in the figure, the comparison result is "001", indicating that the index element to be merged is less than the first and second index elements from the left in the buffer circuit, and greater than or equal to the third index element from the left. Therefore, the insertion position is between the second and third index elements, that is, between "10" and "9".

[0128] In some embodiments, the buffer circuit 833 may be configured to insert the index element to be merged at the insertion position according to the instruction of the control circuit 810. In the example shown in the figure, the sequence after the index element is inserted in the buffer circuit 833 becomes "100,10,9,9".

[0129] To enable the retrieval of data corresponding to the index during the fusion process, in some embodiments, the buffer circuit 833 can be further configured to: orderly store the compared index elements and their corresponding data elements according to the value order of the index elements. As shown in the figure, the buffer circuit 833 caches not only the index elements but also the information of their corresponding data elements. Therefore, after each comparison of the index elements to determine the insertion position, the information of the data element corresponding to that index element can also be inserted into the buffer circuit. Since the data element can be a scalar, vector, or higher-dimensional tensor, an address pointing to the data element can be used to represent the data element. For example, in the example in the figure, each element in the K-way data to be fused is an address, pointing to the corresponding data element, regardless of whether the data element is a scalar, vector, or high-dimensional tensor. In the description herein, data element may refer to an address or the final scalar, vector, or high-dimensional tensor data; those skilled in the art can distinguish its meaning based on the context of the description.

[0130] Next, the buffer circuit 833 can output the rightmost index element "9". At this time, the control circuit 810 can be further configured to determine the memory access information of the next index element to be merged based on the index element output from the buffer circuit. Specifically, the control circuit retrieves the next index element to be merged from which index in the K-way index the output index element belongs, and sends it to the comparison circuit 831 for comparison.

[0131] Furthermore, during ordered output, the buffer circuit 833 can be configured to output the compared index elements in order of their values ​​(e.g., from smallest to largest) as fusion indices, and simultaneously output their corresponding data elements as fusion data. The output data is provided to the output circuit 836 for further processing.

[0132] For clarity, the diagram also shows the index sequence cached in buffer circuit 833 as the sorting progresses. As shown, initially, the first index element of each of the K-way indices is stored in buffer circuit 833 in descending order. In some implementations, these four index elements can be retrieved, sorted, and stored in the buffer circuit all at once. In other implementations, the data in the buffer circuit can be initialized to negative numbers, and the first index element of each indices can be retrieved sequentially (e.g., from indices 1 to 4), compared with the data in the buffer circuit, and placed in the appropriate position. In this example, the first index element of all four indices is 0, so they can be arranged according to the index number based on the order of retrieval; for example, the "0" of indices 1 is placed on the far right, the "0" of indices 2 is placed in the second position from the right, and so on.

[0133] Next, the rightmost "0" belonging to the first channel in the buffer circuit is output. Based on which channel this output index element belongs to, the next index element to be merged is retrieved from that channel, namely the second index element "2" of the first channel. "2" is sent to the comparison circuit and compared with the remaining three "0"s in the buffer circuit. The comparison result is "111", which is greater than all three existing "0"s in the buffer circuit. Therefore, "2" is inserted at the end of the sequence, and the sequence in the buffer circuit becomes "2,0,0,0".

[0134] Next, the rightmost "0" belonging to the second path in the buffer circuit is output. Therefore, the second element "3" of the second path is taken out and compared with the remaining "2,0,0" in the buffer circuit. The comparison result is "111", so "3" is inserted at the end of the sequence. At this time, the sequence in the buffer circuit becomes "3,2,0,0".

[0135] Next, the rightmost "0" belonging to the third path in the buffer circuit is output. Therefore, the second element of the third path, "100", is taken out and compared with the remaining "2,0,0" in the buffer circuit. The comparison result is "111", so "100" is inserted at the end of the sequence. At this time, the sequence in the buffer circuit becomes "100,3,2,0".

[0136] Next, the rightmost "0" belonging to the 4th channel of the output buffer circuit is taken out and the second element "2" of the 4th channel is compared with the remaining "100,3,2" in the buffer circuit. The comparison result is "001", so "2" is inserted after the rightmost first element of the sequence. At this time, the sequence in the buffer circuit becomes "100,3,2,2".

[0137] Similarly, the index elements in the K-way index can be compared one by one, sorted according to size, and inserted into the appropriate positions in the buffer circuit before being output by the buffer circuit. For example, the smallest index element output by the buffer circuit each time can be output to the output circuit 836 in sequence. Those skilled in the art will understand that if the buffer circuit has sufficient space, the merged and sorted elements can also be output uniformly after the sorting is completed.

[0138] As can be seen from the merged sorted index elements, when there are index elements of the same size, the sorting circuit 832 still retains these index elements of the same size and does not perform deduplication. Instead, it provides them to the output circuit 836 for processing.

[0139] In some embodiments, the output circuit 836 may include a comparator 837, a buffer 835, and a structure generator 839.

[0140] Comparator 837 can be configured to compare the index elements output in order from sorting circuit 832 with the previous merged index element and output the comparison result. The comparison result can be "1" indicating that they are the same, "0" indicating that they are different, or vice versa.

[0141] Buffer 835 can be configured to control the output index element based on the comparison result of comparator 837. In some embodiments, buffer 835 can output the current index element as a new fused index element only when the comparison results indicate that they are different. In other words, when the comparison results indicate that they are the same, buffer 835 does not output the current index element, that is, it discards the index element that is the same as the previous fused index element. As shown, there are no duplicate fused index elements in the fused index of the second storage circuit 824.

[0142] The structure generator 839 can be configured to control the accumulation of data elements and generate corresponding operational structure elements based on the comparison result of the comparator 837. Specifically, at least when the comparison results indicate the same result, an operational structure element is generated based on the data element corresponding to the index element. This operational structure element represents the accumulation operation of adding the current data element to the fused data element corresponding to the previous fused index element.

[0143] Figure 8The embodiment illustrates a method for generating operation structure elements. In this embodiment, operation structure elements are generated not only for fused data elements formed by fusing data elements with the same index, but also for other fused data elements. That is, each fused data element is represented using an operation structure element. Specifically, in this embodiment, when the comparison results of comparator 837 are different, an operation structure element is generated based on the data element corresponding to the current index element. This operation structure element represents adding the data element corresponding to the current index element to the data at a specified address, where the data at the specified address can be pre-set to a value of 0. In this way, operation structure elements can be generated for all fused data elements, thereby unifying the expression method, simplifying operations, and providing flexible processing for subsequent operations.

[0144] Figure 8 The diagram illustrates the operational structure generated in the current implementation. As shown, when the output circuit 836 receives the first "0" output by the sorting circuit 832, since this is the first element, the comparator 837 outputs, for example, "0" to indicate that it is different from the previous fused index element (e.g., initialized to a negative number). The buffer 835 outputs index "0" as the first fused index element. Based on the result of the comparator 837, the structure generator 839 allocates an address for the new fused data element, for example, the address of the first fused data element is base_addr. At this time, even if the indices are considered different, an associated operational structure element is generated for the fused data element, which is simply an accumulation operation of adding the current data element to 0. In one embodiment, the arithmetic structure element can indicate an in-situ addition operation, which includes addresses pointing to two addends, such as {base_addr, Pt11}, where the data at the address pointed to by base_addr is pre-set to 0, and Pt11 represents the address of the current data element. The result of the addition is still stored at the address pointed to by base_addr, which corresponds to the address of the first merged data element allocated above. The generated arithmetic structure element can be stored in the second storage circuit 824.

[0145] As mentioned above, each operation structure element can include two elements, which indicate the addresses of the two addends respectively, such as {src0_addr, src1_addr}, where src0_addr represents the address of the first addend and src1_addr represents the address of the second addend, and the addition result is stored in the address pointed to by src0_addr, that is, the in-place addition operation is performed.

[0146] Next, the sorting circuit 832 outputs the second "0". Since the previous fusion index element was "0", the comparator 837 will output, for example, "1" to indicate that it is the same as the previous fusion index element ("0"). At this time, the buffer 835 does not output. Based on the result of the comparator 837, the structure generator 839 considers that no new fusion data element has been generated, so it does not allocate an address, that is, there is no new element in the fusion data. At this time, the structure generator 839 will also generate an operation structure element representing the accumulation of data elements, which accumulates the data element into the fusion data element corresponding to the previous fusion index element, for example, {base_addr, Pt21}, where base_addr represents the address of the fusion data element corresponding to the previous fusion index element, that is, the address of the first fusion data element allocated in the previous step, and Pt21 represents the address of the current data element. The addition result is still stored in base_addr, that is, the data element is accumulated into the first fusion data element allocated above. The generated operation structure element is also stored in the second storage circuit 824.

[0147] When the sorting circuit 832 continues to output the 3rd and 4th "0", the fusion index will not increase, and the fusion data elements in the corresponding fusion data will not increase. The structure generator will output the associated structure operation elements: {base_addr,Pt31}, {base_addr,Pt41}.

[0148] Next, the sorting circuit 832 outputs the first "2". Referring to the previous steps, the buffer 835 outputs index "2" as the second fused index element. The structure generator 839, based on the result of the comparator 837, allocates an address for the new fused data element, for example, by adding an offset to the address of the first fused data element. This offset is the storage address size of the corresponding final data element (e.g., a scalar, vector, or high-dimensional tensor). At this time, an associated operational structure element is generated for this new fused data element, such as {base_addr+offset,Pt12}. Similarly, the data at the address pointed to by base_addr+offset is pre-set to 0. The generated operational structure element is stored in the second storage circuit 824.

[0149] When the sorting circuit 832 outputs the second "2", the fusion index will not increase, and the fusion data elements in the corresponding fusion data will not increase. The structure generator will output the associated structure operation element: {base_addr+offset,Pt42}.

[0150] Based on the above description, those skilled in the art can similarly deduce other fusion results, which will not be described one by one here.

[0151] Those skilled in the art will understand that other forms of hardware circuits can be designed to implement the above-mentioned merge sorting and fusion process, and this disclosure is not limited in this respect.

[0152] In this disclosed embodiment, the data merging and sorting fusion process can be implemented using the aforementioned exemplary hardware circuit by invoking a fusion instruction. The fusion instruction operates on the following objects: the input K-way data to be merged, the K-way indices corresponding to the K-way data, the size of the K-way data, and the output fusion index and fusion data, where K > 1. Among these objects, the index elements in the K-way index indicate the index information of the corresponding data elements in the K-way data; the index elements of each index in the K-way index are arranged in a first order; the data elements of each data path in the K-way data are arranged in the order of their corresponding indices; the fusion index elements in the fusion index are arranged in a second order, wherein identical index elements are merged into the same fusion index element; the fusion data elements in the fusion data correspond one-to-one with the fusion index elements, and the fusion data element composed of data elements with the same index is represented by an operation structure element representing the accumulation operation of the corresponding data elements.

[0153] In some embodiments, the operation object of the fusion instruction may also include the total number of output fusion index elements, used to indicate the number of index elements in one output fusion index. It can be understood that, since there is a one-to-one correspondence between fused data and fusion indexes, this operation object is also simply the number of data elements in one output fusion data stream.

[0154] In some embodiments, the operation object of the fusion instruction may also include the total number of elements of the output operation structure.

[0155] As mentioned above, the first order and the second order can be the same or different, and the first order and the second order can be selected from either the following: an order from smallest to largest, or an order from largest to smallest.

[0156] As mentioned earlier, in fusion instructions involving tensor data, at least one operand includes at least one descriptor to indicate the shape and / or spatial information of the tensor data.

[0157] Figure 9 The instructions of various descriptors in fusion instructions according to some embodiments of this disclosure are illustrated by way of example.

[0158] As shown in the figure, the size offset includes the offset of the K-way input data and the offset of the K-way input indices. This means the size of the K-way input data is preceded by `size_offset` units of data; that is, the number of data items in the K-way input data offset + the number of data items in the K-way input index offset = `size_offset`. Accessing the i-th element of the "size of the K-way input data" can be written as `size_offset + i`. The fusion instruction can include the first input descriptor `TID0`, which indicates the address offset of the K-way input data to be fused, the address offset of the K-way input indices, and the size of the K-way input data. It can be understood that each data item is stored contiguously, but the size of each data item can be different, i.e., the length can be different. Therefore, by identifying the K-way data using K address offsets, the base address of each data item can be quickly determined, such as the starting address of each data item. Similarly, by identifying the K-way indices using K address offsets, the base address of each index can be quickly determined, such as the starting address. The size of the K-way input data consists of K elements, where the i-th element size[i] represents the number of data elements in the i-th way data, and 0 < i ≤ K. Since there is a one-to-one correspondence between the K-way data and the K-way indices, size[i] also represents the number of index elements in the i-th way index.

[0159] Optionally or additionally, the fusion instruction may also include a second descriptor TID1 for indicating the data of all input data. More specifically, TID1 may indicate the base point address of the K-channel data, such as the starting address tid1.base_addr of the first channel data. This base point address and the address offset of the K-channel data indicated by the first descriptor TID0 can be combined to access the K-channel data. For example, the starting address tid1.base_addr of the first channel data + the offset data2_offset of the second channel data can determine the starting address of the second channel data. Based on the starting addresses of each channel data, the address of each data element can be calculated. For example, the address of the i-th data element in the first channel data can be calculated as: tid1.base_addr + i * co * input_type, where co represents the length of a single data element (the length of each data element is fixed), and input_type represents the data type of the input data. Thus, by combining TID1 and TID0, the numerical information of each data element of each channel of all K-channel input data can be obtained.

[0160] Optionally or additionally, the fusion instruction may also include a third descriptor TID2 for indicating the data of all input indices. More specifically, TID2 may indicate the base address of the K-way index, such as the starting address tid2.base_addr of the first index, whereby this base address and the address offset of the K-way index indicated by the first descriptor TID0 can be combined to access the K-way index. For example, the starting address tid2.base_addr of the first index plus the offset index2_offset of the second index can determine the starting address of the second index. Based on the starting addresses of each index, the address of each index element can be calculated. For example, the address of the i-th index element in the first index can be calculated as: tid2.base_addr + i * index_type, where index_type represents the data type of the input index. Thus, by combining TID2 and TID0, the numerical information of each index element of each index of each of all K-way input indices can be obtained.

[0161] Optionally or additionally, the fusion instruction may also include a fourth descriptor TID4, indicating the storage address of the final result of the operation structure element. As described above, each fused data element is represented by one or more operation structure elements, which represent in-situ addition operations. After these operations, the final result can be stored at a specified address, i.e., the address indicated by the fourth descriptor TID4. For example, TID4 may indicate the starting address of the final result, tid4.base_addr. In some embodiments, each final result is a fixed-length vector, with a length of, for example, co, the same as the length of the input data element. Therefore, the address of the fused data element obtained from the i-th operation can be calculated as: tid4.base_addr + i * co * output_type, where output_type represents the data type of the output data.

[0162] Optionally or additionally, the merging instruction may also include a fifth descriptor TID3 of the output, indicating the merging index and the elements of the operation structure. In some embodiments, the index elements in the input K-way index are arranged in ascending order, for example, and the final output merging index elements may also be arranged in ascending order. In the merge sort merging process disclosed herein, duplicate indexes are removed when they exist.

[0163] Each fused data element can be associated with one or more operation structure elements. For example, when a fused data element consists of multiple input data with the same index, multiple operation structure elements are generated to accumulate these input data one by one onto the same fused data element. When a fused data element consists of a single input data, only one operation structure element is generated, representing the addition operation between that input data and the 0-valued data at the specified address.

[0164] Each operand structure element includes two sub-elements, each being an address representing the addresses of the two addends, one of which also serves as the address for the sum of the results. In some embodiments, an operand structure element may include two 64-bit elements representing the two addresses.

[0165] There are various operations related to data fusion, such as merge sort, sort accumulation, and sort merging. Multiple instruction schemes can be designed to implement these operations.

[0166] In one approach, a fusion instruction can be designed, which may include an operation mode bit to indicate different operation modes of the fusion instruction, thereby performing different operations.

[0167] In another approach, multiple fusion instructions can be designed, each corresponding to one or more different operating modes to perform different operations. In one implementation, a corresponding fusion instruction can be designed for each operating mode. In another implementation, operating modes can be categorized based on their characteristics, and a fusion instruction can be designed for each category of operating modes. Furthermore, when a certain category of operating modes includes multiple operating modes, an operating mode bit can be included in the fusion instruction to indicate the corresponding operating mode.

[0168] Regardless of the approach taken, fusion instructions can indicate their corresponding operating mode through the operation mode bit and / or the instruction itself.

[0169] In the context of this disclosure, the aforementioned fusion instruction may be a microinstruction or control signal that runs within one or more multi-stage computational pipelines, and may include (or instruct) one or more computational operations that need to be executed by the multi-stage computational pipelines. Depending on the different computational operation scenarios, the computational operations may include, but are not limited to, arithmetic operations such as convolution and matrix multiplication, logical operations such as AND, XOR, and OR, shift operations, or any combination of the aforementioned types of computational operations.

[0170] Figure 10 An exemplary flowchart of a data processing method 1000 according to an embodiment of this disclosure is shown.

[0171] like Figure 10As shown, in step 1010, a fusion instruction is parsed, which instructs the fusion processing to be performed on multiple streams of data to be fused, and at least one operand of the fusion instruction includes at least one descriptor, which indicates at least one of the following: shape information and spatial information of the tensor data. This step can be performed, for example, by... Figure 7 The control circuit 710 is used to execute this.

[0172] Next, in step 1020, the descriptor is resolved. This step can be performed, for example, by... Figure 7 The tensor interface circuit 712 is used for execution. Specifically, the data address of the tensor data corresponding to the operand in the data storage space can be determined based on the shape information of the tensor data; and / or the dependencies between instructions can be determined based on the spatial information of the tensor data.

[0173] Next, in step 1030, based on the parsed descriptor and according to the fusion instruction, the data elements in the multiple data to be fused are grouped into one ordered fused data according to their corresponding indices. Each fused data element is represented by an operational structure element, and the data element may include any of the following: scalar, vector, or higher-dimensional data.

[0174] Finally, in step 1040, the fused data is output. Steps 1030 and 1040 can, for example, be derived by... Figure 7 The operation is performed by the 730 arithmetic circuit.

[0175] Those skilled in the art will understand that each step of the above method corresponds to the respective circuits described above in conjunction with the example circuit diagrams. Therefore, the features described above can be applied equally to the method steps, and will not be repeated here.

[0176] As described above, this disclosure provides a fusion instruction for performing fusion processing of multiple streams of data to be fused. In some embodiments, the fusion instruction is a hardware instruction, implemented through dedicated hardware circuitry to accelerate processing and better support operations related to the sparsed processing, such as those in radar algorithms. In some embodiments, the fusion instruction can merge multiple ordered streams of data into a single ordered stream of fused data, and data with the same index can be merged, represented in the form of an operation structure, thus facilitating subsequent computation. In some embodiments, the fusion instruction may include an operation mode bit to indicate that the fusion instruction is a merge sort fusion processing operation, or the fusion instruction itself may indicate a merge sort fusion processing operation. By providing a dedicated fusion instruction to perform operations related to the fusion processing of multiple streams of data, processing can be simplified.

[0177] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0178] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0179] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0180] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0181] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0182] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A data processing apparatus, comprising: A control circuit configured to parse a fusion instruction that instructs the performance of a fusion process on multiple streams of data to be fused, wherein at least one operand of the fusion instruction includes at least one descriptor that indicates at least one of the following: shape information of the tensor data and spatial information of the tensor data; Tensor interface circuitry configured to parse the descriptor; Storage circuitry configured to store information before and / or after fusion processing; as well as The arithmetic circuit is configured to, based on the parsed descriptor, merge data elements from the multiple streams of data to be merged into one ordered fused data stream according to their corresponding indices, according to the fusion instruction, wherein each fused data element is represented by an arithmetic structure element, and the data element includes any of scalar, vector, or higher-dimensional data. The fusion instruction operates on K-channel data to be fused, K-channel indices corresponding to the K-channel data, the dimensions of the K-channel data, and one fusion index and one fusion data output, where K > 1. The index elements in the K-way index indicate the index information of the corresponding data elements in the K-way data; In the K-way index, the index elements of each way index are arranged in a first order; The data elements of each data path in the K-path data are arranged in order according to the corresponding index. The fused index elements in the fused index are arranged in a second order, wherein identical index elements are merged into the same fused index element; and The fused data elements in the fused data correspond one-to-one with the fused index elements, and the fused data elements composed of data elements with the same index are characterized by the operation structure elements representing the accumulation operation of the data elements.

2. The data processing apparatus according to claim 1, wherein the first order is the same as or different from the second order, and the first order and the second order are selected from either: an order from smallest to largest, or an order from largest to smallest.

3. The data processing apparatus according to claim 1, wherein the arithmetic circuit includes a sorting circuit and an output circuit, wherein: The sorting circuit is configured to sort the K-way indexes according to the size of the index elements and output them in an orderly manner to the output circuit; and The output circuit is configured to, in response to receiving an index element identical to the previous fused index element from the sorting circuit, generate an operation structure element representing an accumulation operation that adds the data element corresponding to the current index element to the fused data element corresponding to the previous fused index element, and remove duplicate index elements.

4. The data processing apparatus according to claim 3, wherein the output circuit is further configured to: In response to receiving an index element different from the previous fusion index element from the sorting circuit, the current index element is used as the new fusion index element, and an operation structure element is generated to represent adding the data element corresponding to the current index element to the data at a specified address, wherein the data at the specified address is 0.

5. The data processing apparatus according to any one of claims 1-4, wherein each arithmetic structure element indicates an in-situ addition operation, which includes addresses pointing to two addends.

6. The data processing apparatus according to any one of claims 1-4, wherein, The tensor interface circuit is configured to determine, based on the shape information, the data address of the tensor data corresponding to the operand in the data storage space; and / or The tensor interface circuit is configured to determine the dependencies between instructions based on the spatial information.

7. The data processing apparatus according to any one of claims 1-4, wherein the shape information of the tensor data includes at least one shape parameter representing the shape of the N-dimensional tensor data, where N is a positive integer, and the shape parameter of the tensor data includes at least one of the following: The dimensions of the data storage space containing the tensor data in at least one of the N dimensions, the dimensions of the storage area of ​​the tensor data in at least one of the N dimensions, the offset of the storage area in at least one of the N dimensions, the positions of at least two vertices at diagonal positions in the N dimensions relative to the data reference point, and the mapping relationship between the data description location and the data address of the tensor data.

8. The data processing apparatus according to any one of claims 1-4, wherein the fusion instruction includes an input first descriptor for indicating the address offset of the K-channel data, the address offset of the K-channel index, and the size of the K-channel data.

9. The data processing apparatus of claim 8, wherein the fusion instruction further includes an input second descriptor for indicating a reference point address of the K-channel data, wherein the reference point address and the address offset of the K-channel data indicated by the first descriptor are combined to access the K-channel data.

10. The data processing apparatus of claim 8, wherein the fusion instruction further includes an input third descriptor for indicating a reference point address of the K-way index, wherein the reference point address and the address offset of the K-way index indicated by the first descriptor are combined to access the K-way index.

11. The data processing apparatus according to any one of claims 1-4, wherein the fusion instruction further includes an input fourth descriptor for indicating the storage address of the final execution result of the operation structure element.

12. The data processing apparatus according to any one of claims 1-4, wherein the fusion instruction includes an output fifth descriptor for indicating the fusion index and the operation structure element.

13. The data processing apparatus according to any one of claims 1-4, wherein the data elements in the data to be fused are effective data elements after sparsification in radar object detection, and the index indicates the position information of the effective data elements in the data before sparsification.

14. The data processing apparatus according to any one of claims 1-4, wherein the fusion instruction includes an operation mode bit to indicate a fusion processing operation of the fusion instruction, or the fusion instruction itself indicates the fusion processing operation.

15. A chip comprising a data processing apparatus according to any one of claims 1-14.

16. A circuit board comprising the chip according to claim 15.

17. A data processing method, comprising: A fusion instruction is parsed, which instructs the fusion processing to be performed on multiple streams of data to be fused, and at least one operand of the fusion instruction includes at least one descriptor, which indicates at least one of the following: shape information of tensor data and spatial information of tensor data; The descriptor is parsed; Based on the parsed descriptor, and according to the fusion instruction, the data elements in the multiple streams of data to be fused are grouped into a single ordered fused data stream according to their corresponding indices. Each fused data element is represented using an operational structure element, and the data element includes any of scalar, vector, or higher-dimensional data; and Output the fused data; The fusion instruction operates on K-channel data to be fused, K-channel indices corresponding to the K-channel data, the dimensions of the K-channel data, and one fusion index and one fusion data output, where K > 1. The index elements in the K-way index indicate the index information of the corresponding data elements in the K-way data; In the K-way index, the index elements of each way index are arranged in a first order; The data elements of each data path in the K-path data are arranged in order according to the corresponding index. The fused index elements in the fused index are arranged in a second order, wherein identical index elements are merged into the same fused index element; and The fused data elements in the fused data correspond one-to-one with the fused index elements, and the fused data elements composed of data elements with the same index are represented by the operation structure elements of the data element accumulation operation.

18. The data processing method according to claim 17, further comprising: The sorting circuit sorts the K-way indexes according to the size of the index elements and outputs them in an orderly manner to the output circuit; as well as In response to receiving an index element identical to the previous fused index element from the sorting circuit, the output circuit generates an operation structure element representing an accumulation operation that adds the data element corresponding to the current index element to the fused data element corresponding to the previous fused index element, and removes duplicate index elements.

19. The data processing method according to claim 18, further comprising: In response to receiving an index element that is different from the previous fused index element from the sorting circuit, the current index element is output as the new fused index element; as well as Generate an operation structure element that represents adding the data element corresponding to the current index element to the data at the specified address, where the data at the specified address is 0.

20. The data processing method according to any one of claims 17-19, wherein each operation structure element indicates an in-situ addition operation, which includes addresses pointing to two addends.

21. The data processing method according to any one of claims 17-19, wherein parsing the descriptor includes: Based on the shape information, determine the data address of the tensor data in the data storage space; and / or Based on the spatial information, the dependencies between instructions are determined.

22. The data processing method according to any one of claims 17-19, wherein the shape information of the tensor data includes at least one shape parameter representing the shape of the N-dimensional tensor data, where N is a positive integer, and the shape parameter of the tensor data includes at least one of the following: The dimensions of the data storage space containing the tensor data in at least one of the N dimensions, the dimensions of the storage area of ​​the tensor data in at least one of the N dimensions, the offset of the storage area in at least one of the N dimensions, the positions of at least two vertices at diagonal positions in the N dimensions relative to the data reference point, and the mapping relationship between the data description location and the data address of the tensor data.

23. The data processing method according to any one of claims 17-19, wherein the fusion instruction includes an input first descriptor for indicating the address offset of the K-way data, the address offset of the K-way index, and the size of the K-way data.

24. The data processing method according to claim 23, wherein the fusion instruction further includes an input second descriptor for indicating the reference point address of the K-channel data, and parsing the descriptor further includes: The address of the K-channel data is obtained by combining the reference point address and the address offset of the K-channel data indicated by the first descriptor.

25. The data processing method according to claim 23, wherein the fusion instruction further includes an input third descriptor for indicating the reference point address of the K-way index, and parsing the descriptor further includes: The address of the K-way index is obtained by combining the reference point address and the address offset of the K-way index indicated by the first descriptor.

26. The data processing method according to any one of claims 17-19, wherein the fusion instruction further includes an input fourth descriptor for indicating the storage address of the final execution result of the operation structure element.

27. The data processing method according to any one of claims 17-19, wherein the fusion instruction includes an output fifth descriptor for indicating the fusion index and the operation structure element.

Images