Data processing circuit, data processing method and related product

Abstract

The present disclosure discloses a data processing circuit, a data processing method and related products. The data processing circuit can be implemented as a computing device included in a combined processing device, which can further include an interface device and other processing devices. The computing device interacts with the other processing devices to jointly complete a user-specified computing operation. The combined processing device can further include a storage device connected to the computing device and the other processing devices respectively, for storing data of the computing device and the other processing devices. The scheme of the present disclosure provides a pipeline implementation scheme for data fusion processing, which can simplify the processing and improve the processing efficiency of the machine.

Description

Technical Field

[0001] This disclosure generally relates to the field of data processing. More specifically, this disclosure relates to data processing circuits, data processing methods, chips, and circuit boards. Background Technology

[0002] In recent years, significant progress has been made in object detection, instance segmentation, and keypoint detection based on convolutional neural networks. These detection methods are typically based on LiDAR or RGB-D data and can be applied to fields such as autonomous driving and robot vision.

[0003] Unlike dense image data, LiDAR point cloud data is typically sparse, and the point density varies dramatically due to factors such as non-uniform sampling in 3D space, the effective range of the sensor, occlusion, and relative pose. Therefore, traditional convolutional neural networks, which are well-suited for dense data, become very inefficient when applied to this type of sparse data, especially when convolution operations are involved, wasting significant computational resources on zero-value data points.

[0004] Therefore, it is desirable to provide an improved data processing scheme suitable for sparse data such as point cloud data, thereby improving processing efficiency. Summary of the Invention

[0005] In order to at least partially solve one or more of the technical problems mentioned in the background art, the present disclosure provides a data processing circuit, a data processing method, a chip, and a board.

[0006] In a first aspect, this disclosure discloses a data processing circuit, including a control circuit, a storage circuit, and a computation circuit, wherein: the control circuit is used to parse fusion instructions and control the storage circuit and the computation circuit to perform fusion processing on multiple streams of data to be fused according to the fusion instructions; the storage circuit is configured with at least two buffers to support data access between one buffer and an external storage circuit while simultaneously performing data access between the other buffer and the computation circuit, the buffers being used to store information, the information including pre-processing and / or post-processing information; and the computation circuit, under the control of the control circuit, is used to group data elements from the multiple streams of data to be fused from the storage circuit into one ordered fused data stream according to their corresponding indices, wherein data elements with the same index are merged into one fused data element, the data element including any of scalar, vector, or higher-dimensional data.

[0007] In a second aspect, this disclosure provides a chip including the data processing circuitry of any embodiment of the first aspect. In a third aspect, this disclosure provides a board including the chip of any embodiment of the second aspect. In a fourth aspect, this disclosure provides a method for processing data using the data processing circuitry of any embodiment of the first aspect.

[0008] By providing the data processing circuit, the method for processing data using the data processing circuit, the chip, and the board as described above, this disclosure provides a pipelined scheme for the data fusion processing step in the convolution operation of sparse data, thereby shortening processing time and improving overall efficiency. Furthermore, when the storage circuit space is insufficient to complete all operations at once, multi-round pipelined processing can be used. In multi-round pipelined processing, bucket sort can be used to allocate the data portions of each round of pipelined processing, thereby achieving the orderly concatenation of the output data from each round of pipelined processing. Attached Figure Description

[0009] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding portions wherein:

[0010] Figure 1 This diagram shows the structure of the board card according to an embodiment of this disclosure;

[0011] Figure 2 This diagram illustrates the structure of the combined processing apparatus according to an embodiment of the present disclosure.

[0012] Figure 3 A schematic diagram illustrating the internal structure of a processor core in a single-core or multi-core computing device according to embodiments of the present disclosure;

[0013] Figure 4 This illustrates the operational principle of a conventional convolution scheme;

[0014] Figure 5 An exemplary schematic diagram of the sparse convolution scheme of this disclosure embodiment is shown;

[0015] Figure 6 This diagram illustrates the preprocessing of high-dimensional sparse input data according to an embodiment of this disclosure.

[0016] Figure 7 This illustrates the meaning of the multiplication operation in the embodiments disclosed herein;

[0017] Figures 8A-8C This shows the index mapping relationship between the product result and the convolution operation result;

[0018] Figure 9This illustrates the impact of padding in convolution operations on the input data index.

[0019] Figure 10 This illustrates an exemplary principle of the MERGE instruction;

[0020] Figure 11 This diagram shows a schematic representation of the data processing circuit in an embodiment of the present disclosure.

[0021] Figure 12 This schematic illustrates the buffer space partitioning in the storage circuit;

[0022] Figure 13 An exemplary schematic diagram of bucket sort is shown;

[0023] Figure 14 An exemplary pipeline process for fusion processing according to embodiments of this disclosure is shown; and

[0024] Figure 15 An exemplary flowchart illustrating a data processing method according to an embodiment of this disclosure is shown. Detailed Implementation

[0025] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0026] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this disclosure are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0027] It should also be understood that the terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0028] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0029] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

[0030] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0031] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0032] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0033] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a storage device 204.

[0034] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0035] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0036] Processing device 203, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0037] Storage device 204 is used to store data to be processed. It may be DRAM or DDR memory, typically 16G or larger in size, and is used to store data of computing device 201 and / or processing device 203.

[0038] Figure 3The diagram shows the internal structure of the processor core when the computing device 201 is a single-core or multi-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three main modules: a control module 31, an arithmetic module 32, and a storage module 33.

[0039] The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 fetches instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.

[0040] The computation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 322 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0041] Storage module 33 is used to store or move relevant data, including neuron RAM (NRAM) 331, weight RAM (WRAM) 332, and direct memory access (DMA) module 333. NRAM 331 is used to store input neurons, output neurons, and intermediate results after computation; WRAM 332 is used to store the convolution kernels of the deep learning network, i.e., the weights; DMA 333 is connected to DRAM 204 through bus 34 and is responsible for data transfer between computing device 301 and DRAM 204.

[0042] Based on the aforementioned hardware environment, in one aspect, this disclosure provides a data processing circuit that performs multi-channel data fusion processing according to specific fusion instructions. In another aspect, this disclosure also provides a convolution operation scheme suitable for sparse data, which can employ the fusion processing of the first aspect of this disclosure. To better understand the role of multi-channel data fusion processing, a convolution operation scheme suitable for sparse data is first described below. This sparse convolution scheme is applicable to multi-dimensional convolution operations, including but not limited to two-dimensional and three-dimensional convolution. For simplicity and ease of understanding, two-dimensional convolution is used as an example in some embodiments.

[0043] In this disclosure, "N-dimensional convolution" refers to the number of convolutional dimensions in which sliding accumulation is performed. For example, when N=2, the convolutional kernel performs translational accumulation along two dimensions (e.g., width W and height H) according to the corresponding convolutional stride. When N=3, the convolutional kernel performs translational accumulation along three dimensions (e.g., width W, height H, and depth D) according to the corresponding convolutional stride. When N=4, the convolutional kernel performs translational accumulation along four dimensions (e.g., width W, height H, depth D, and batch) according to the corresponding convolutional stride. In this disclosure, "non-convolutional dimension" refers to a dimension in which the convolutional kernel does not perform sliding accumulation.

[0044] To better understand the convolution scheme of this disclosure embodiment, the operational principle of a conventional convolution scheme will be described first using two-dimensional convolution as an example.

[0045] Figure 4 This illustrates the operational principle of a conventional convolution scheme. In this example, the convolution kernel 410 is dense, a 3×3 matrix, with the numbers in the kernel representing the corresponding weights. The input data 420 is a 6×6 matrix that is sparse, containing only three non-zero values: 2, 3, and 5, as shown by the dark squares. For simplicity, in this exemplary convolution process, the stride for both dimensions is set to 1, with zero padding (no dilation). The 3×3 gray squares in the diagram represent the sliding accumulation process of the convolution kernel across the input data. 430 shows the computation at the start of the convolution, 440 shows the computation one step to the right, and 450 shows the computation one step down. In each step, the weights of the convolution kernel are multiplied bitwise with the input data and accumulated. 460 shows the final computation result as the output data. The output data is a 4×4 matrix. It can be seen that the calculation of 430 corresponds to the data at coordinate (1,1) in the output data, the calculation of 440 corresponds to the data at coordinate (1,2) in the output data, and the calculation of 450 corresponds to the data at coordinate (2,1) in the output data.

[0046] In the sparse convolution operation of the embodiments disclosed herein, the convolution kernel is dense, and its input format can be the same as that of conventional convolution; while the input data is sparse, and its input format can be different from that of conventional convolution input data, thereby saving storage space. In some embodiments, the input data is sparse data, and each input data element has index and numerical information, which can be represented as (index, value). Depending on the application, data type, or scale, the index here can be one-dimensional, two-dimensional, or more-dimensional, and this disclosure is not limited in this respect. Indices of different dimensions can be converted to each other, for example, two-dimensional or more-dimensional indices can be converted to one-dimensional indices according to a predetermined traversal rule, and vice versa. Similarly, depending on the application, data type, or scale, the numerical information here can include scalars, vectors, or higher-dimensional data.

[0047] by Figure 4 Taking the example in the example, the input data has three non-sparse points: ((1,4),2), ((3,3),3), and ((5,6),5). For the first non-sparse point, (1,4) represents the two-dimensional index coordinates, and 2 represents the specific numerical value of the position pointed to by the index, which is a scalar in this example; the meanings of the other non-sparse points are similar.

[0048] from Figure 4 As can be seen from the description, the final result of sparse convolution depends only on the results of operations on non-zero input data elements. Therefore, multiplication and addition operations with the convolution kernel can be performed only on these non-zero input data elements. Furthermore, from... Figure 4 As can be seen from the output data 460, these multiply-accumulate operations can be further broken down into multiplication operations and corresponding positional accumulation operations. Accordingly, in the sparse convolution scheme of this disclosure embodiment, the sparse convolution operation can include three steps: calculating the operation result of the convolution kernel and the sparse input data (e.g., the product result or the multiply-accumulate result, considering that some dimensions of the data also need to be accumulated, such as the input channel dimension Ci described later); determining the index of each operation result; and merging the multi-way operation results into one fused data in index order as the result of the sparse convolution operation. During the merging process, operation results with the same index are accumulated. In the following description, depending on the dimension of the data, in scenarios where the input channel dimension Ci is not mentioned, the multi-way operation result is sometimes directly referred to as the multi-way multiplication result. Those skilled in the art can understand its corresponding meaning based on the context.

[0049] Figure 5 An exemplary principle of a sparse convolution scheme according to an embodiment of this disclosure is shown. Figure 5 Still with Figure 4 The sparse convolution operation scheme implemented in this disclosure will be described using data as an example.

[0050] like Figure 5 As shown, in the MAC step, the input data element 520 is multiplied with the convolution kernel 510 to obtain a multi-way product result 530. When calculating the product result of the convolution kernel and the sparse input data, the weight data of each convolution dimension of the convolution kernel can be merged into one dimension, with the dimension size being offset, where offset is the total number of weight data in the convolution dimension.

[0051] As mentioned above, the sparse convolution scheme provided in this disclosure embodiment can be applied to multidimensional convolution operations, including but not limited to two-dimensional convolution and three-dimensional convolution. Figure 5 The diagram illustrates merging the weight data from the two convolutional dimensions (width W and height H) of a two-dimensional convolutional kernel into a single dimension. For example, the weight data from a 3×3 matrix can be expanded into a column containing nine weight data points, which can also be referred to as nine weight scalars.

[0052] It's understandable that when the convolution kernel is a three-dimensional kernel, the weight data across its three convolutional dimensions (width W, height H, and depth D) can be merged into a single dimension. For example, the 27 weight data from a 3×3×3 cubic convolution kernel can be unfolded into a single column, resulting in 27 weight scalars. Other cases can be deduced similarly. If the convolution kernel also has non-convolutional dimensions, such as the input channel Ci dimension and / or the output channel Co dimension, these dimensions can be retained since convolution operations are not performed on them. In this case, for example, if the convolution kernel includes a 3×3 convolutional dimension and also a non-convolutional dimension Ci = 2, then after merging these dimensions, the convolution kernel becomes 9×2, which can be viewed as nine weight vectors of length 2. For example, if the convolution kernel includes a 3×3×3 convolution dimension, as well as non-convolution dimensions Ci=2 and Co=4, then after merging the above dimensions, the convolution kernel becomes 27×2×4, which can be regarded as 4 sets of weights on the Co dimension, each set including 27 weight vectors of length 2.

[0053] The input data to be processed by convolution can include multidimensional data, and it is sparse across multiple dimensions. For example, in target detection based on LiDAR data, the input data is detection data in three-dimensional space, which may represent the grayscale value, RGB values, signal strength, etc., of each three-dimensional spatial coordinate point. Therefore, depending on the information content to be represented, the input data element at each coordinate point can be one-dimensional, two-dimensional, three-dimensional, or higher-dimensional data. Due to the characteristics of point cloud data, coordinate points with non-zero value data elements are sparse, that is, they are sparse in three spatial dimensions (e.g., width W, height H, and depth D).

[0054] Depending on the initial state of the input data, preprocessing can be performed before the sparse input data is provided to the computation circuit for processing. In some embodiments, such preprocessing may include, for example, merging multiple sparse dimensions into one dimension; densifying sparse data points in the input data along the merged dimension to form input data elements; and representing each input data element using indexes and numerical information. The index can be a one-dimensional or multi-dimensional index, and the numerical information can include any of scalar, vector, or higher-dimensional data.

[0055] In one example, for instance, reference Figure 4 The input data is a two-dimensional 6×6 matrix, sparse in both width (W) and height (H). During preprocessing, W and H are merged into a single dimension, and the sparse data points (2, 3, and 5 in this example) are compacted along this merged dimension, resulting in three compacted input data elements. Each input data element is then represented using indexes and numerical information. The index of a data element indicates its position within the sparse input data before compaction. For example, Figure 4 The indices of the three input data elements in the example are (1,4), (3,3), and (5,6). These indices are two-dimensional, but can be converted to one-dimensional indices, such as 4, 15, and 29. The numerical values ​​of these three data elements are 2, 3, and 5, representing three scalars.

[0056] In another example, for example, reference Figure 6The figure illustrates a schematic diagram of preprocessing high-dimensional sparse input data according to an embodiment of this disclosure. As shown, the sparse input data 610 includes five dimensions: a batch (B) dimension, a three-dimensional HWD space dimension, and an input channel Ci dimension. The input data is sparse in the B dimension and the HWD three-dimensional space. In the figure, the dark squares in the HWD matrix represent locations with numerical values, while the rest are all zero values. Multiple such HWD matrices exist in the B dimension, and the sparsity pattern (i.e., the positions of the dark squares) on each matrix can be different. The input data is dense in the Ci dimension, which is the lowest dimension. Due to the limited representation capabilities of the figures, only four dimensions are shown in the figure 610, but the Ci dimension can be understood as the thickness of each dark square. The size of the Ci dimension is uniform, meaning the thickness of each dark square is the same. During preprocessing, the four sparse dimensions of the input data (the B dimension and the HWD three-dimensional space dimension) can be merged into a single dimension Ni, and the sparse data points (the dark squares in the figure) can be densified in the merged dimension to form dense input data elements. That is, each HWD 3D matrix in the B dimension undergoes the same dimension merging and compaction process to obtain the preprocessed compact form of the input data 620, which is a two-dimensional matrix with Ci in the low dimension and Ni in the high dimension of the BHWD merging dimension.

[0057] Next, index and numerical information are used to represent each densified input data element. Figure 6 The indices of the input data elements in the example can be represented using their coordinates in the uncompacted BHWD four-dimensional space, or they can be converted into one-dimensional indices. The numerical information of each input data element can be viewed as a Ci vector.

[0058] The input data, after the above preprocessing, can be provided to the arithmetic circuit for further processing.

[0059] The input data elements can be arranged into an input vector to perform multiplication with the convolution kernel. In some embodiments, the input data elements can be arranged in their index order (e.g., ascending order) to accommodate subsequent merging. Each vector element in the input vector comprises one input data element. As described above, each input data element can be a scalar, vector, or higher-dimensional data.

[0060] Back Figure 5 Next, a multiplication operation can be performed on the convolutional dimension of the convolutional kernel 510 after dimension merging and the input vector 520 composed of input data elements, resulting in offset data 530, where each data path includes several product results, and each product result includes any of the following: scalar, vector, or higher-dimensional data. Figure 5In the example, the dimension-merged convolution kernel 510 (the nine scalars in the diagram) performs scalar vector multiplication with the input vector 520 (a vector of length 3 consisting of 2, 3, and 5), resulting in nine vectors, or nine-way product results 530. In this example, each product result is a scalar.

[0061] The MAC step is equivalent to performing a 1×1 point-level convolution on the input vector for each row of data in the dimension-merged convolution kernel, resulting in a convolution result. To better understand the meaning of the multiplication operation in the MAC step above, Figure 7 The meaning of several of these multiplication operations is illustrated by example.

[0062] As shown in the figure, for Figure 5 The operation on the first weight scalar in the diagram: 1*[2,3,5]=[2,3,5], can be understood as corresponding to the operations at positions 710, 720, and 730 in the diagram, respectively, that is, the product results generated when 2, 3, and 5 are located at the first position of the convolution kernel. Similarly, for... Figure 5 The operation of the second weight scalar in the diagram: 2*[2,3,5]=[4,6,10], can be understood as the operation corresponding to 740, 750 and 760 in the diagram, that is, the product result when 2, 3 and 5 are located at the second position of the convolution kernel.

[0063] As mentioned earlier, input data can also include non-convolutional dimensions, such as those described in the previous reference. Figure 6 The input channel Ci dimension is described. In some embodiments, the input data may include N convolutional dimensions and at least one non-convolutional dimension, and correspondingly, the convolutional kernel may also include N convolutional dimensions and at least one non-convolutional dimension. In this case, in the above MAC step, the corresponding operation processing can be performed on the non-convolutional dimensions according to the specific operation requirements. For example, in some examples, the non-convolutional dimensions of the input data include the input channel Ci dimension, and the non-convolutional dimensions of the convolutional kernel include the input channel Ci dimension and the output channel Co dimension. The operation processing on these non-convolutional dimensions may include, but is not limited to: keeping the results of the Ci dimension from being accumulated (e.g., in depthwise convolution); performing positional accumulation operation on the multiplication results of the convolutional dimensions on the Ci dimension (e.g., accumulation on the Ci dimension); and / or stacking the accumulation results on the Ci dimension on the Co dimension (e.g., accumulation on the Ci dimension, but not on the Co dimension) to obtain the multi-way operation result, where each operation result is a vector on the Co dimension.

[0064] continue Figure 5The MAC step obtains the operation results related to non-zero values. To obtain the final convolution operation result in the subsequent MERGE step, it is necessary to determine the indices of these operation results for corresponding accumulation. Therefore, the corresponding indices of these operation results can be obtained in the INDEX step. Specifically, in some embodiments, the index of each operation result in the multiplication results is determined according to the index mapping relationship. Each operation result is obtained by multiplying or multiplying and adding the weight data in the convolution kernel with the input data elements. Therefore, the index mapping relationship indicates the relationship between the position of the weight data, the index of the input data element, and the corresponding result element in the convolution operation result. In other words, the index of the operation result obtained by multiplying or multiplying and adding the weight data can be determined based on the position of the weight data in the convolution kernel and the index of the input data element.

[0065] Figure 8A This example illustrates the index mapping relationship between partial product results and convolution operation results in the MAC step. The meanings of 810A, 820A, and 830A in the figure are... Figure 7 Similar to 710, 720, and 730, these represent the product operation of the first weight scalar and the input vector: 1*[2,3,5]=[2,3,5]. The arrows in the diagram indicate the corresponding positions of each product result (810A, 820A, and 830A) in the convolution result (840A). Specifically, the product result of 810A corresponds to position (1,4) in the 4×4 convolution result, the product result of 820A corresponds to position (3,4) in the convolution result, and the product result of 830A overflows the convolution result range and is therefore invalid.

[0066] from Figure 8A It can be seen that when the input vector is multiplied with the first weight data of the convolution kernel, the index has the following mapping relationship: Assuming that the index of the vector element (i.e., data points 2, 3 and 5) in the input vector is (x, y), then the index of the product result of the product operation with the first weight data is also (x, y).

[0067] Figure 8B This example illustrates the index mapping between partial product results and convolution operation results in the MAC step. In the figure, 810B, 820B, and 830B represent the product operation of the second weight scalar and the input vector: 2*[2,3,5]=[4,6,10]. The arrows in the figure indicate the corresponding positions of each product result (810B, 820B, and 830B) in the convolution operation result (840B). Specifically, the product result of 810B corresponds to position (1,3) in the 4×4 convolution result, the product result of 820B corresponds to position (3,2) in the convolution result, and the product result of 830B overflows the range of the convolution result and is an invalid result.

[0068] from Figure 8B It can be seen that when the input vector is multiplied with the second weight data of the convolution kernel, the index has the following mapping relationship: Assuming that the index of the vector element (i.e., data points 2, 3 and 5) in the input vector is (x, y), then the index of the product result of the product operation with the second weight data is (x, y-1).

[0069] Figure 8C This example illustrates the index mapping between partial product results and convolution operation results in the MAC step. In the figure, 810C, 820C, and 830C represent the product operation of the 9th weight scalar with the input vector: 1*[2,3,5]=[2,3,5]. The arrows in the figure indicate the corresponding positions of each product result (810C, 820C, and 830C) in the convolution operation result (840C). Specifically, the product result of 810C overflows the convolution result range and is invalid; the product result of 820C corresponds to position (1,1) in the 4×4 convolution result, and the product result of 830C corresponds to position (3,4) in the convolution result.

[0070] from Figure 8C It can be seen that when the input vector is multiplied by the 9th weight data of the convolution kernel, the index has the following mapping relationship: Assuming that the index of the vector element (i.e., data points 2, 3 and 5) in the input vector is (x, y), then the index of the product result of the product operation with the 9th weight data is (x-2, y-2).

[0071] comprehensive Figures 8A-8C As can be seen, each vector element in the input vector (i.e., data points 2, 3, and 5) sequentially traverses the 3×3 convolution kernel. Therefore, the offset of each data point relative to the center point of the convolution kernel (here, the center of the 3×3 convolution kernel is the 5th weight data) is fixed. Based on this characteristic, the index of the center point of the convolution kernel can be obtained sequentially based on the specific location of the data points. Then, the index of the center point can be mapped to the index of the output point. Thus, the index of the product result associated with each vector element in the input vector can be directly determined based on the index of that vector element. That is, knowing only the index of the input data element allows us to determine the index of the product result of that input data element multiplied by all the weight data.

[0072] For example, for a 3×3 two-dimensional convolution kernel, the coordinate offset relative to the center point of this two-dimensional convolution kernel is a constant when data points traverse the kernel. In this example, these 9 coordinate offsets can be constructed as follows:

[0073] (1,1),(0,1),(-1,1);

[0074] (1,0),(0,0),(-1,0);

[0075] (1,-1),(0,-1),(-1,-1).

[0076] For example, for a 3×3×3 convolution kernel, the coordinate offset relative to the center point of this 3D convolution kernel is a constant when data points traverse the kernel. In this example, these 27 coordinate offsets can be constructed as follows:

[0077] (1,1,1),(0,1,1),(-1,1,1),(1,0,1),(0,0,1),(-1,0,1),(1,-1,1),(0,-1,1),(-1,-1,1);

[0078] (1,1,0),(0,1,0),(-1,1,0),(1,0,0),(0,0,0),(-1,0,0),(1,-1,0),(0,-1,0),(-1,-1,0);

[0079] (1,1,-1),(0,1,-1),(-1,1,-1),(1,0,-1),(0,0,-1),(-1,0,-1),(1,-1,-1),(0,-1,-1),(-1,-1,-1).

[0080] Therefore, based on the indices of the input data points and the aforementioned fixed coordinate offsets, the indices of the convolution kernel center points corresponding to each convolution kernel traversal can be sequentially calculated. Then, mapping the center point indices to the output point indices determines the index of each product result generated by the input data points.

[0081] from Figures 8A-8C The diagram also shows that some product results have overflowed the range of the convolution results and are therefore invalid. For these cases, the indexes exceeding the range of the convolution results (i.e., the range of the output data dimensions) can be set to a predetermined value, such as -1, to identify these invalid results in subsequent processing and prevent them from being merged.

[0082] In some embodiments, convolution operations involve padding. For example, in object detection algorithms based on LiDAR data, same padding is required, meaning that padding ensures the shape of the input data is the same as the shape of the output data after the convolution operation. It is understandable that different padding rules may exist in other applications of convolution operations.

[0083] Figure 9 The effect of padding in convolution operations on the input data index is shown.

[0084] As shown in the figure, matrix 910 represents the initial input data, and matrix 920 represents the padded input data. The initial input data is, for example, a 2×3 matrix. The padded input data is processed according to the padded rules, by adding one column to the left, three columns to the right, four rows to the top, and one row to the bottom of the initial input data. The padded data can be, for example, zero.

[0085] For any data point (x, y) in the initial input data, its coordinates in the input data after padding become (x + pad_left, y + pad_top), where pad_left is the padding amount on the left and pad_top is the padding amount on the top. Therefore, the indices of the input data can be adjusted using simple addition operations and according to the padding rules.

[0086] In this embodiment, the indices of the input data elements can be adjusted based on the padding rules of the convolution operation before determining the index of the product result according to the index mapping relationship.

[0087] Those skilled in the art will understand that index adjustment processing can also be performed after or during index mapping, as long as the impact of the filling rules is taken into account, and the embodiments disclosed herein are not limited in this respect.

[0088] Back Figure 5 Figure 540 shows the indices corresponding to the 9-way product results determined by the INDEX step, with each product result having a corresponding index. Dark squares indicate invalid results, whose indices are set to -1.

[0089] After obtaining the multi-way product results through the MAC step and determining the index of each product result through the INDEX step, these multi-way product results can be further fused to obtain the convolution operation result.

[0090] Specifically, in the MERGE step, the multiplication results are merged and combined according to their index order to obtain the convolution operation result, where multiplication results with the same index are accumulated. Figure 550 shows the merged indexes, where duplicate indices, such as indices 2 and 3, have been removed. Figure 560 shows the merged data, where multiplication results with the same index are accumulated; for example, the data corresponding to two indices 2, 6 and 12, are accumulated, and the data corresponding to two indices 3, 4 and 3, are accumulated.

[0091] By comparison Figure 4 and Figure 5 The calculation results show that, based on Figure 5 The fused index 550 and fused data 560 can recover the sparse form of the convolution operation result, which completely corresponds to Figure 4 The result of a regular 4×4 convolution operation is 460.

[0092] Therefore, the above describes a convolution operation scheme for sparse data, in which the effective product results can be sorted and accumulated through data fusion processing (MERGE step). This data fusion step can be implemented using a specially designed hardware instruction, the MERGE instruction. The basic function of the MERGE instruction is to merge multiple streams of data into a single stream according to their index order. The MERGE instruction can have multiple processing modes to adapt to different needs. The MERGE instruction can include a mode flag to indicate different processing modes.

[0093] Figure 10 The diagram illustrates the exemplary principle of the MERGE command. The figure exemplifies four streams of data to be fused, each stream comprising six data elements. Data elements can be scalars, vectors, or higher-dimensional tensors. In the figure, data elements are exemplarily shown as vectors, such as D11, D12, ..., D46. These vectors have a uniform length; for example, D11 is (d1, d2, d3, ..., dn) with a length of n. Each data element has an associated index indicating its position within the corresponding stream of data. For example, the original stream of data might contain 1000 data elements, but only some of these elements are valid. In this case, these valid elements can be extracted to form the data to be fused, and their corresponding indices can be extracted to indicate their positions in the original data; these indices form the fusion index.

[0094] The diagram schematically illustrates the four indices to be merged, with each index corresponding to one channel of data to be merged. The first index identifies the position of each data element in the first channel, the second index identifies the position of each data element in the second channel, and so on. Furthermore, the index elements in each channel are stored in an ordered manner and correspond one-to-one with the data elements in the corresponding channel. In the example shown, the index elements in each channel are arranged in a first order (e.g., ascending order), and the data elements in each channel are also arranged in the order of their corresponding indices. For example, the first index element in the first channel indicates that the index of the first data element in the first channel is 0, i.e., the first element; the second index element in the first channel indicates that the index of the second data element in the first channel is 2, i.e., the third element; and so on.

[0095] The figure shows exemplary results of the MERGE command in different processing modes.

[0096] In the first processing mode, Mode 1, also known as the "merge sort mode," only the indices of the aforementioned data are processed. Specifically, after the merging process, the indices of the data from each path are merged into a single merged index, and the merged index elements are arranged in a second order (e.g., ascending order). Duplicate index elements are retained in this merge sort process. As shown in the figure, the four indices to be merged are merged into a single merged index, comprising 24 data elements.

[0097] In the second processing mode, Mode 2, also known as the "sorting and accumulating mode," data elements from multiple streams of data to be merged are grouped into a single ordered merged data stream according to their corresponding indices. Data elements with the same index are accumulated and merged into a single merged data element. As shown in the figure, four streams of data to be merged are grouped into a single ordered merged data stream according to their corresponding indices, and data elements with the same index are accumulated and merged into a single merged data element. In this example, the merged index includes 16 index elements, arranged in a second order (e.g., ascending order), where duplicate index elements in the original merged indexes have been removed, as shown by the dark squares in the figure. Correspondingly, the merged data also includes 16 data elements, arranged in the order of their corresponding indices, and data elements with the same index are accumulated and merged into a single merged data element, as shown by the dark squares in the figure. The second processing mode, Mode 2, is commonly used in sparse matrix multiplication operations; therefore, it can also be called the "matrix multiplication mode."

[0098] In the third processing mode, Mode3, similar to the second processing mode, sorting and accumulation processing is also required. Figure 10 The second and third processing modes are illustrated using the same processing result. The difference between these two modes lies in the output format. In the second processing mode, for cases where accumulation exists in the fused data elements, the accumulated result is directly output. In the third processing mode, for at least cases where accumulation exists in the fused data elements, the accumulated result is not output; instead, the relevant accumulation expression is output. In some implementations, all fused data elements can be output as expressions for easier, uniform processing. For example, fused data elements without accumulation can be represented as an accumulation expression with zero. This accumulation expression can be called an operation structure. In some implementations, each operation structure indicates an in-situ addition operation, including addresses pointing to the two addends. The third processing mode is particularly suitable for cases where the data elements to be fused are vectors or higher-dimensional tensors, such as in radar algorithms based on LiDAR data; therefore, the third processing mode can also be called the "radar algorithm mode."

[0099] Those skilled in the art will understand that the first order and the second order mentioned above may be the same or different, and both may be selected from either: an ascending order or a descending order. Those skilled in the art will also understand that although the diagram shows that each data path has an equal number of data elements, the number of data elements in each data path may be the same or different, and this disclosure is not limited in this respect. Furthermore, those skilled in the art will understand that since the MERGE instruction can have multiple processing modes, the required parameters may also vary accordingly in different processing modes. For example, in the first processing mode, it is not necessary to provide the data to be merged because only the index of the data is sorted. In the third processing mode, the output parameters also include the operation structure.

[0100] To accelerate the processing efficiency of MERGE instructions, this disclosed embodiment provides a pipeline scheme to support parallel processing of MERGE instructions.

[0101] Figure 11 An exemplary structural diagram of a data processing circuit that can implement embodiments of the present disclosure is shown. As shown, the data processing circuit 1100 includes a control circuit 1110, a storage circuit 1120, and an arithmetic circuit 1130.

[0102] The control circuit 1110 is responsible for handling various functions on the data processing circuit 1100, including but not limited to control, instruction fetching, decoding, and calculation. The control circuit 1110 may include, for example, […]. Figure 3 The control module 31 in the middle.

[0103] In some embodiments, the control circuit 1110 may be configured to parse fusion instructions and control the storage circuit 1120 and the arithmetic circuit 1130 to perform fusion processing on multiple channels of data to be fused according to the fusion instructions.

[0104] Storage circuit 1120 can be used to store information, which includes at least pre-processing and / or post-processing information, and may also include intermediate information that needs to be cached during processing, such as... Figure 3 The various RAMs shown are also known as on-chip caches.

[0105] In some embodiments, the storage circuit 1120 may be configured with at least two buffers 1121 and 1122 to support data access between one buffer (e.g., 1121) and an external storage circuit, while simultaneously performing data access between the other buffer (e.g., 1122) and the arithmetic circuit 1130. These two buffers may be referred to as ping-pong buffer space and pong buffer space, i.e., employing a ping-pong pipelined approach.

[0106] Specifically, when the arithmetic circuit 1130 performs calculations on data in the buffer space of the storage circuit 1120, the storage circuit 1120 can retrieve data from an external storage circuit (e.g., Figure 2 The storage device 204 (e.g., DRAM) loads the next computation data into its pong buffer space. The memory access interface between the storage circuit 1120 and the external storage circuit is different from the memory access interface between the storage circuit 1120 and the arithmetic circuit 1130. Therefore, the above-mentioned parallel mode can be supported, thereby forming a pipelined processing.

[0107] The arithmetic circuit 1130 can be configured to perform various arithmetic operations according to relevant instructions. Specifically, the arithmetic circuit 1130 can be configured, under the control of the control circuit 1110, to group data elements from multiple channels of data to be fused from the storage circuit 1120 into one ordered fused data channel according to their corresponding indices, wherein data elements with the same index are merged into one fused data element. Here, the data element can include any of scalar, vector, or higher-dimensional data.

[0108] In some embodiments, the arithmetic circuit 1130 may also be configured to perform the MAC step and INDEX step in the sparse convolution operation described in the preceding embodiments, that is, to perform multiplication operations on the input data elements and the convolution kernel to obtain multi-way product results and to calculate the corresponding index.

[0109] In one embodiment, the arithmetic circuit 1130 may further include an arithmetic processing circuit (not shown), which can be configured to preprocess the data before the arithmetic circuit performs the operation or postprocess the data after the operation according to the arithmetic instructions. In some application scenarios, the aforementioned preprocessing and postprocessing may include, for example, data splitting and / or data concatenation operations.

[0110] As mentioned earlier, the storage circuit 1120 can be, for example, Figure 3 The diagram illustrates various types of RAM, such as NRAM and WRAM. These RAMs have limited usable space, such as 512KB, and space must be reserved for the compiler, for example, 128KB. Therefore, the usable space during the execution of the MERGE instruction is only 512-128=384KB. Furthermore, to support pipelining, the available space for the memory circuitry can be divided into two buffer spaces: a ping-pong buffer and a ping-pong buffer. In some implementations, these two buffer spaces are evenly distributed to maximize pipelining efficiency. In the aforementioned example, the space available for each MERGE instruction execution, RAM_merge_size, is 384 / 2=192KB.

[0111] Due to the limited available space on the storage circuitry, space management is necessary to efficiently execute MERGE instructions. Two factors need to be considered during space management: first, space allocations must not be polluted or stacked; second, the buffer space must be large enough to hold the data processed each time.

[0112] Based on the above considerations, in some embodiments, the control circuit 1110 may be further configured to determine the maximum number of indexes Nmax that can be processed in one fusion process according to the available space size of the storage circuit 1120 and the relevant parameters of the fusion process, and then allocate buffer space in each buffer for each relevant parameter of the fusion process according to the determined Nmax.

[0113] The relevant parameters for fusion processing mainly include at least one of the following: the size of the multiple data streams to be fused (size_addr); the value of the multiple data streams to be fused (merge_input_mac_result); the index of the multiple data streams to be fused (merge_input_output_index); the value of the fused data (output_data); the index of the fused data (output_index); the operation structure representing the fused data elements (out_op_addr); and the data to be fused in each round of fusion processing (Compute_buffer).

[0114] The size of the multiple streams of data to be merged refers to the size of the K streams of input data that need to be merged, which can be indicated, for example, by the starting address of each stream. This address is a first-level pointer, which can be labeled size_addr, and includes K elements, where the i-th element represents the number of data elements in the i-th stream, and 0 < i ≤ K. It can be understood that when the MERGE instruction is used for the sparse convolution operation in the aforementioned embodiment, K = offset. A buffer space needs to be reserved in the storage circuit for size_addr, with a size of K * index_data_type = offset * index_data_type, where index_data_type represents the data type of the element at that address.

[0115] The number of other parameters is related to the maximum number of indices Nmax that can be processed in a single fusion process. Therefore, the maximum number of indices Nmax that can be processed can be determined based on the available space of the storage circuit and the requirements of these parameters, thereby further determining the space occupied by each parameter. In the following description, the space occupied by each parameter is described using the sparse convolution operation scenario of the aforementioned embodiment as an example.

[0116] For the numerical values ​​of multiple data streams to be fused, i.e., the input K data streams, their space can be calculated as: Nmax * Co * input_data_type, where Co represents the output channel dimension and input_data_type represents the data type of the input data. In the example of sparse convolution operation, the numerical values ​​of these multiple data streams to be fused are... Figure 5 The product result calculated in the MAC step can therefore be represented as merge_input_mac_result.

[0117] For the index of multiple streams of data to be merged, since there is a one-to-one correspondence between data and index, the space occupied by the K indices corresponding to the input K streams of data can be calculated as: Nmax * index_data_type, where index_data_type represents the data type of the index. In the example of sparse convolution operation, the index of the multiple streams of data to be merged is... Figure 5 The index calculated in the INDEX step can therefore be represented as merge_input_output_index. In some embodiments, the index elements in the input K-way index are ordered in each way, for example, in ascending order.

[0118] Regarding the numerical values ​​of the fused data, it's understandable that since the data to be fused undergoes fusion processing, the number of output data items after fusion will always be less than or equal to the number of data items to be fused. Its maximum space requirement is: Nmax * Co * output_data_type, where output_data_type represents the data type of the output data. The numerical values ​​of the fused data can be represented using output_data.

[0119] Similarly, the index of the merged data, i.e. the output index (denoted as output_index), occupies a maximum space of: Nmax*index_data_type.

[0120] For an operation structure representing fused data elements (denoted as out_op_addr), its maximum space requirement is Nmax*2*8. In this case, the operation structure has at most Nmax elements, each element being a structure, and each operation structure element indicates an in-place addition operation, including addresses pointing to the two addends. Each address can, for example, use 8 bytes.

[0121] For the data to be merged in each round of fusion processing (denoted as Compute_buffer), it represents the input index of the actual execution of the MERGE instruction, also known as the computation buffer space, the meaning of which will be described in detail later. The space occupied by this part can be calculated at most as: Nmax * index_data_type.

[0122] The above analysis of the space occupied by the relevant parameters of the fusion processing shows that the total space occupied, which is the sum of the space occupied by each item, is at most equal to the available space of the storage circuit. This relationship can be expressed as the following formula (1):

[0123] K*index_data_type+Nmax*Co*input_data_type+Nmax*index_data_type+Nmax*Co*output_data_type+Nmax*index_data_type+Nmax*2*8+

[0124] Nmax*index_data_type=RAM_merge_size (1)

[0125] Therefore, the maximum number of indices that can be processed in a single fusion process, Nmax, can be determined as follows: Nmax = (RAM_merge_siz - K * index_data_type) / (Co * input_data_type + index_data_type + Co * output_data_type + index_data_type + 2 * 8 + ...

[0126] index_data_type) (2)

[0127] Once Nmax is determined, the space occupied by each of the above parameters can also be determined.

[0128] Figure 12 The diagram illustrates the partitioning of buffer space in a storage circuit. As shown, the available space in the storage circuit is evenly divided into two buffer spaces: a ping buffer space 1210 and a pong buffer space 1220. Each buffer space is allocated corresponding buffer space for the aforementioned parameters, thus conforming to the principle of non-contamination and non-interference of space.

[0129] After allocating space resources for each parameter, the MERGE instruction can be executed in a pipelined manner. As described earlier regarding the MERGE principle, this fusion process is strongly correlated with the K-way index (i.e., merge_input_output_index) of the data to be merged; therefore, its pipeline is also strongly correlated with the K-way index of the data to be merged. In some embodiments, where the available space in the storage circuit cannot process all the data to be merged at once, a bucket sorting method can be used to process the data to be merged in batches. Furthermore, within each batch, the fusion processing of multiple data streams is implemented using a load-compute-store (LCS) pipelined approach.

[0130] Figure 13 This diagram illustrates an exemplary principle of bucket sort. The basic working principle of bucket sort is to divide the data to be sorted into a finite number of buckets, and then sort each bucket separately.

[0131] As shown in the diagram, assume the original array to be sorted contains 12 data items in a random order, and there are 4 buckets. Each bucket is responsible for sorting a specific range of data, and the data ranges within each bucket are ordered. For example, in the example shown, the data ranges of the four buckets are roughly evenly distributed, in ascending order: Bucket 1 has a data range of 0-25, Bucket 2 has a data range of 26-50, Bucket 3 has a data range of 51-75, and Bucket 4 has a data range of 76-100. Therefore, based on the data range of each bucket, the data in the original array can be assigned to the corresponding bucket. The diagram shows that Bucket 1 has 5 numbers, Bucket 2 has 2 numbers, Bucket 3 has 1 number, and Bucket 4 has 4 numbers. Next, sorting is performed within each bucket. Finally, the sorted arrays from each bucket are concatenated according to the bucket order to obtain the final sorted result. The diagram shows the concatenated sorted result.

[0132] The process of allocating data to buckets can also be represented as a mapping function f. Through the mapping function f, the key k to be sorted is mapped to the i-th bucket, and at this time the key k is the element in bucket B[i].

[0133] As described above, in some embodiments, the available space of the storage circuit is insufficient to process all the data to be fused at once, and its capacity is limited. Therefore, the storage circuit can be viewed as a bucket in the bucket sorting described above, and its capacity is also limited. Dividing the data to be fused into multiple batches and processing them on the storage circuit is equivalent to putting the data to be fused into multiple buckets for processing. Then, the fusion processing results of each batch are directly stored back to the external storage circuit (e.g., Figure 2 On the storage device 204 (e.g., DRAM), the results of sorting all buckets can be sequentially concatenated. The merging operation performed by the MERGE instruction can be viewed as a sorting operation within each bucket.

[0134] Therefore, in some embodiments, bucket sort is applied to utilize the limited space of the storage circuitry to achieve pipelined fusion processing of all data to be merged. In each round of pipelined processing, the control circuitry can select input indices that fall within the index range corresponding to the current round from the indices of the multiple data to be merged, and load them into the computation buffer space of the storage circuitry (i.e., the previously allocated compute_buffer). Subsequently, the arithmetic circuitry can execute the fusion processing indicated by the MERGE instruction on the indices in the computation buffer space and the data to be merged corresponding to these indices. Finally, the fused data is stored back to a designated location, such as the off-chip memory circuitry DDR.

[0135] To make the most of the storage space in each round of pipeline processing, it can be divided according to the index distribution of the data to be merged, thereby ensuring that the storage space is filled as much as possible in each round of pipeline processing.

[0136] In some embodiments, the control circuit can allocate the index range of each batch of water processing as follows, so that the amount of data processed in each batch is approximately the same and the available storage space is filled as much as possible. Based on the space management calculation method described above, the maximum buffer space allocated to the merged data value in the buffer can be determined as: Nmax * Co * output_data_type. Therefore, Nmax * Co can be used as the number of data items that can be processed in each batch of water processing (i.e., each bucket), which can also be called the bucket spacing N. spacing .

[0137] Once the distance between buckets is determined, the number of buckets N can be determined accordingly. bucket Divide the total number of data points by the bucket spacing N spacing When applied to sparse convolution operations, the total number of data points to be fused can be represented as Nin * offset, where Nin represents the number of non-sparse points in the input data, and offset represents the total number of weight data points in the convolution dimension of the convolution kernel. Therefore, the number of buckets can be represented as:

[0138] N bucket =Nin*offset / N spacing (3)

[0139] As can be seen from the above formula, the number of cycles (buckets) in the flow processing is directly proportional to the number of non-sparse points in the input data; in other words, it is directly proportional to the sparsity of the input data.

[0140] After determining the bucket spacing and number of buckets, the indexes of the data to be merged can be sorted, and then the sorted indexes can be ordered according to the bucket spacing N. spacingSequential partitioning is performed to determine the index range corresponding to each round of water processing, or the index range or index interval of each bucket. This partitioning method ensures that the data to be merged in each round of water processing fills as much of the available space in the storage circuit as possible.

[0141] In some embodiments, the sorting of the indexes of the data to be merged can be performed using the first processing mode of the MERGE instruction described above. Specifically, the merge sort mode of the merging instruction is invoked to sort the indexes of the multiple data to be merged. It is understood that other sorting methods can also be used to pre-sort the indexes, thereby dividing the index ranges of the various buckets mentioned above.

[0142] After determining the index range for each round of pipeline processing based on the aforementioned method, the indexes and associated data falling within the corresponding index range can be retrieved in each round of pipeline processing and then fusion processing can be performed.

[0143] Figure 14 An exemplary pipeline process for fusion processing according to an embodiment of this disclosure is shown.

[0144] Figure 1410 shows the indexes of 9 channels of data to be merged, with 6 index elements per channel, totaling 54 index elements. Assuming a bucket spacing of 20, 3 rounds of pipelined processing are required. Based on the previous sorted interval partitioning, the index range of the first round of pipelined processing (the first "bucket") is 0–13, with 20 numbers; the index range of the second "bucket" is 14–31, with 20 numbers; and the index range of the third "bucket" is 32–99, with 14 numbers. Those skilled in the art will understand that the interval partitioning is not limited to the above example. When the number of data points in the last bucket is small, the allocation can be appropriately adjusted to maximize overall efficiency.

[0145] In the first round of pipelined processing, as shown in Figure 1420, all indices falling between 0 and 13 and their corresponding data are extracted from the data to be merged. The dark squares in the figure represent the indices extracted in the first round.

[0146] As mentioned earlier, the indexes of the multiple streams of data to be merged are ordered within each stream, for example, arranged in ascending order. Therefore, during each round of data retrieval, each stream can be selected sequentially to identify all indexes falling within the corresponding index range.

[0147] Specifically, in some embodiments, indices falling within the index range corresponding to the current round of pipelined processing and their corresponding data to be fused are selected sequentially from the indices of multiple streams of data to be fused. The number of indices extracted from each stream does not exceed the remaining processable number. Here, the remaining processable number is equal to the number of data points N. spacing The difference between the number of indexes selected and the number of indexes already selected.

[0148] by Figure 14 For example, in each round of pipeline processing, the storage circuit reserves space for 20 (bucket spacing) numbers. Therefore, 20 numbers can be retrieved for each index, and then a size comparison operation (e.g., the `__bang_ge` (greater than) / `__bang_le` (less than) function) is used to select the indexes falling within the 0-13 range. The results are stored in the computation buffer space. In the example shown in the diagram, 9 loads (9 indexes) are required, loading 20 numbers each time, because it's uncertain how many indices in each index are between 0 and 13, but it's certain that there are at most 20.

[0149] In some implementations, the data retrieval steps can be as follows: 20 numbers are retrieved from the first index, and 2 numbers meet the requirements; then 20-2=18 numbers are retrieved from the second index, and 2 numbers meet the requirements; then 20-2-2=16 numbers are retrieved from the third index, and so on. As can be seen from the above steps, since the number of indices retrieved gradually decreases each time, the amount of I / O can be reduced.

[0150] Next, in each round of pipeline processing, as shown in 1430, the MERGE instruction is executed to merge the retrieved index and its corresponding data to be merged. The merged data can be directly stored in a specified location, such as an external storage circuit, as shown in 1440. It can be understood that since the pipeline processes are also ordered, i.e., the buckets are ordered, the merged data obtained from each round of pipeline processing can be concatenated according to the round order to obtain the final result.

[0151] Furthermore, as described above regarding sparse convolution operations, the multiplication-addition result of the MAC step may be invalid, with its corresponding index set to a fixed value, such as -1. In this case, when executing the MERGE instruction, the hardware can avoid outputting any data when encountering an index of -1, thus preventing invalid processing.

[0152] When an invalid index (e.g., -1) exists in the index of the data to be merged, the bucket spacing can be adjusted appropriately when performing pipelined processing using bucket sort.

[0153] As mentioned earlier, Nmax is a fixed value calculated based on the available space of the storage circuit and the relevant parameters of the fusion processing; that is, the maximum number of indexes that can be processed at one time. If there are many invalid indexes (-1), the bucket distance will decrease. This is because the number of invalid indexes also occupies the number of indexes in Nmax. Therefore, after pre-sorting the indexes of the data to be fused, the number of invalid indexes can be counted, which can be denoted as N. invaildThen, a corresponding buffer space is reserved in the storage circuit. This way, each time the index to be merged is retrieved and stored in the storage circuit, N buffer space must always be reserved in the storage circuit. invaild The number of spaces is used to prevent data overflow.

[0154] From the previous reference Figure 7 As shown in Figure 8, only the data within the four outermost borders (top, bottom, left, and right) of the input data will have an invalid index "-1". Therefore, even with a large input data set, only the data within these four outermost borders will have an invalid index of "-1". This demonstrates that the number of invalid indices will be very small, and reserving a small amount of space is sufficient.

[0155] In practical applications, such as radar algorithms, convolution operations often require padding with zeros around the edges (as mentioned in the previous reference). Figure 9 (As described), at least one layer of zeros will be added to the top, bottom, left, and right edges (e.g., padding amount = 1 or 2). After padding, the four outermost borders of the input data are all 0, thus significantly reducing the number of indices containing "-1". Furthermore, the radar algorithm processes point cloud data. Observations of real data show that point cloud data becomes sparser towards the edges, meaning point cloud objects are generally located in the center of the input image. Therefore, the probability of invalid indices appearing at the borders is low, and reserving a small amount of space for invalid indices is sufficient.

[0156] The above describes a pipelined scheme for data fusion processing in conjunction with the data processing circuitry of this disclosure embodiment. Figure 15 An exemplary flowchart of a data processing method implemented by a data processing circuit according to an embodiment of this disclosure is shown. In this data processing method, a control circuit parses a fusion instruction and controls a storage circuit and a processing circuit to perform fusion processing on multiple streams of data to be fused according to the fusion instruction. Under the control of the control circuit, the processing circuit groups data elements from the multiple streams of data to be fused from the storage circuit into a single ordered fused data stream according to their corresponding indices, wherein data elements with the same index are merged into a single fused data element. The data element may include any of scalar, vector, or higher-dimensional data.

[0157] The storage circuitry is configured with at least two buffers to support simultaneous data access between one buffer and external storage circuitry, and between the other buffer and processing circuitry. The buffers are used to store information, including pre-processed and / or post-processed information.

[0158] More specifically, in step 1510, the control circuit parses the fusion instruction.

[0159] Next, in step 1520, the control circuit can allocate corresponding buffer space in the storage circuit according to the parsed fusion instructions. Specifically, the control circuit can determine the maximum number of indices Nmax that can be processed in one fusion process based on the available space size of the storage circuit and the relevant parameters in the fusion instructions. Then, according to Nmax, buffer space is allocated in each buffer of the storage circuit for each relevant parameter of the fusion process.

[0160] In some embodiments, the execution of fusion instructions can be implemented through multi-round pipelined processing to address the problem of insufficient available space in the storage circuitry. Each round of pipelined processing can be implemented using a load-compute-store pipelined approach.

[0161] Specifically, in step 1531, the indices and their data that fall within the index range corresponding to the current round of pipeline processing are loaded. The method for selecting the index is described above and will not be repeated here.

[0162] Next, in step 1532, the MERGE instruction is executed to merge the loaded index and data, resulting in merged data.

[0163] Finally, in step 1533, the fused data is stored back to the designated location, thus completing this round of pipeline processing.

[0164] In step 1540, it is determined whether the data has been processed. If not, the process returns to step 1531 to execute the next round of processing. If the data has been processed, the process proceeds to step 1550 to end the processing.

[0165] It is understandable that although the flow processing is shown as sequential in the diagram, in actual execution, different steps of multiple flow processing rounds can be performed in parallel. For example, the calculation step of the first flow processing round can be executed simultaneously with the loading step of the second flow processing round. As another example, the storage step of the first flow processing round can be executed simultaneously with the calculation step of the second flow processing round. Those skilled in the art can reasonably design parallel flow processes based on the principles described herein to improve processing efficiency.

[0166] Those skilled in the art will understand that the description of the data fusion processing of the embodiments of this disclosure described above in conjunction with the accompanying drawings can also be applied to... Figure 15 The data processing methods are already described, so they will not be repeated here.

[0167] This disclosure also provides a chip that may include the data processing circuitry of any of the embodiments described above in conjunction with the accompanying drawings. Furthermore, this disclosure also provides a board that may include the aforementioned chip.

[0168] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0169] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0170] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0171] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0172] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0173] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A data processing circuit, comprising a control circuit, a storage circuit, and an arithmetic circuit, wherein: The control circuit is used to parse the fusion instruction and control the storage circuit and the arithmetic circuit to perform fusion processing on multiple data to be fused according to the fusion instruction; The storage circuit is configured with at least two buffers to support data access between one buffer and an external storage circuit while simultaneously performing data access between the other buffer and the arithmetic circuit. The buffers are used to store information, including information before and / or after processing. as well as The arithmetic circuit, under the control of the control circuit, merges data elements from multiple channels of data to be merged from the storage circuit into a single ordered merged data according to their corresponding indices. The multiple channels of data to be merged include K channels of data and corresponding K-channel indices. The index elements in the K-channel indices indicate the index information of the corresponding data elements in the K channels of data. The index elements of each channel in the K-channel indices are arranged in an ordered manner, and the data elements of each channel of data in the K channels are arranged in an ordered manner according to the order of their corresponding indices. The merged data includes a fusion index and a fusion data. The fusion index elements in the fusion index are arranged in an ordered manner, and the fusion data elements in the fusion data correspond one-to-one with the fusion index elements. Identical index elements are merged into the same fusion index element, and data elements with the same index are accumulated and merged into a single fusion data element. The data elements include any of the following: scalar, vector, or higher-dimensional data.

2. The data processing circuit according to claim 1, wherein, The control circuit is further used for: Based on the available space of the storage circuit and the relevant parameters of the fusion process, determine the maximum number of indexes Nmax that can be processed in a single fusion process; and Based on the Nmax, buffer space is allocated in each of the buffers for the various relevant parameters of the fusion process.

3. The data processing circuit according to claim 2, wherein the relevant parameters of the fusion processing include at least one of the following: The size of the multiple streams of data to be merged; The numerical values ​​of the multiple streams of data to be fused; The index of the multi-path data to be merged; The numerical value of the fused data; The index of the fused data; The operational structure representing the fused data elements; and Data to be merged in each round of fusion processing.

4. The data processing circuit according to any one of claims 1-3, wherein the control circuit is further configured to control the storage circuit and the arithmetic circuit to perform fusion processing on the multiple streams of data to be fused in a load-compute-store (LCS) pipelined manner, wherein in each round of pipelined processing: The control circuit is used to select indices falling within the corresponding index range from the indices of the multiple data streams to be fused, and load them into the computation buffer space of the storage circuit; and The computing circuit is used to perform the fusion process on the index in the computing buffer space and the data to be fused corresponding to the index; and to store the fused data back to the specified location.

5. The data processing circuit according to claim 4, wherein the control circuit is further configured to: Based on the size of the buffer space allocated in the buffer for the values ​​of the fused data, determine the number N of data that can be processed in each round of flow processing. spacing ; Sort the indexes of the multiple streams of data to be merged; as well as The sorted indexes are arranged according to the N... spacing Perform sequential partitioning to determine the index range corresponding to each round of water treatment.

6. The data processing circuit according to claim 5, wherein the control circuit is further configured to: sequentially select from the indices of the multiple streams of data to be fused an index that falls within the index range corresponding to the current round of processing and the corresponding data to be fused, wherein the number of indices selected from each stream does not exceed the remaining processable number, and the remaining processable number is equal to the number of data N. spacing The difference between the number of indexes selected and the number of indexes already selected.

7. The data processing circuit according to any one of claims 5-6, wherein the control circuit is further configured to: Count the number of invalid indexes in the indexes of the multi-path data to be merged; and Based on the number of invalid indexes, a corresponding buffer space is reserved in the storage circuit.

8. The data processing circuit according to any one of claims 5-6, wherein the control circuit is further configured to: invoke the merge sorting mode of the fusion instruction to sort the indexes of the multi-channel data to be fused.

9. The data processing circuit according to any one of claims 5-6, wherein the number of rounds of pipeline processing is determined based on the total number of data to be merged from the multiple streams and the number of data that can be processed in each round of pipeline processing.

10. The data processing circuit according to any one of claims 5-6, wherein the control circuit is further configured to: splice the fused data obtained from each round of water treatment in the order of rounds to obtain the final result.

11. A chip comprising a data processing circuit according to any one of claims 1-10.

12. A circuit board comprising the chip according to claim 11.

13. A method for processing data using the data processing circuit according to any one of claims 1-10.

Images