Data processing circuits, data processing methods and related products
By performing operations only on non-zero data and convolution kernels in the data processing circuit, and using the MERGE instruction to merge and accumulate the results of multiple operations, the problem of low efficiency of convolutional neural networks in processing sparse LiDAR point cloud data is solved, and efficient sparse convolution processing is achieved.
Patent Information
- Application Number
- CN202110482878.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-30
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2041-04-30
AI Technical Summary
Existing convolutional neural networks are inefficient when processing sparse LiDAR point cloud data, especially wasting a lot of computing resources during convolution operations.
A data processing circuit and method are provided, which performs N-dimensional convolution operations through control circuit, storage circuit and arithmetic circuit. The operation is performed only on non-zero data and convolution kernel, and the MERGE instruction is used to merge and accumulate the results of multiple operations. It is suitable for convolution processing of sparse data.
It greatly saves computational resources, improves processing efficiency, and is suitable for multidimensional convolution operations, including two-dimensional and three-dimensional convolution, especially for processing LiDAR point cloud data.
Smart Images

Figure CN115221106B_ABST
Abstract
Description
Technical Field
[0001] This disclosure generally relates to the field of data processing. More specifically, this disclosure relates to data processing circuits, data processing methods, chips, and circuit boards. Background Technology
[0002] In recent years, significant progress has been made in object detection, instance segmentation, and keypoint detection based on convolutional neural networks. These detection methods are typically based on LiDAR or RGB-D data and can be applied to fields such as autonomous driving and robot vision.
[0003] Unlike dense image data, LiDAR point cloud data is typically sparse, and the point density varies dramatically due to factors such as non-uniform sampling in 3D space, the effective range of the sensor, occlusion, and relative pose. Therefore, traditional convolutional neural networks, which are well-suited for dense data, become very inefficient when applied to this type of sparse data, especially when convolution operations are involved, wasting significant computational resources on zero-value data points.
[0004] Therefore, it is desirable to provide an improved convolution scheme that is suitable for sparse data such as point cloud data, thereby improving processing efficiency. Summary of the Invention
[0005] In order to at least partially solve one or more of the technical problems mentioned in the background art, the present disclosure provides a data processing circuit, a data processing method, a chip, and a board.
[0006] In a first aspect, this disclosure discloses a data processing circuit, comprising: a control circuit, a storage circuit, and a computation circuit, wherein: the control circuit is configured to control the storage circuit and the computation circuit to perform N-dimensional convolution operations on input data and a convolution kernel, where N>1, and N represents the number of convolution dimensions in the convolution operation where sliding accumulation is performed, wherein the input data is sparse data, and each input data element has an index and numerical information; the storage circuit is configured to store information, the information including at least pre-processing and / or post-processing information; and the computation circuit is configured to, under the control of the control circuit, perform operations on the input data elements and the convolution kernel to obtain multiple operation results and corresponding indices; and to merge the multiple operation results into a single fused data stream according to their corresponding indices as the result of the convolution operation, wherein operation results with the same index are accumulated.
[0007] In a second aspect, this disclosure provides a chip that includes the data processing circuitry of any of the embodiments of the first aspect.
[0008] In a third aspect, this disclosure provides a board including the chip of any of the embodiments of the second aspect above.
[0009] In a fourth aspect, this disclosure provides a method for processing data using the aforementioned data processing circuitry.
[0010] Through the data processing circuit, the method for processing data using the data processing circuit, the chip, and the board provided above, this disclosure embodiment offers a convolution scheme suitable for sparse data. By only performing operations with non-zero / non-empty data and the convolution kernel, it can significantly reduce computational load and improve processing efficiency. Furthermore, when accumulating the results of the operation between non-zero data and the convolution kernel, a dedicated hardware instruction `MERGE` is used to perform merge sorting and accumulation processing, which can accelerate data processing. In addition, the sparse convolution scheme provided in this disclosure embodiment is applicable to multi-dimensional convolution operations, including but not limited to two-dimensional and three-dimensional convolution, and therefore can be applied to the processing of LiDAR point cloud data. Attached Figure Description
[0011] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding portions wherein:
[0012] Figure 1 This diagram shows the structure of the board card according to an embodiment of this disclosure;
[0013] Figure 2 This diagram illustrates the structure of the combined processing apparatus according to an embodiment of the present disclosure.
[0014] Figure 3 A schematic diagram illustrating the internal structure of a processor core in a single-core or multi-core computing device according to embodiments of the present disclosure;
[0015] Figure 4 This illustrates the operational principle of a conventional convolution scheme;
[0016] Figure 5 An exemplary schematic diagram of the sparse convolution scheme of this disclosure embodiment is shown;
[0017] Figure 6 This diagram illustrates the preprocessing of high-dimensional sparse input data according to an embodiment of this disclosure.
[0018] Figure 7 This illustrates the meaning of the multiplication operation in the embodiments disclosed herein;
[0019] Figures 8A-8C This shows the index mapping relationship between the product result and the convolution operation result;
[0020] Figure 9 This illustrates the impact of padding in convolution operations on the input data index.
[0021] Figure 10 This illustrates an exemplary principle of the MERGE instruction;
[0022] Figure 11 This illustrates an exemplary scheme of invoking the MERGE instruction multiple times in an embodiment of this disclosure;
[0023] Figure 12 An exemplary schematic diagram of a sparse 3D convolution scheme according to an embodiment of this disclosure is shown.
[0024] Figure 13 A schematic diagram of the data processing circuit of this disclosure embodiment is shown; and
[0025] Figure 14 An exemplary flowchart illustrating a data processing method according to an embodiment of this disclosure is shown. Detailed Implementation
[0026] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0027] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this disclosure are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.
[0028] It should also be understood that the terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.
[0029] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."
[0030] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.
[0031] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.
[0032] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.
[0033] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).
[0034] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown...) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a storage device 204.
[0035] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.
[0036] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.
[0037] Processing device 203, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.
[0038] Storage device 204 is used to store data to be processed. It may be DRAM or DDR memory, typically 16G or larger in size, and is used to store data of computing device 201 and / or processing device 203.
[0039] Figure 3The diagram shows the internal structure of the processor core when the computing device 201 is a single-core or multi-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three main modules: a control module 31, an arithmetic module 32, and a storage module 33.
[0040] The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 fetches instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.
[0041] The computation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 322 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.
[0042] Storage module 33 is used to store or move relevant data, including neuron RAM (NRAM) 331, weight RAM (WRAM) 332, and direct memory access (DMA) module 333. NRAM 331 is used to store input neurons, output neurons, and intermediate results after computation; WRAM 332 is used to store the convolution kernels of the deep learning network, i.e., the weights; DMA 333 is connected to DRAM 204 through bus 34 and is responsible for data transfer between computing device 301 and DRAM 204.
[0043] Based on the aforementioned hardware environment, the embodiments disclosed herein provide a data processing circuit that supports convolution operations on sparse data. By providing an optimized convolution scheme, convolution processing related to sparse data such as LiDAR point cloud data can be simplified and accelerated. The sparse convolution scheme provided in the embodiments of this disclosure is applicable to multidimensional convolution operations, including but not limited to two-dimensional convolution and three-dimensional convolution. For simplicity and ease of understanding, two-dimensional convolution is used as an example in some embodiments.
[0044] In this disclosure, "N-dimensional convolution" refers to the number of convolutional dimensions in which sliding accumulation is performed. For example, when N=2, the convolutional kernel performs translational accumulation along two dimensions (e.g., width W and height H) according to the corresponding convolutional stride. When N=3, the convolutional kernel performs translational accumulation along three dimensions (e.g., width W, height H, and depth D) according to the corresponding convolutional stride. When N=4, the convolutional kernel performs translational accumulation along four dimensions (e.g., width W, height H, depth D, and batch) according to the corresponding convolutional stride. In this disclosure, "non-convolutional dimension" refers to a dimension in which the convolutional kernel does not perform sliding accumulation.
[0045] To better understand the convolution scheme of this disclosure embodiment, the operational principle of a conventional convolution scheme will be described first using two-dimensional convolution as an example.
[0046] Figure 4 This illustrates the operational principle of a conventional convolution scheme. In this example, the convolution kernel 410 is dense, a 3×3 matrix, with the numbers in the kernel representing the corresponding weights. The input data 420 is a 6×6 matrix that is sparse, containing only three non-zero values: 2, 3, and 5, as shown by the dark squares. For simplicity, in this exemplary convolution process, the stride for both dimensions is set to 1, with zero padding (no dilation). The 3×3 gray squares in the diagram represent the sliding accumulation process of the convolution kernel across the input data. 430 shows the computation at the start of the convolution, 440 shows the computation one step to the right, and 450 shows the computation one step down. In each step, the weights of the convolution kernel are multiplied bitwise with the input data and accumulated. 460 shows the final computation result as the output data. The output data is a 4×4 matrix. It can be seen that the calculation of 430 corresponds to the data at coordinate (1,1) in the output data, the calculation of 440 corresponds to the data at coordinate (1,2) in the output data, and the calculation of 450 corresponds to the data at coordinate (2,1) in the output data.
[0047] In the sparse convolution operation of the embodiments disclosed herein, the convolution kernel is dense, and its input format can be the same as that of conventional convolution; while the input data is sparse, and its input format can be different from that of conventional convolution input data, thereby saving storage space. In some embodiments, the input data is sparse data, and each input data element has index and numerical information, which can be represented as (index, value). Depending on the application, data type, or scale, the index here can be one-dimensional, two-dimensional, or more-dimensional, and this disclosure is not limited in this respect. Indices of different dimensions can be converted to each other, for example, two-dimensional or more-dimensional indices can be converted to one-dimensional indices according to a predetermined traversal rule, and vice versa. Similarly, depending on the application, data type, or scale, the numerical information here can include scalars, vectors, or higher-dimensional data.
[0048] by Figure 4 Taking the example in the example, the input data has three non-sparse points: ((1,4),2), ((3,3),3), and ((5,6),5). For the first non-sparse point, (1,4) represents the two-dimensional index coordinates, and 2 represents the specific numerical value of the position pointed to by the index, which is a scalar in this example; the meanings of the other non-sparse points are similar.
[0049] from Figure 4 As can be seen from the description, the final result of sparse convolution depends only on the results of operations on non-zero input data elements. Therefore, multiplication and addition operations with the convolution kernel can be performed only on these non-zero input data elements. Furthermore, from... Figure 4 As can be seen from the output data 460, these multiply-accumulate operations can be further broken down into multiplication operations and corresponding positional accumulation operations. Accordingly, in the sparse convolution scheme of this disclosure embodiment, the sparse convolution operation can include three steps: calculating the operation result of the convolution kernel and the sparse input data (e.g., the product result or the multiply-accumulate result, considering that some dimensions of the data also need to be accumulated, such as the input channel dimension Ci described later); determining the index of each operation result; and merging the multi-way operation results into one fused data in index order as the result of the sparse convolution operation. During the merging process, operation results with the same index are accumulated. In the following description, depending on the dimension of the data, in scenarios where the input channel dimension Ci is not mentioned, the multi-way operation result is sometimes directly referred to as the multi-way multiplication result. Those skilled in the art can understand its corresponding meaning based on the context.
[0050] Figure 5 An exemplary principle of a sparse convolution scheme according to an embodiment of this disclosure is shown. Figure 5 Still with Figure 4 The sparse convolution operation scheme implemented in this disclosure will be described using data as an example.
[0051] like Figure 5 As shown, in the MAC step, the input data element 520 is multiplied with the convolution kernel 510 to obtain a multi-way product result 530. When calculating the product result of the convolution kernel and the sparse input data, the weight data of each convolution dimension of the convolution kernel can be merged into one dimension, with the dimension size being offset, where offset is the total number of weight data in the convolution dimension.
[0052] As mentioned above, the sparse convolution scheme provided in this disclosure embodiment can be applied to multidimensional convolution operations, including but not limited to two-dimensional convolution and three-dimensional convolution. Figure 5 The diagram illustrates merging the weight data from the two convolutional dimensions (width W and height H) of a two-dimensional convolutional kernel into a single dimension. For example, the weight data from a 3×3 matrix can be expanded into a column containing nine weight data points, which can also be referred to as nine weight scalars.
[0053] It's understandable that when the convolution kernel is a three-dimensional kernel, the weight data across its three convolutional dimensions (width W, height H, and depth D) can be merged into a single dimension. For example, the 27 weight data from a 3×3×3 cubic convolution kernel can be unfolded into a single column, resulting in 27 weight scalars. Other cases can be deduced similarly. If the convolution kernel also has non-convolutional dimensions, such as the input channel Ci dimension and / or the output channel Co dimension, these dimensions can be retained since convolution operations are not performed on them. In this case, for example, if the convolution kernel includes a 3×3 convolutional dimension and also a non-convolutional dimension Ci = 2, then after merging these dimensions, the convolution kernel becomes 9×2, which can be viewed as nine weight vectors of length 2. For example, if the convolution kernel includes a 3×3×3 convolution dimension, as well as non-convolution dimensions Ci=2 and Co=4, then after merging the above dimensions, the convolution kernel becomes 27×2×4, which can be regarded as 4 sets of weights on the Co dimension, each set including 27 weight vectors of length 2.
[0054] The input data to be processed by convolution can include multidimensional data, and it is sparse across multiple dimensions. For example, in target detection based on LiDAR data, the input data is detection data in three-dimensional space, which may represent the grayscale value, RGB values, signal strength, etc., of each three-dimensional spatial coordinate point. Therefore, depending on the information content to be represented, the input data element at each coordinate point can be one-dimensional, two-dimensional, three-dimensional, or higher-dimensional data. Due to the characteristics of point cloud data, coordinate points with non-zero value data elements are sparse, that is, they are sparse in three spatial dimensions (e.g., width W, height H, and depth D).
[0055] Depending on the initial state of the input data, preprocessing can be performed before the sparse input data is provided to the computation circuit for processing. In some embodiments, such preprocessing may include, for example, merging multiple sparse dimensions into one dimension; densifying sparse data points in the input data along the merged dimension to form input data elements; and representing each input data element using indexes and numerical information. The index can be a one-dimensional or multi-dimensional index, and the numerical information can include any of scalar, vector, or higher-dimensional data.
[0056] In one example, for instance, reference Figure 4 The input data is a two-dimensional 6×6 matrix, sparse in both width (W) and height (H). During preprocessing, W and H are merged into a single dimension, and the sparse data points (2, 3, and 5 in this example) are compacted along this merged dimension, resulting in three compacted input data elements. Each input data element is then represented using indexes and numerical information. The index of a data element indicates its position within the sparse input data before compaction. For example, Figure 4 The indices of the three input data elements in the example are (1,4), (3,3), and (5,6). These indices are two-dimensional, but can be converted to one-dimensional indices, such as 4, 15, and 29. The numerical values of these three data elements are 2, 3, and 5, representing three scalars.
[0057] In another example, for example, reference Figure 6This diagram illustrates the preprocessing of high-dimensional sparse input data according to embodiments of this disclosure. As shown, the sparse input data 610 comprises five dimensions: a batch (B) dimension, a three-dimensional HWD space dimension, and an input channel Ci dimension. The input data is sparse in the B dimension and the HWD three-dimensional space. In the diagram, dark squares in the HWD matrix represent numerical values, while other parts are all zero values. Multiple such HWD matrices exist in the B dimension, and the sparsity pattern (i.e., the position of the dark squares) on each matrix can be different. The input data is dense in the Ci dimension, which is the lowest dimension. Due to the limited representation capabilities of the diagram, only four dimensions are shown in 610, but the Ci dimension can be understood as the thickness of each dark square. The size of the Ci dimension is uniform, meaning the thickness of each dark square is the same. During preprocessing, the four sparse dimensions of the input data (B dimension and HWD three-dimensional space dimension) can be merged into a single dimension Ni, and the sparse data points (dark squares in the diagram) can be densified in the merged dimension to form dense input data elements. That is, each HWD 3D matrix in the B dimension undergoes the same dimension merging and compaction process to obtain the preprocessed compact form of the input data 620, which is a two-dimensional matrix with Ci in the low dimension and Ni in the high dimension of the BHWD merging dimension.
[0058] Next, index and numerical information are used to represent each densified input data element. Figure 6 The indices of the input data elements in the example can be represented using their coordinates in the uncompacted BHWD four-dimensional space, or they can be converted into one-dimensional indices. The numerical information of each input data element can be viewed as a Ci vector.
[0059] The input data, after the above preprocessing, can be provided to the arithmetic circuit for further processing.
[0060] The input data elements can be arranged into an input vector to perform multiplication with the convolution kernel. In some embodiments, the input data elements can be arranged in their index order (e.g., ascending order) to accommodate subsequent merging. Each vector element in the input vector comprises one input data element. As described above, each input data element can be a scalar, vector, or higher-dimensional data.
[0061] Back Figure 5 Next, a multiplication operation can be performed on the convolutional dimension of the convolutional kernel 510 after dimension merging and the input vector 520 composed of input data elements, resulting in offset data 530, where each data path includes several product results, and each product result includes any of the following: scalar, vector, or higher-dimensional data. Figure 5In the example, the dimension-merged convolution kernel 510 (the nine scalars in the diagram) performs scalar vector multiplication with the input vector 520 (a vector of length 3 consisting of 2, 3, and 5), resulting in nine vectors, or nine-way product results 530. In this example, each product result is a scalar.
[0062] The MAC step is equivalent to performing a 1×1 point-level convolution on the input vector for each row of data in the dimension-merged convolution kernel, resulting in a convolution result. To better understand the meaning of the multiplication operation in the MAC step above, Figure 7 The meaning of several of these multiplication operations is illustrated by example.
[0063] As shown in the figure, for Figure 5 The operation on the first weight scalar in the diagram: 1*[2,3,5]=[2,3,5], can be understood as corresponding to the operations at positions 710, 720, and 730 in the diagram, respectively, that is, the product results generated when 2, 3, and 5 are located at the first position of the convolution kernel. Similarly, for... Figure 5 The operation of the second weight scalar in the diagram: 2*[2,3,5]=[4,6,10], can be understood as the operation corresponding to 740, 750 and 760 in the diagram, that is, the product result when 2, 3 and 5 are located at the second position of the convolution kernel.
[0064] As mentioned earlier, input data can also include non-convolutional dimensions, such as those described in the previous reference. Figure 6 The input channel Ci dimension is described. In some embodiments, the input data may include N convolutional dimensions and at least one non-convolutional dimension, and correspondingly, the convolutional kernel may also include N convolutional dimensions and at least one non-convolutional dimension. In this case, in the above MAC step, the corresponding operation processing can be performed on the non-convolutional dimensions according to the specific operation requirements. For example, in some examples, the non-convolutional dimensions of the input data include the input channel Ci dimension, and the non-convolutional dimensions of the convolutional kernel include the input channel Ci dimension and the output channel Co dimension. The operation processing on these non-convolutional dimensions may include, but is not limited to: keeping the results of the Ci dimension from being accumulated (e.g., in depthwise convolution); performing positional accumulation operation on the multiplication results of the convolutional dimensions on the Ci dimension (e.g., accumulation on the Ci dimension); and / or stacking the accumulation results on the Ci dimension on the Co dimension (e.g., accumulation on the Ci dimension, but not on the Co dimension) to obtain the multi-way operation result, where each operation result is a vector on the Co dimension.
[0065] continue Figure 5The MAC step obtains the operation results related to non-zero values. To obtain the final convolution operation result in the subsequent MERGE step, it is necessary to determine the indices of these operation results for corresponding accumulation. Therefore, the corresponding indices of these operation results can be obtained in the INDEX step. Specifically, in some embodiments, the index of each operation result in the multiplication results is determined according to the index mapping relationship. Each operation result is obtained by multiplying or multiplying and adding the weight data in the convolution kernel with the input data elements. Therefore, the index mapping relationship indicates the relationship between the position of the weight data, the index of the input data element, and the corresponding result element in the convolution operation result. In other words, the index of the operation result obtained by multiplying or multiplying and adding the weight data can be determined based on the position of the weight data in the convolution kernel and the index of the input data element.
[0066] Figure 8A This example illustrates the index mapping relationship between partial product results and convolution operation results in the MAC step. The meanings of 810A, 820A, and 830A in the figure are... Figure 7 Similar to 710, 720, and 730, these represent the product operation of the first weight scalar and the input vector: 1*[2,3,5]=[2,3,5]. The arrows in the diagram indicate the corresponding positions of each product result (810A, 820A, and 830A) in the convolution result (840A). Specifically, the product result of 810A corresponds to position (1,4) in the 4×4 convolution result, the product result of 820A corresponds to position (3,4) in the convolution result, and the product result of 830A overflows the convolution result range and is therefore invalid.
[0067] from Figure 8A It can be seen that when the input vector is multiplied with the first weight data of the convolution kernel, the index has the following mapping relationship: Assuming that the index of the vector element (i.e., data points 2, 3 and 5) in the input vector is (x, y), then the index of the product result of the product operation with the first weight data is also (x, y).
[0068] Figure 8B This example illustrates the index mapping between partial product results and convolution operation results in the MAC step. In the figure, 810B, 820B, and 830B represent the product operation of the second weight scalar and the input vector: 2*[2,3,5]=[4,6,10]. The arrows in the figure indicate the corresponding positions of each product result (810B, 820B, and 830B) in the convolution operation result (840B). Specifically, the product result of 810B corresponds to position (1,3) in the 4×4 convolution result, the product result of 820B corresponds to position (3,2) in the convolution result, and the product result of 830B overflows the range of the convolution result and is an invalid result.
[0069] from Figure 8B It can be seen that when the input vector is multiplied with the second weight data of the convolution kernel, the index has the following mapping relationship: Assuming that the index of the vector element (i.e., data points 2, 3 and 5) in the input vector is (x, y), then the index of the product result of the product operation with the second weight data is (x, y-1).
[0070] Figure 8C This example illustrates the index mapping between partial product results and convolution operation results in the MAC step. In the figure, 810C, 820C, and 830C represent the product operation of the 9th weight scalar with the input vector: 1*[2,3,5]=[2,3,5]. The arrows in the figure indicate the corresponding positions of each product result (810C, 820C, and 830C) in the convolution operation result (840C). Specifically, the product result of 810C overflows the convolution result range and is invalid; the product result of 820C corresponds to position (1,1) in the 4×4 convolution result, and the product result of 830C corresponds to position (3,4) in the convolution result.
[0071] from Figure 8C It can be seen that when the input vector is multiplied by the 9th weight data of the convolution kernel, the index has the following mapping relationship: Assuming that the index of the vector element (i.e., data points 2, 3 and 5) in the input vector is (x, y), then the index of the product result of the product operation with the 9th weight data is (x-2, y-2).
[0072] comprehensive Figures 8A-8C As can be seen, each vector element in the input vector (i.e., data points 2, 3, and 5) sequentially traverses the 3×3 convolution kernel. Therefore, the offset of each data point relative to the center point of the convolution kernel (here, the center of the 3×3 convolution kernel is the 5th weight data) is fixed. Based on this characteristic, the index of the center point of the convolution kernel can be obtained sequentially based on the specific location of the data points. Then, the index of the center point can be mapped to the index of the output point. Thus, the index of the product result associated with each vector element in the input vector can be directly determined based on the index of that vector element. That is, knowing only the index of the input data element allows us to determine the index of the product result of that input data element multiplied by all the weight data.
[0073] For example, for a 3×3 two-dimensional convolution kernel, the coordinate offset relative to the center point of this two-dimensional convolution kernel is a constant when data points traverse the kernel. In this example, these 9 coordinate offsets can be constructed as follows:
[0074] (1,1),(0,1),(-1,1);
[0075] (1,0),(0,0),(-1,0);
[0076] (1,-1),(0,-1),(-1,-1).
[0077] For example, for a 3×3×3 convolution kernel, the coordinate offset relative to the center point of this 3D convolution kernel is a constant when data points traverse the kernel. In this example, these 27 coordinate offsets can be constructed as follows:
[0078] (1,1,1),(0,1,1),(-1,1,1),(1,0,1),(0,0,1),(-1,0,1),(1,-1,1),(0,-1,1),(-1,-1,1);(1,1,0),(0,1,0),(-1,1,0),(1,0,0),(0,0,0),(-1,0,0),(1,-1,0),(0,-1,0),(-1,-1,0);(1,1,-1),(0,1,-1),(-1,1,-1),(1,0,-1),(0,0,-1),(-1,0,-1),(1,-1,-1),(0,-1,-1),(-1,-1,-1).
[0079] Therefore, based on the indices of the input data points and the aforementioned fixed coordinate offsets, the indices of the convolution kernel center points corresponding to each convolution kernel traversal can be sequentially calculated. Then, mapping the center point indices to the output point indices determines the index of each product result generated by the input data points.
[0080] from Figures 8A-8C The diagram also shows that some product results have overflowed the range of the convolution results and are therefore invalid. For these cases, the indexes exceeding the range of the convolution results (i.e., the range of the output data dimensions) can be set to a predetermined value, such as -1, to identify these invalid results in subsequent processing and prevent them from being merged.
[0081] In some embodiments, convolution operations involve padding. For example, in object detection algorithms based on LiDAR data, same padding is required, meaning that padding ensures the shape of the input data is the same as the shape of the output data after the convolution operation. It is understandable that different padding rules may exist in other applications of convolution operations.
[0082] Figure 9 The effect of padding in convolution operations on the input data index is shown.
[0083] As shown in the figure, matrix 910 represents the initial input data, and matrix 920 represents the padded input data. The initial input data is, for example, a 2×3 matrix. The padded input data is processed according to the padded rules, by adding one column to the left, three columns to the right, four rows to the top, and one row to the bottom of the initial input data. The padded data can be, for example, zero.
[0084] For any data point (x, y) in the initial input data, its coordinates in the input data after padding become (x + pad_left, y + pad_top), where pad_left is the padding amount on the left and pad_top is the padding amount on the top. Therefore, the indices of the input data can be adjusted using simple addition operations and according to the padding rules.
[0085] In this embodiment, the indices of the input data elements can be adjusted based on the padding rules of the convolution operation before determining the index of the product result according to the index mapping relationship.
[0086] Those skilled in the art will understand that index adjustment processing can also be performed after or during index mapping, as long as the impact of the filling rules is taken into account, and the embodiments disclosed herein are not limited in this respect.
[0087] Back Figure 5 Figure 540 shows the indices corresponding to the 9-way product results determined by the INDEX step, with each product result having a corresponding index. Dark squares indicate invalid results, whose indices are set to -1.
[0088] After obtaining the multi-way product results through the MAC step and determining the index of each product result through the INDEX step, these multi-way product results can be further fused to obtain the convolution operation result.
[0089] Specifically, in the MERGE step, the multiplication results are merged and combined according to their index order to obtain the convolution operation result, where multiplication results with the same index are accumulated. Figure 550 shows the merged indexes, where duplicate indices, such as indices 2 and 3, have been removed. Figure 560 shows the merged data, where multiplication results with the same index are accumulated; for example, the data corresponding to two indices 2, 6 and 12, are accumulated, and the data corresponding to two indices 3, 4 and 3, are accumulated.
[0090] By comparison Figure 4 and Figure 5 The calculation results show that, based on Figure 5 The fused index 550 and fused data 560 can recover the sparse form of the convolution operation result, which completely corresponds to Figure 4 The result of a regular 4×4 convolution operation is 460.
[0091] In some embodiments, the MERGE step can be implemented using a hardware instruction called MERGE. The basic function of the MERGE instruction is to merge multiple streams of data to be merged into a single stream of merged data according to their index order, and to accumulate data with the same index.
[0092] Figure 10 The diagram illustrates the exemplary principle of the MERGE command. The figure exemplifies four streams of data to be fused, each stream comprising six data elements. Data elements can be scalars, vectors, or higher-dimensional tensors. In the figure, data elements are exemplarily shown as vectors, such as D11, D12, ..., D46. These vectors have a uniform length; for example, D11 is (d1, d2, d3, ..., dn) with a length of n. Each data element has an associated index indicating its position within the corresponding stream of data. For example, the original stream of data might contain 1000 data elements, but only some of these elements are valid. In this case, these valid elements can be extracted to form the data to be fused, and their corresponding indices can be extracted to indicate their positions in the original data; these indices form the fusion index.
[0093] The diagram schematically illustrates the four indices to be merged, with each index corresponding to one channel of data to be merged. The first index identifies the position of each data element in the first channel, the second index identifies the position of each data element in the second channel, and so on. Furthermore, the index elements in each channel are stored in an ordered manner and correspond one-to-one with the data elements in the corresponding channel. In the example shown, the index elements in each channel are arranged in a first order (e.g., ascending order), and the data elements in each channel are also arranged in the order of their corresponding indices. For example, the first index element in the first channel indicates that the index of the first data element in the first channel is 0, i.e., the first element; the second index element in the first channel indicates that the index of the second data element in the first channel is 2, i.e., the third element; and so on.
[0094] After data fusion, the four data streams are merged into a single ordered fused data stream according to their corresponding indices, with data elements having the same index being combined into a single fused data element. As shown in the figure, the fused index comprises 16 index elements, arranged in a second order (e.g., ascending order), with duplicate index elements removed from the indices to be fused, as indicated by the dark squares in the figure. Correspondingly, the fused data also comprises 16 data elements, arranged in the order of their corresponding indices, with data elements having the same index being combined, as indicated by the dark squares in the figure. Since data elements may be vectors or higher-dimensional tensors, in some embodiments disclosed herein, at least for the merging of data elements with the same index, the form of operational structure elements can be used.
[0095] In this example, the merging of data elements with the same index is illustrated using an addition formula. For instance, for the merged index element "0", its corresponding merged data element is the sum of the first data elements of each channel (D11+D21+D31+D41). As another example, for the merged index element "9", its corresponding merged data element is the sum of the 5th data element of channel 1 and the 3rd data element of channel 4 (D15+D43). When the data elements are vectors, the merged data element is the sum of the corresponding vectors. Depending on the specific settings of the merging instruction, the merged data element can have different representations.
[0096] Those skilled in the art will understand that the first order and the second order mentioned above may be the same or different, and both may be selected from either: an ascending order or a descending order. Those skilled in the art will also understand that although the data streams shown in the diagram have an equal number of data elements, the number of data elements in each data stream may be the same or different, and this disclosure is not limited in this respect.
[0097] As described above, the result of a MAC step may be invalid, with its corresponding index set to a fixed value, such as -1. In this case, when executing the MERGE instruction, the hardware can avoid outputting any data when encountering an index of -1, thus preventing invalid processing.
[0098] The MERGE instruction is a hardware instruction, therefore its processing power is limited by the hardware configuration. For example, the MERGE instruction has an upper limit K for merging data, where K > 1. When the offset of the number of paths in the data to be merged exceeds K, it needs to be completed by calling the MERGE instruction multiple times.
[0099] In some embodiments, the offset-path operation results to be merged can first be split into M groups according to the processing upper limit K, where M = ceil(offset / K). This splitting ensures that the number of paths to be merged in each group will not exceed the processing upper limit K. Then, M MERGE instructions are executed on the split M groups of operation results to obtain M paths of one-time fused data.
[0100] Continue fusing these M streams of data from the first fusion. During this continued fusing process, if M does not exceed K, execute the MERGE instruction once more on these M streams of data from the first fusion to obtain the final result. Similarly, during this continued fusing process, if M exceeds K, M can be split again based on K, and the aforementioned steps can be repeated until the number of streams to be merged does not exceed K.
[0101] Furthermore, since the MERGE instruction cannot perform in-situ operations, a temporary buffer is required. In the case of multiple MERGE calls, the output data of the first MERGE instruction execution can be placed in the temporary buffer, the result of the second MERGE instruction execution can be placed in the input data storage space of the first MERGE instruction, and so on, with the output result of each subsequent MERGE instruction placed in the input data storage space of the previous MERGE instruction.
[0102] Figure 11 An exemplary scheme for calling the MERGE instruction multiple times in an embodiment of this disclosure is illustrated. In this example, it is assumed that the convolution kernel is a 7×7 two-dimensional convolution, so the number of data paths to be merged is offset = 49, and the merging limit of the MERGE instruction is K = 32. As described above, the 49 data paths can be split into M = ceil(49 / 32) = 2 groups, one group containing 32 data paths and the other group containing 17 data paths.
[0103] Next, the MERGE instruction is called on each of these two sets of data to obtain two primary fused data streams, namely MERGE_1 and MERGE_2 in the diagram. At this point, only one more MERGE instruction needs to be called to merge these two primary fused data streams into the final result MERGE.
[0104] It's understandable that other calling schemes may exist. For example, in one implementation, for Figure 11 In the example, the 49 channels of data to be merged can still be split into two groups: the first group contains 32 channels of data, and the second group contains 17 channels of data. However, unlike... Figure 11 The calling scheme can first call the MERGE instruction once on the first group of 32 data channels to obtain one channel of fused data, and then call the MERGE instruction again on this channel of fused data and the second group of 17 data channels to obtain the final fused result. This calling scheme is superior to... Figure 11The calling scheme can reduce one call to the MERGE instruction.
[0105] In another implementation, when splitting the data to be merged, it can be split roughly evenly, so that the number of data paths merged in each MERGE instruction call is relatively similar, thus minimizing the performance difference of the MERGE instruction call. For example, for Figure 11 For example, 49 channels of data to be merged can be split into two groups: a first group of 25 channels and a second group of 24 channels. Then, the MERGE instruction is called on each of these two groups separately, and the data from the first merge is then merged again to obtain the final result. Similarly, in another implementation, the MERGE instruction can be called first on the first group of 25 channels to merge, and then the MERGE instruction can be called again on the merged result and the second group of 24 channels. Thus, only two MERGE instructions need to be called, each merging 25 channels of data.
[0106] When selecting a calling scheme, the following factors should be considered: number of calls, processing time, parallel efficiency, caching requirements, and storage space management difficulty.
[0107] The above description and illustrations primarily focus on two-dimensional convolution, while some descriptions (e.g., preprocessing of input data) also address three-dimensional or higher-dimensional convolution. To better understand the case of three-dimensional convolution, the following, in conjunction with the accompanying drawings, more clearly illustrates the application of the sparse convolution scheme of the present disclosure embodiments to three-dimensional convolution.
[0108] Figure 12 An exemplary schematic diagram of a sparse 3D convolution scheme according to an embodiment of this disclosure is shown. Figure 5 Similarly, this sparse 3D convolution scheme also includes three steps: the MAC step to calculate the result, the INDEX step to determine the index of the result, and the MERGE step to merge the results from multiple paths. The following focuses only on the differences from the sparse 2D convolution scheme.
[0109] As shown in the figure, the weight data of each convolutional dimension of kernel 1210 can be merged into one dimension, with the dimension size being offset, which is the total number of weight data in the convolutional dimension. Although the figure shows an example with an offset of 9, it can be understood that this is formed by merging the three convolutional dimensions of the kernel.
[0110] In this example, the convolution kernel 1210 also has a non-convolution dimension, an input channel Ci dimension, and an output channel Co dimension. In the convolution operation of this example, the results of the operation on the Ci dimension need to be accumulated, while the results of the operation on the Co dimension are not accumulated. For the sake of simplicity, only the Ci dimension is indicated schematically with arrows. Since the accumulation is not performed on the Co dimension, the weight data is stacked on the Co dimension.
[0111] The sparse input data to be convolved can be preprocessed to transform it into a dense input data. Specific preprocessing operations can be found in the previous description. The figure simplifies the dense input data 1220, with arrows indicating the Ci dimension.
[0112] After the aforementioned MAC processing, the multi-path operation result 1230 is obtained. It can be understood that in this MAC processing, in addition to performing multiplication operations on the convolution dimension, accumulation is also performed on the Ci dimension. Since no operations are performed on the Co dimension, each operation result in the obtained multi-path operation result 1230 can be viewed as a vector in the Co direction.
[0113] As described above regarding the INDEX step, after determining the fixed offset based on the convolution kernel, the index of the computation result is only related to the index of the input data. Since the input data does not have a Co dimension, the indices of each computation result can be reused along the Co direction. As shown in the figure, only one output index, 1240, needs to be calculated.
[0114] Next, in the MERGE step, the results of the offset path operations are also merged, but this time the merging is performed on vectors in the Co direction. As shown by the shaded area in Figure 1230, the result of this operation is a vector of length 4 in the Co direction. During the merging process, results with the same index from each path are accumulated; in this example, vectors are accumulated. Duplicate indices are also removed. Figure 1250 shows the merged index, which, after deduplication, has 12 numbers. Figure 1260 shows the merged data; since each data element is a vector, the corresponding vector address is shown here.
[0115] The convolution operation scheme for sparse data in this disclosure embodiment has been described above from multiple aspects. Compared with conventional convolution schemes, this disclosure embodiment avoids excessive invalid operations by processing only non-zero / non-empty data in the sparse data, greatly saving computational resources and improving processing efficiency. Furthermore, by using a dedicated hardware MERGE instruction to merge and fuse valid product results, data processing can be accelerated. The sparse convolution scheme provided in this disclosure embodiment is particularly suitable for processing LiDAR point cloud data.
[0116] This disclosure also provides a data processing circuit for performing convolution operations on the sparse data described above, and a data processing method implemented by the data processing circuit.
[0117] Figure 13 A schematic structural diagram of a data processing circuit that can implement embodiments of this disclosure is shown as an example. Figure 13 As shown, the data processing circuit 1300 includes a control circuit 1310, a storage circuit 1320, and an arithmetic circuit 1330.
[0118] The control circuit 1310 is responsible for handling various functions on the data processing circuit 1300, including but not limited to control, instruction fetching, decoding, and calculation. The control circuit 1310 may include, for example, […]. Figure 3 The control module 31 in the middle.
[0119] In some embodiments, the control circuit 1310 can be configured to control the storage circuit 1320 and the arithmetic circuit 1330 to perform N-dimensional convolution operations on the input data and the convolution kernel, where N>1, and N represents the number of convolution dimensions in which sliding accumulation is performed. In this convolution operation, the input data is sparse data, and each input data element has index and numerical information.
[0120] Storage circuit 1320 can be used to store information, which includes at least pre-processing and / or post-processing information, and may also include intermediate information that needs to be cached during processing, such as... Figure 3 The various RAMs shown are referred to as on-chip caches. In some embodiments, the storage circuitry 1320 may be configured to store input data, convolution kernels, convolution operation results, and / or cache intermediate results, for example, to provide cache space required during the execution of a MERGE instruction.
[0121] The arithmetic circuit 1330 can be configured to perform various arithmetic operations according to relevant instructions. Specifically, the arithmetic circuit 1330 can be configured to perform operations on input data elements and convolution kernels under the control of the control circuit 1310 to obtain multiple operation results and corresponding indices; and to merge these multiple operation results into one fused data as the result of the convolution operation according to their corresponding indices, wherein operation results with the same index are accumulated.
[0122] In one embodiment, the arithmetic circuit 1330 may further include an arithmetic processing circuit (not shown), which can be configured to preprocess the data before the arithmetic circuit performs the operation or postprocess the data after the operation according to the arithmetic instructions. In some application scenarios, the aforementioned preprocessing and postprocessing may include, for example, data splitting and / or data concatenation operations.
[0123] In some embodiments, the computation circuit 1330 may be further configured to perform the following operations: merging the weight data of each convolution dimension of the convolution kernel into one dimension with a size of offset, where offset is the total number of weight data in the convolution dimension; arranging the input data elements in index order into an input vector, where each vector element includes one input data element; and performing a multiplication operation on the merged convolution kernel and the input vector in the convolution dimension to obtain offset-path data, wherein each path data includes several product results, and each product result includes any of scalar, vector, or higher-dimensional data.
[0124] In some embodiments, the input data includes N convolutional dimensions and at least one non-convolutional dimension, and the convolutional kernel includes N convolutional dimensions and at least one non-convolutional dimension. The operation circuit 1330 can be further configured to perform corresponding operation processing on the non-convolutional dimension according to the operation requirements.
[0125] In some embodiments, the non-convolutional dimension of the input data includes the input channel Ci dimension, and the non-convolutional dimension of the convolutional kernel includes the input channel Ci dimension and the output channel Co dimension. The operation circuit 1330 can be further configured to: perform bitwise accumulation operation on the multiplication result in the convolutional dimension in the Ci dimension; and stack the accumulation result in the Ci dimension in the Co dimension as a multi-path operation result, wherein each operation result is a vector in the Co dimension.
[0126] In some embodiments, the arithmetic circuit 1330 may be further configured to obtain the indices of each arithmetic result as follows: determining the index of each arithmetic result in the multiplexed arithmetic results according to an index mapping relationship, wherein the index mapping relationship indicates that the index of the arithmetic result associated with the input data element is determined based on the index of the input data element. Further, the index mapping relationship may also indicate that indices exceeding the size range of the output data dimension are set to predetermined values. In some implementations, when the multiplexed arithmetic results include a Co dimension, the indices of each arithmetic result are reused on the Co dimension.
[0127] Optionally or additionally, in some implementations, the arithmetic circuitry may be further configured to: adjust the indices of the input data elements based on the padding rules of the convolution operation before determining the indices according to the index mapping relationship.
[0128] In some embodiments, the arithmetic circuit 1330 may be further configured to perform the above-described merging process according to the hardware instruction MERGE.
[0129] In some embodiments, the upper limit of the merging processing of the MERGE instruction is K, where K>1. When the number of multiple operation results to be merged exceeds K, the operation circuit 1330 can be further configured to execute multiple MERGE instructions to merge the multiple operation results.
[0130] In some embodiments, the control circuit 1310 may be configured to execute multiple MERGE instructions according to the control operation circuit 1330 as follows:
[0131] A. Based on K, split the offset of the operation result to be merged into M groups, where M = ceil(offset / K);
[0132] B. Execute M MERGE instructions on the M sets of calculation results to obtain M channels of fused data;
[0133] C. When M does not exceed K, execute the MERGE instruction once for the M channels of fused data to obtain the final result; or
[0134] D. When M exceeds K, return to step A and split M again according to K until the number of paths to be merged does not exceed K.
[0135] In some embodiments, the input data includes multidimensional data that is sparse across multiple dimensions, and the input data is preprocessed before being provided to the computational circuitry. This preprocessing may include, for example, merging the sparse multiple dimensions into a single dimension; densifying the sparse data points in the input data along the merged dimension to form input data elements; and representing each input data element using indexes and numerical information, where the indexes are one-dimensional or multi-dimensional indices, and the numerical information includes any of scalar, vector, or higher-dimensional data.
[0136] The convolution scheme disclosed in this embodiment can be applied to two-dimensional convolution, three-dimensional convolution, or higher-dimensional convolution.
[0137] Figure 14 An exemplary flowchart of a data processing method implemented by a data processing circuit according to an embodiment of this disclosure is shown. In this data processing method, a control circuit controls a storage circuit and a processing circuit to perform an N-dimensional convolution operation on the input data and the convolution kernel, where N>1, and N represents the number of convolution dimensions in which sliding accumulation is performed during the convolution operation. The input data is sparse data, and each input data element has index and numerical information.
[0138] More specifically, in step 1410, the arithmetic circuit, under the control of the control circuit, performs operations on the input data elements and the convolution kernel to obtain multi-channel operation results.
[0139] In step 1420, the arithmetic circuit, under the control of the control circuit, determines the index corresponding to each arithmetic result.
[0140] Finally, in step 1430, the arithmetic circuit merges the multiple arithmetic results into one fused data stream according to their corresponding indices as the result of the convolution operation, where the product results with the same index are accumulated.
[0141] Although the steps of calculating the results of multiplexed operations and determining the index are described separately and sequentially in the above description, those skilled in the art will understand that these two steps can also be performed simultaneously or in reverse order, and the embodiments disclosed herein are not limited in this respect. Furthermore, although the step of determining the index is described as being performed by the computing circuitry in the above description, those skilled in the art will understand that the step of determining the index can also be performed by software, such as by control circuitry. Further, although the various processing steps are generally described as being performed on the computing circuitry in the above description, the computing circuitry here can also be distributed, for example, including computing circuitry in a heterogeneous system, so that some operations are performed, for example, on the CPU, and other operations are performed, for example, on the GPU. In one implementation, the preprocessing of the input data can be performed, for example, on the CPU. This preprocessing may include, for example, densification of the sparse input data, index sorting of the densified data elements, etc. Processing such as multiplication of the input data with the convolution kernel, merging and fusing the results of multiplexed operations, etc., can be performed on the GPU, thereby fully utilizing the advantages of heterogeneous systems.
[0142] Those skilled in the art will understand that the description of the convolution operation processing of sparse data in the embodiments of this disclosure described above in conjunction with the accompanying drawings can also be applied to... Figure 13 Data processing circuits and Figure 14 The data processing methods are already described, so they will not be repeated here.
[0143] This disclosure also provides a chip that may include the data processing apparatus of any of the embodiments described above in conjunction with the accompanying drawings. Furthermore, this disclosure also provides a circuit board that may include the aforementioned chip.
[0144] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.
[0145] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.
[0146] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.
[0147] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.
[0148] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.
[0149] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.
Claims
1. A data processing circuit, applied in the field of target detection based on lidar, comprising a control circuit, a storage circuit, and a processing circuit, wherein: The control circuit is configured to control the storage circuit and the computing circuit to perform N-dimensional convolution operations on the input data and convolution kernel to implement target detection, where N>1, and N represents the number of convolution dimensions in which sliding accumulation is performed. The input data is sparse point cloud data representing detection data in three-dimensional space obtained based on LiDAR target detection, and each input data element has index and numerical information. The storage circuit is configured to store information, which includes at least pre-processing and / or post-processing information. as well as The computation circuit is configured to perform operations on the input data elements and the convolution kernel under the control of the control circuit to obtain multiple operation results and corresponding indices; and to merge the multiple operation results into one fused data stream according to their corresponding indices as the result of the convolution operation, wherein operation results with the same index are accumulated.
2. The data processing circuit according to claim 1, wherein the arithmetic circuit is further configured to perform the following arithmetic operations: The weight data of each convolutional dimension of the convolutional kernel are merged into one dimension, the size of which is offset, and offset is the total number of weight data in the convolutional dimension. The input data elements are arranged into an input vector in indexed order, and each vector element includes one input data element; and Perform a multiplication operation on the convolution dimension of the merged convolution kernel and the input vector to obtain offset data, where each data path includes several product results, and each product result includes any of the following: scalar, vector, or higher-dimensional data.
3. The data processing circuit according to claim 2, wherein the input data includes N convolutional dimensions and at least one non-convolutional dimension, and the convolutional kernel includes N convolutional dimensions and at least one non-convolutional dimension, and the computation circuit is further configured to: According to the computational requirements, perform the corresponding computational processing on the non-convolutional dimensions.
4. The data processing circuit according to claim 3, wherein the non-convolutional dimension of the input data includes the input channel Ci dimension, the non-convolutional dimension of the convolution kernel includes the input channel Ci dimension and the output channel Co dimension, and the computation circuit is further configured to: Perform a bitwise summation operation on the multiplication results in the convolution dimension along the Ci dimension; and The accumulated results in the Ci dimension are stacked in the Co dimension to serve as the result of the multi-way operation, where each result is a vector in the Co dimension.
5. The data processing circuit according to claim 1, wherein the arithmetic circuit is further configured to obtain indices for each arithmetic result as follows: Based on the index mapping relationship, the index of each operation result in the multi-way operation result is determined, wherein the index mapping relationship indicates that the index of the operation result associated with the input data element is determined based on the index of the input data element.
6. The data processing circuit according to claim 5, wherein the index mapping relationship further indicates that an index exceeding the range of the output data dimension size is set to a predetermined value.
7. The data processing circuit according to claim 5, wherein when the multiplexing result includes the output channel Co dimension, the index of each calculation result is reused on the Co dimension.
8. The data processing circuit according to claim 5, wherein the arithmetic circuit is further configured to: Before determining the index based on the index mapping relationship, the index of the input data element is adjusted based on the padding rule of the convolution operation.
9. The data processing circuit of claim 1, wherein the arithmetic circuit is further configured to perform a merge process according to the hardware instruction MERGE.
10. The data processing circuit according to claim 9, wherein the upper limit of the merging processing of the MERGE instruction is K, K>1, and when the number of multiple operation results to be merged exceeds K, the operation circuit is further configured to execute multiple MERGE instructions to merge the multiple operation results.
11. The data processing circuit of claim 10, wherein the control circuit is configured to control the arithmetic circuit to execute multiple MERGE instructions as follows: A. Based on K, split the offset of the operation result to be merged into M groups, where M = ceil(offset / K); B. Execute M MERGE instructions on the M sets of calculation results to obtain M channels of fused data; C. When M does not exceed K, execute the MERGE instruction once for the M channels of fused data to obtain the final result; or D. When M exceeds K, return to step A and split M again according to K until the number of paths to be merged does not exceed K.
12. The data processing circuit according to any one of claims 1-11, wherein the input data comprises multidimensional data that is sparse in multiple dimensions, and the input data is preprocessed before being provided to the arithmetic circuit, the preprocessing comprising: Combine multiple sparse dimensions into one dimension; The sparse data points in the input data are densified in the merging dimension to form input data elements; as well as Each input data element is represented using an index and numerical information, wherein the index is a one-dimensional or multi-dimensional index, and the numerical information includes any of scalar, vector, or higher-dimensional data.
13. The data processing circuit according to any one of claims 1-11, wherein N=2, N=3 or N=4.
14. A chip comprising a data processing circuit according to any one of claims 1-13.
15. A circuit board comprising the chip according to claim 14.
16. A method for processing data using the data processing circuit according to any one of claims 1-13.
Citation Information
Patent Citations
Sparse neural network-oriented system on chip
CN112631983A
Systems and methods for generation of sparse code for convolutional neural networks
WO2021054990A1
