Computer system and data processing method

Abstract

The application provides a computer system and a data processing method. The computer system comprises a matrix calculation unit composed of two-dimensional calculation nodes. The matrix calculation unit is used for performing convolution operation on input data according to convolution parameters, and each calculation node completes the calculation of a single convolution window. The application can accelerate deep convolution processing and reduce system energy consumption.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a computer system and a data processing method. BACKGROUND

[0002] The current mainstream technical solution of machine vision adopts a convolutional neural network (CNN). However, the classic CNN has a large number of parameters and a large amount of computation, and on an edge computing platform with small computing power and power, real-time performance and endurance are the main obstacles for product landing. Depthwise convolution can significantly reduce the number of CNN parameters, and bring great improvement in calculation speed with limited accuracy loss, which has great engineering value in the field of edge computing.

[0003] However, the current mainstream computer architecture cannot well support depthwise convolution, and has problems of slow calculation speed and high system energy consumption. SUMMARY

[0004] Embodiments of the present application provide a computer system and a data processing method, which can accelerate depthwise convolution processing and reduce system energy consumption.

[0005] The first aspect of the present application provides a computer system, comprising a matrix calculation unit composed of two-dimensional calculation nodes, the matrix calculation unit is configured to perform convolution operation on input data according to convolution parameters, and each calculation node completes the calculation of a single convolution window.

[0006] The present application uses a matrix calculation unit composed of two-dimensional calculation nodes to perform convolution operation of depthwise convolution, and each clock cycle can complete the calculation of multiple data points, thereby accelerating depthwise convolution processing and reducing system energy consumption.

[0007] In some optional embodiments, the number of columns of the matrix calculation unit is equal to the number of channels of the input data.

[0008] In some optional embodiments, the computer system further comprises an external memory configured to store the input data and the convolution parameters.

[0009] In some optional embodiments, the computer system further comprises a controller, a data buffer write unit, a data buffer read unit, a data supply unit, a data backwrite unit, a data buffer, a parameter buffer, and an output result buffer, the parameter buffer is connected to the external memory and used to cache the convolution parameters; the controller is configured to control the data buffer write unit to read the input data from the external memory and write the read input data into the data buffer for caching; the controller is further configured to control the data supply unit to send an input data read request to the data buffer read unit; the data buffer read unit is configured to read the input data from the data buffer according to the input data read request and transmit the input data to the matrix calculation unit through the data supply unit; the controller is further configured to control the matrix calculation unit to read the convolution parameters from the parameter buffer, perform convolution operation on the input data according to the convolution parameters to obtain output data, and cache the output data into the output result buffer; the controller is further configured to control the data backwrite unit to read the output data cached in the output result buffer, send the read output data to the data buffer write unit, and send an output data write request to the data buffer write unit; and the data buffer write unit is further configured to write the output data into the data buffer according to the output data write request for caching.

[0010] In some optional embodiments, the input data comprises subgraphs obtained by splitting feature maps, and the external memory is further configured to store padding data; the controller is further configured to control the data buffer write unit to read the padding data from the external memory and generate a padding data write request, and write the read padding data into the data buffer according to the padding data write request; and the controller is further configured to control the data buffer read unit to generate a padding data read request and read new padding data from the output data cached in the data buffer according to the padding data read request, and store the new padding data from the data buffer to the external memory.

[0011] In some optional embodiments, the data buffer write unit is further configured to perform write conflict checking on the output data write request and the padding data write request.

[0012] In some optional embodiments, the priority of the output data write request is higher than the priority of the padding data write request.

[0013] In some optional embodiments, the data buffer write unit is further configured to perform read conflict checking on the input data read request and the padding data read request.

[0014] In some optional embodiments, the priority of the input data reading request is higher than the priority of the padding data reading request.

[0015] In some optional embodiments, the sub-graph is split along the height direction or the width direction of the feature map.

[0016] In some optional embodiments, the computer system performs deep convolution on the input data in a pipelined manner.

[0017] In some optional embodiments, the input data is split into data blocks along the width direction or the height direction, the data blocks are split into sub-data blocks along the height direction or the width direction, and the matrix calculation unit performs convolution operation on one sub-data block per clock cycle.

[0018] In some optional embodiments, the height of the sub-data block is 1.

[0019] The second aspect of the present application provides a data processing method applied to a computer system, the computer system comprising a matrix calculation unit composed of two-dimensional calculation nodes, the method comprising: performing convolution operation on input data according to convolution parameters by the matrix calculation unit, and each calculation node completing the calculation of a single convolution window.

[0020] In some optional embodiments, the number of columns of the matrix calculation unit is equal to the number of channels of the input data.

[0021] In some optional embodiments, the computer system further comprises an external memory for storing the input data and the convolution parameters.

[0022] In some optional embodiments, the computer system further comprises a controller, a data buffer write unit, a data buffer read unit, a data supply unit, a data backwrite unit, a data buffer, a parameter buffer, and an output result buffer, the parameter buffer is connected to the external memory and used to cache the convolution parameters, and the method further comprises: the controller controls the data buffer write unit to read the input data from the external memory and write the read input data to the data buffer for caching; the controller controls the data supply unit to send an input data read request to the data buffer read unit; the data buffer read unit reads the input data from the data buffer according to the input data read request and transmits the input data to the matrix calculation unit through the data supply unit; the controller controls the matrix calculation unit to read the convolution parameters from the parameter buffer, performs convolution operation on the input data according to the convolution parameters to obtain output data, and caches the output data to the output result buffer; the controller controls the data backwrite unit to read the output data cached in the output result buffer, sends the read output data to the data buffer write unit, and sends an output data write request to the data buffer write unit; and the data buffer write unit writes the output data to the data buffer according to the output data write request for caching.

[0023] In some optional embodiments, the input data comprises subgraphs obtained by splitting feature maps, and the external memory is further used to store padding data; and the method further comprises: the controller controls the data buffer write unit to read the padding data from the external memory and generates a padding data write request, and writes the read padding data to the data buffer according to the padding data write request; and the controller controls the data buffer read unit to generate a padding data read request and read new padding data from the output data cached in the data buffer according to the padding data read request, and stores the new padding data from the data buffer to the external memory.

[0024] In some optional embodiments, the method further comprises: the data buffer write unit is further used to perform write conflict checking on the output data write request and the padding data write request.

[0025] In some optional embodiments, the priority of the output data write request is higher than the priority of the padding data write request.

[0026] In some optional embodiments, the method further comprises: the data buffer write unit performs read conflict checking on the input data read request and the padding data read request.

[0027] In some optional embodiments, the input data reading request has a higher priority than the padding data reading request.

[0028] In some optional embodiments, the sub-graphs are obtained by cutting along the height direction or the width direction of the feature map.

[0029] In some optional embodiments, the data processing method performs deep convolution on the input data in a pipeline manner.

[0030] In some optional embodiments, the input data is cut into data blocks along the width direction or the height direction, and the data blocks are cut into sub-data blocks along the height direction or the width direction, and the matrix calculation unit performs convolution operation on one sub-data block per clock cycle.

[0031] In some optional embodiments, the height of the sub-data block is 1.

[0032] It should be understood that the data processing method provided in the second aspect above corresponds to the computer system of the first aspect above, and thus the beneficial effects achieved thereby can refer to the beneficial effects of the corresponding method provided above, which will not be described herein again. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a schematic diagram of a computer system provided in an embodiment of the present application.

[0034] Figure 2 is a schematic diagram of data supply of a matrix calculation unit in Figure 1

[0035] Figure 3 is a schematic diagram of a computing node in an embodiment of the present application.

[0036] Figure 4 is a schematic diagram of data multiplexing in a matrix calculation unit in an embodiment of the present application.

[0037] Figure 5 is a schematic diagram of cutting a feature map into sub-graphs and processing the sub-graphs.

[0038] Figure 6 is a flowchart of processing a sub-graph in an embodiment of the present application.

[0039] Figure 7 is a schematic diagram of the relationship between a feature map, a sub-graph, a data block, and a sub-data block in an embodiment of the present application.

[0040] Figure 8 is a schematic diagram of performing deep convolution on a data block in an input feature map / input sub-graph in an embodiment of the present application. ​

[0041] Figure 9 is a timing relationship diagram of deep convolution in a pipeline manner according to an embodiment of the present application.

[0042] Figure 10 is a flowchart of a data processing method provided by an embodiment of the present application.

[0043] Figure 11 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0044] For the convenience of understanding, some descriptions of concepts related to the embodiments of the present application are exemplarily given for reference.

[0045] It should be noted that “at least one” in the present application means one or more, and “multiple” means two or more than two. “And / or” describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The terms “first”, “second”, “third”, “fourth” and the like (if any) in the description and claims of the present application and the drawings are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0046] Figure 1 is a schematic diagram of a computer system provided by an embodiment of the present application.

[0047] The computer system 10 provided by the embodiments of the present application includes a chip 100 (i.e. a system on chip, SOC) and an external memory 101. The chip 100 includes a controller 1001, a data buffer write unit 1002, a data buffer read unit 1003, a data supply unit 1004, a data backwrite unit 1005, a matrix calculation unit 1006, a data buffer 1007, a parameter buffer 1008, and an output result buffer 1009. The external memory 101 is connected to the data buffer read unit 1003, the data buffer write unit 1002, and the parameter buffer 1008. The data buffer 1007 is connected to the data buffer write unit 1002 and the data buffer read unit 1003. The controller 1001 is connected to the data buffer write unit 1002, the data buffer read unit 1003, the data supply unit 1004, the data backwrite unit 1005, and the matrix calculation unit 1006. The data supply unit 1004 is connected to the data buffer read unit 1003 and the matrix calculation unit 1006. The data backwrite unit 1005 is connected to the data buffer write unit 1002 and the output result buffer 1009. The matrix calculation unit 1006 is connected to the parameter buffer 1008 and the output result buffer 1009.

[0048] Exemplarily, the controller 1001 can include at least one of the following types: a central processing unit (CPU), a microcontroller unit (MCU), a digital signal processor (DSP), an application processor (AP), a graphics processing unit (GPU), and a neural-network processing unit (NPU).

[0049] The computer system 10 can include a robot, a mobile phone, a vehicle-mounted computer, etc., and the corresponding chip 100 can be a robot chip, a mobile phone chip, a vehicle-mounted chip, etc. The computer system 10 can implement machine vision tasks such as recognition and classification through related software.

[0050] The computer system 10 performs convolution operation of deep convolution by using the matrix calculation unit 1006, and can implement fast and low-energy deep convolution. The computer system 10 can improve the storage bandwidth by cooperation of each unit in the chip 100. In an embodiment of the present application, the computer system 10 can be used for MobileNet. MobileNet is an efficient model proposed for mobile and embedded devices. MobileNet is based on streamlined architecture and uses depthwise separable convolutions to build a lightweight deep neural network. Depthwise separable convolution includes two parts of depthwise convolution and pointwise convolution. The computer system 10 provided in the embodiments of the present application can be used for deep convolution of MobileNet. It can be understood that the computer system 10 provided in the embodiments of the present application can be used to implement other convolutional neural networks.

[0051] The external memory 101 is used to store input data and convolution parameters. The chip 100 is used to perform deep convolution on the input data stored in the external memory 101 according to the convolution parameters.

[0052] The input data can be a feature map, or a subgraph (i.e., a sub-feature map) obtained by cutting the feature map.

[0053] The feature map has three dimensions of width (W), height (H) and channel (H), the width of the feature map can be represented as w, the height can be represented as h, and the channel number can be represented as c, and the feature map size can be represented as w*h*c. The convolution parameters can include weights and biases.

[0054] It should be noted that the input data can be an image (the image can be regarded as a special feature map), and the chip 100 can perform deep convolution on the image stored in the external memory 101 according to the convolution parameters.

[0055] In order to support deep convolution on a large size feature map, when the size of the feature map is large, the feature map can be cut into subgraphs, and the subgraphs are stored in the external memory 101. The feature map can be cut into subgraphs along the height direction, or the feature map can be cut into subgraphs along the width direction. For example, the size of the feature map is 1024*1024*64 (width is 1024, height is 1024, and channel number is 64), the feature map can be cut along the height direction, and the feature map is cut into 4 subgraphs of 1024*256*64. The 4 subgraphs of 1024*256*64 are stored in the external memory 101.

[0056] The chip 100 can perform one layer of deep convolution on the input data (i.e. the feature map or the subgraph), or can perform multiple layers of deep convolution on the input data.

[0057] If the input data includes multiple subgraphs, the chip 100 performs deep convolution on each subgraph. If multiple layers of deep convolution are required, the chip 100 performs deep convolution of all layers on one subgraph, and then performs deep convolution on the next subgraph.

[0058] When the chip 100 performs multiple layers of deep convolution on the subgraph, the padding data required for the next layer of deep convolution can be obtained from the output data (i.e. the output subgraph) obtained from each layer of deep convolution, and the padding data is stored in the external memory 101 for the next layer of deep convolution. In an embodiment of the present application, the size of the convolution kernel is k*k, and the bottom k-1 rows of data along the height direction of the output subgraph are used as padding data. For example, the size of the convolution kernel is 3*3, and the padding data is the bottom 2 rows of data along the height direction of the output subgraph.

[0059] The data buffer 1007 is used to buffer input data, output data and padding data. The data buffer 1007 is composed of a plurality of random access memories (RAMs). The random access memories constituting the data buffer 1007 can be single-port random access memories or dual-port random access memories. The single-port random access memory has only one set of data lines and address lines and cannot read and write at the same time; the dual-port random access memory has two sets of data lines and address lines and can read and write at the same time. The random access memories in the data buffer 1007 are used to buffer the input data (i.e., input feature map / input submap) of deep convolution, the output data (i.e., output feature map / output submap) and the padding data according to a certain address mapping. The input feature map / input submap and the output feature map / output submap are relative to one deep convolution. The input feature map / input submap refers to the feature map / submap before deep convolution, for example, the feature map / submap from the external memory 101, and the output feature map / output submap refers to the feature map / submap after deep convolution. The data buffer 1007 includes at least two sets of random access memories to support simultaneous reading and writing. Each set of random access memories includes a plurality of random access memories to support reading / writing of a plurality of data points in the feature map / submap width direction at a time. Figure 1 The data buffer 1007 includes two sets of random access memories, the width of the input feature map / input submap is w, the size of the convolution kernel is k*k, and each set includes at least w+k-1 random access memories. One set of random access memories stores the input feature map / input submap, and the other set of random access memories stores the output feature map / output submap, and at least w+k-1 random access memories are independently addressable for reading and writing at the same time. The more sets of random access memories that can read and write at the same time in the data buffer 1007, the smaller the probability of data read / write conflict.

[0060] In an embodiment of the present application, the data buffer 1007 can read and write data in a ping-pong manner. Ping-pong is a data buffering method that uses two data buffers to achieve continuous data transmission and improve data transmission rate. Since the data obtained by a single buffer is easily overwritten during transmission and processing, the ping-pong method can maintain reading data from one buffer and writing data to another buffer, i.e., the two buffers alternately read and write.

[0061] In an embodiment of the present application, the data in the data buffer 1007 is stored in the NHWC data format, where N represents the number of pictures in a batch, H represents the height, W represents the width, and C represents the number of channels. Storing data in the NHWC data format means storing data in the order of C direction, W direction, H direction and N direction.

[0062] The parameter buffer 1008 is used to buffer the convolution parameters. The parameter buffer 1008 can be composed of a small-capacity random access memory. The parameter buffer 1008 can buffer the convolution parameters in a ping-pong manner. The computer system 10 can include a DMA (Direct Memory Access) controller (not shown in the figure). The computer system can write the convolution parameters from the external memory 101 to the parameter buffer through the DMA controller by using the DMA mode.

[0063] The output result buffer 1009 is used to buffer the operation results (i.e., output data) of the matrix calculation unit 1006. The output result buffer 1009 supports the reading of the data backwriting unit 1005. The output result buffer 1009 can be composed of a small-capacity random access memory. The output result buffer 1009 can buffer the operation results of the matrix calculation unit 1006 in a ping-pong manner.

[0064] The controller 1001 is used to send control instructions to the data buffer writing unit 1002, the data buffer reading unit 1003, the data supply unit 1004, the data backwriting unit 1005, and the matrix calculation unit 1006, to indicate the size, position, and other information of the data to be processed.

[0065] The controller 1001 controls the data buffer writing unit 1002 to read the input data from the external memory 101 and write the read input data to the data buffer 1007 for buffering.

[0066] The controller 1001 is also used to control the data supply unit 1004 to send an input data reading request to the data buffer reading unit 1003.

[0067] The data buffer reading unit 1003 is used to read the input data from the data buffer 1007 according to the input data reading request and transmit the input data to the matrix calculation unit 1006 through the data supply unit 1004. The data supply unit 1004 can process the input data by using an activation function. In an embodiment of the present application, the activation function used by the data supply unit 1004 is a Relu function.

[0068] The controller 1001 is also used to control the matrix calculation unit 1006 to read the convolution parameters from the parameter buffer 1008, perform convolution operation on the input data according to the convolution parameters to obtain output data, and buffer the output data to the output result buffer 1009.

[0069] The controller 1001 is further configured to control the data backwrite unit 1005 to read the output data cached in the output data buffer 1009, send the read output data to the data buffer write unit 1002, and send an output data write request to the data buffer write unit 1002. The data backwrite unit 1005 can quantize the operation result. Quantizing the operation result is to convert the operation result, for example, from a 32-bit floating point number to an 8-bit fixed point number.

[0070] The data buffer write unit 1002 is further configured to write the output data into the data buffer memory 1007 according to the output data write request.

[0071] If the input data is a subgraph obtained by splitting a feature map, the external memory 101 is further configured to store padding data. The controller 1001 is further configured to control the data buffer write unit 1002 to read the padding data from the external memory 101 and generate a padding data write request, and write the read padding data into the data buffer 1007 according to the padding data write request. The controller 1001 is further configured to control the data buffer read unit 1003 to generate a padding data read request, read new padding data from the output data cached in the data buffer 1007 according to the padding data read request, and store the new padding data from the data buffer 1007 to the external memory 101.

[0072] In an embodiment of the present application, the data buffer read unit 1003 performs read conflict checking on the input data read request and the padding data read request. Since the amount of padding data is small, the priority of the input data read request can be higher than that of the padding data read request.

[0073] In an embodiment of the present application, the data buffer write unit 1002 performs write conflict checking on the output data write request and the padding data write request. Since the amount of padding data is small, the priority of the output data write request can be higher than that of the padding data write request.

[0074] In an embodiment of the present application, the address requested by the input data read request of the data supply unit 1004 and the address requested by the output data write request of the data backwrite unit 1005 are continuous, so as to avoid read conflict of the data supply unit 1004 and write conflict of the data backwrite unit 1005. The data supply unit 1004 is a read data buffer, the data backwrite unit 1005 is a write data buffer, and the data supply unit 1004 and the data backwrite unit 1005 simultaneously access multiple random memories of the data buffer 1007.

[0075] In the data buffer 1007, a single random access memory can receive only read requests or write requests at the same time, or it can receive both read requests and write requests simultaneously. Read requests and write requests are executed serially, with write requests taking priority.

[0076] Figure 1 In the illustrated embodiment, the data buffer write unit 1002 and the data buffer read unit 1003 alternately occupy the two sets of random access memory to avoid read-write conflicts and improve the utilization rate of the data buffer 1007.

[0077] The matrix computation unit 1006 consists of two-dimensional computation nodes. For example... Figure 1 As shown, the matrix computation unit 1006 includes m*n computation nodes, where m is the number of rows in the matrix computation unit 1006 and n is the number of columns in the matrix computation unit 1006. The number of rows m in the matrix computation unit 1006 is equal to the width of the input data provided by the data supply unit 1004, and the number of columns n in the matrix computation unit 1006 is equal to the number of channels c of the input data provided by the data supply unit 1004. In other words, the width and channels of the input data are mapped to the rows and columns of the matrix computation unit 1006, respectively. Using the matrix computation unit 1006 can improve the performance of depthwise convolution.

[0078] Figure 2 yes Figure 1 A schematic diagram of the data supply for the matrix calculation unit.

[0079] The data supply unit provides input data (input feature map / input submap) to the matrix calculation unit. Figure 2 In the illustrated embodiment, the data supply unit transmits the input data of one sub-data block to the matrix computation unit at a time. The sub-data block is, for example, the gray area in the upper left corner of the figure. In one embodiment of this application, the height of the sub-data block is 1. The sub-data block will be described in detail later. In one embodiment of this application, the size of the convolution kernel is k*k, and the data supply unit transmits k data points from the sub-data block to each computation node of the matrix computation unit at a time. The matrix computation unit completes one convolution operation through k data transmissions. For example, if the size of the convolution kernel is 3*3, the data supply unit transmits 3 data points from the sub-data block to each computation node of the matrix computation unit at a time, and the matrix computation unit completes one convolution operation through 3 data transmissions. For example, in one data transmission, the data supply unit transmits a0, a1, a2 to computation node (0,0), a1, a2 to computation node (1,0), a2, a3 to computation node (2,0), ...

[0080] The parameter buffer provides convolution parameters for the matrix computation unit. Figure 2In the illustrated embodiment, the convolution parameters provided by the parameter buffer to the matrix calculation unit include weights (denoted as weight in the figure) and biases (denoted as bias in the figure). The parameter buffer provides a set of weights and a bias for each column of the matrix calculation unit. For example, the parameter buffer provides weights d0, d1, d2, d3, d4, d5, d6, d7, d8 and bias e0 for the first column of the matrix calculation unit.

[0081] Each column of the matrix calculation unit performs data calculation for one channel of the input data, and each row of the matrix calculation unit performs data calculation for one width of the input data. Therefore, the matrix calculation unit generates m*n operation results per clock cycle.

[0082] The matrix calculation unit can realize data multiplexing in the width or height direction of the input data. Referring to Figure 2 As shown, the convolution kernel size is 3*3, and the first to third data points a0, a1, a2 are input to the calculation node (0, 0) in the width (W) direction of the input feature map / input subgraph, the second to fourth data points a1, a2, a3 are input to the calculation node (1, 0), and the second to third data points a1, a2 are multiplexed. The data of the input feature map / input subgraph is read out and then unfolded inside the matrix calculation unit 1006, and the data multiplexing in the width / height direction saves the connection overhead between the data buffer 1007 and the matrix calculation unit 1006.

[0083] Figure 3 FIG. 1 is a schematic diagram of a calculation node in an embodiment of the present application.

[0084] In an embodiment of the present application, each calculation node can include a first register, a second register, a multiplier, an addition tree (the multiplier and the addition tree are collectively denoted as MAC in the figure), and an accumulator (denoted as bias adder in the figure). The first register is used to store a data point (denoted as data in the figure) of the input data, the second register is used to store a convolution parameter (including a weight and a bias, the weight is denoted as weight in the figure, and the bias is denoted as bias), the multiplier is used to multiply each data point and a corresponding weight, the addition tree is used to add the results of the multipliers, and the accumulator is used to add the result of the adder and the bias. The number of multipliers included in each calculation node is equal to the convolution kernel size. The convolution kernel size is k*k, and the number of multipliers included in each calculation node is k*k. For example, the convolution kernel size is 3*3, and the number of multipliers included in each calculation node is 9.

[0085] Each computing node of the matrix computation unit completes the computation of a single convolution window. Taking a 3*3 convolution as an example, the data of the input feature map / input subgraph is represented as in[0], in[1], …, in[8], the weight (i.e. the data of the convolution kernel) is represented as w[0], w[1], …, w[8], and the bias is represented as bias. The computation completed by the computing node can be represented as:

[0086] Out = (in[0]*w[0] + in[1]*w[1] + … + in[8]*w[8]) + bias.

[0087] For example, in[0] = a0, in[1] = a1, in[2] = a2, in[3] = b0, in[4] = b1, in[5] = b2, in[6] = c0, in[7] = c1, in[8] = c2, w[0] = d0, w[1] = d1, w[2] = d2, w[3] = d3, w[4] = d4, w[5] = d5, w[6] = d6, w[7] = d7, w[8] = d8, and bias = e0. The computation completed by the computing node can be represented as:

[0088] Out = (a0*d0 + a1*d1 + a2*d2 + b0*d3 + b1*d4 + b2*d5 + c0*d6 + c1*d7 + c2*d2 + b0*d8) + e0.

[0089] In another embodiment of the present application, each computing node can include a first register, a second register, a multiplier, and an addition tree. The addition tree will add the results of the respective multipliers and add the bias.

[0090] In an embodiment of the present application, the matrix computation unit can employ a method of window data loopback (see Figure 4 ) to reduce the line width of the data buffer to the matrix computation unit.

[0091] Figure 4 is a schematic diagram of data multiplexing in the matrix computation unit in an embodiment of the present application.

[0092] During depthwise convolution, the input data (input feature map / input sub-map) is slid along the height (H) or width (W) direction to acquire data points (i.e., data stored in the first register) for the matrix computation unit. Each clock cycle, the matrix computation unit stores k*k (e.g., 9) data points in the first register. In the next clock cycle, when the input data is slid along the height / width direction, the first register within the computation node has already stored k*k data points. Only the top 1*k data points need to be replaced with the newly input 1*k data points to form a new window for the next computation. This fully utilizes the data in the height direction of the input feature map / input sub-map, achieving data reuse (e.g., a 3*3 window with a stride of 1, where adjacent windows (top / bottom / left / right) have 2*3 / 3*2 data reuse). See, for example, [link to documentation]. Figure 4 In steps 4 and 5, after performing a convolution operation in step 4, simply replace a0, a1, and a2 with the newly input d0, d1, and d2 to perform the next convolution operation. In this embodiment, it is not necessary to repeatedly fetch data from the data buffer, allowing for efficient computation in each clock cycle while saving memory overhead and reducing system power consumption. If the matrix computation unit does not buffer the data, the data supply unit needs to buffer and transmit the data, which significantly increases the number of traces between the data supply unit and the matrix computation unit. This embodiment greatly reduces the number of traces between the data supply unit and the matrix computation unit.

[0093] Current mainstream computer architectures do not support deep convolution well. This is partly because mainstream architectures handle general convolution operations, specifically three-dimensional (3D) convolution, which reuses data along the channel direction to reduce system power consumption; deep convolution does not require channel-wise data reuse. Furthermore, while deep convolution has low data dependency and is computationally fast, storage becomes a major bottleneck.

[0094] The computer system provided in this application performs convolution operations through a matrix computation unit, completing the calculation of multiple data points per clock cycle, thus accelerating depthwise convolution processing and reducing system power consumption. The computer system provided in this application also reuses width or height data, further reducing system power consumption. Furthermore, the computer system provided in this application utilizes a data buffer to cache input and output data; the data entering and leaving the data buffer is mainly small-volume fill data, greatly alleviating the bandwidth requirements of the memory.

[0095] Figure 5 This is a schematic diagram illustrating the process of dividing a feature map into sub-maps and processing those sub-maps. Figure 5 It demonstrates a three-layer depthwise convolution.

[0096] For a large size feature map, the feature map can be cut into sub-maps along the height or width direction. Figure 5 The feature map is cut into sub-maps along the height direction.

[0097] Referring to Figure 5 As shown, sub-map [0, i-1] represents the i-1th input sub-map of the first layer of deep convolution, sub-map [0, i] represents the ith input sub-map of the first layer of deep convolution, sub-map [0, i+1] represents the i+1th input sub-map of the first layer of deep convolution, sub-map [1, i-1] represents the i-1th input sub-map of the second layer of deep convolution, sub-map [1, i] represents the ith input sub-map of the second layer of deep convolution, sub-map [1, i+1] represents the i+1th input sub-map of the second layer of deep convolution, sub-map [2, i-1] represents the i-1th input sub-map of the third layer of deep convolution, sub-map [2, i] represents the ith input sub-map of the third layer of deep convolution, and sub-map [2, i+1] represents the i+1th input sub-map of the third layer of deep convolution. After the input sub-map is subjected to one layer of deep convolution to obtain an output sub-map, padding data is obtained from the output sub-map (i.e. output data), and the padding data is stored in an external memory for splicing the input sub-map of the next layer of deep convolution.

[0098] The input sub-map (e.g. sub-map [0, i]) of the first layer of deep convolution can be directly loaded without using padding data for splicing. After one layer of deep convolution of one sub-map (e.g. sub-map [0, i]) is completed, the data buffer writing unit writes the output sub-map (e.g. the output sub-map of sub-map [0, i]) of the sub-map to the data buffer, and the padding data (e.g. the bottom 2 rows of data of the output sub-map of sub-map [0, i-1]) obtained from the output sub-map of the previous sub-map (e.g. sub-map [0, i-1]) of the sub-map is cached from the external memory to the data buffer to splice the output sub-map of the sub-map into the input sub-map (e.g. sub-map [1, i]) of the next layer of deep convolution, and the data buffer reading unit obtains the padding data from the output sub-map of the sub-map cached in the data buffer for splicing use by the output sub-map of the next sub-map (e.g. sub-map [0, i+1]) of the sub-map. For the last layer of deep convolution, since there is no convolution operation afterwards, the padding data can no longer be read out and stored to the external memory. In an embodiment of the present application, when the padding data is spliced with the output sub-map, the padding data is added to the top of the output sub-map. It should be noted that for the first sub-map, the padding data can not need to be read from the external memory and written to the data buffer; and for the last sub-map, the padding data can not need to be read from the data buffer and stored to the external memory.

[0099] Figure 6 is a flowchart for processing the sub-maps in an embodiment of the present application.

[0100] As Figure 6As shown, the chip performs deep convolution on all layers of one sub-graph, and then performs deep convolution on the next sub-graph. For example, the chip performs deep convolution on all layers of the i-1th sub-graph, performs deep convolution on all layers of the i th sub-graph, and then performs deep convolution on all layers of the i+1th sub-graph, and so on.

[0101] In the embodiments of the present application, in addition to the first layer of deep convolution, the main input data and output data (input feature map / input sub-graph and output feature map / output sub-graph) are always stored in the data buffer, and the main data passing in and out of the data buffer is the filling data with a small amount of data. This way can greatly alleviate the bandwidth requirement of the memory.

[0102] In the embodiments of the present application, the chip can perform deep convolution on the input feature map / input sub-graph in a pipeline manner. When the chip performs deep convolution in a pipeline manner, data processing is performed in units of data blocks (data blocks can be represented as tiles). Each input feature map / input sub-graph includes a plurality of data blocks. When the matrix calculation unit performs convolution operation, data processing is performed in units of sub-data blocks, and each data block includes a plurality of sub-data blocks.

[0103] Figure 7 FIG. 1 is a schematic diagram of the relationship among the feature map, the sub-graph, the data block, and the sub-data block in the embodiments of the present application.

[0104] The feature map is the single-layer data to be processed in the convolutional neural network, and has three dimensions of width (W), height (H), and channel number (C). The feature map can be too large in size to be stored in the memory (for example, the data buffer in the embodiments of the present application) of the chip for processing, and needs to be cut into sub-graphs. The cut data can be stored in the memory of the chip and is called a sub-graph. The channel number of the sub-graph is the same as that of the feature map. The feature map can be cut into sub-graphs along the height direction (for example, cut into three sub-graphs along the height direction in the figure), or can be cut into sub-graphs along the width direction.

[0105] The sub-graph can be stored in the memory of the chip, but is not processed at once and needs to be cut into data blocks, which can be represented as tiles. The channel number of the data block is the same as that of the sub-graph and the feature map. If the feature map is cut into sub-graphs along the height direction, the sub-graphs can be cut into data blocks along the width direction; if the feature map is cut into sub-graphs along the width direction, the sub-graphs can be cut into data blocks along the height direction. The controller sends a control instruction to the data buffer reading unit, the data buffer writing unit, the data supply unit, the data write-back unit, and the matrix calculation unit once for each data block processed. After receiving the control instruction, the entire data flow works in a pipeline manner of ping-pong.

[0106] The matrix calculation unit cannot complete the calculation of the entire data block at one time when performing convolution operation, and the data block also needs to be cut into sub-data blocks (sub-data blocks can be represented as slices), and one sub-data block of data is transmitted to the matrix calculation unit every clock cycle. The number of channels of the sub-data block is the same as the number of channels of the data block, the sub-graph and the feature map. The data block can be cut into sub-data blocks along the height direction. In an embodiment of the present application, the height of the sub-data block is 1. For example, as shown in FIG. 6, the width of the sub-data is m+k-1, the height is 1, the number of channels is n, and m<=w. Figure 7

[0107] Figure 8 FIG. 7 is a schematic diagram of performing deep convolution on the data block in the input feature map / input sub-graph in an embodiment of the present application. As shown in FIG. 7, the chip processes one data block in the input feature map / input sub-graph, and then processes the next data block. After processing each data block, each sub-data block of the data block is acquired and processed one by one. After processing all the sub-data blocks of the data block, the next data block is processed. For example, the data block can be moved along the height direction, and each sub-data block of the data block is acquired and processed one by one. Figure 8

[0108] Figure 9 FIG. 8 is a timing relationship diagram of performing deep convolution in a pipeline manner in an embodiment of the present application. The time relationship of data buffer writing unit, data supply unit, matrix calculation unit, data write-back unit, and data buffer reading unit for data processing is shown in the figure. Tile0, Tile1, Tile2, …, Tilez in the figure represent data block 0~data block z.

[0109] The data buffer writing unit shown in the figure refers to that the data buffer writing unit reads padding data from the external memory and writes the read padding data to the data buffer. The data supply unit shown in the figure refers to that the data supply unit transmits input data to the matrix calculation unit. The matrix operation unit shown in the figure refers to that the matrix operation unit performs convolution operation on the input data. The data write-back unit shown in the figure refers to that the data write-back unit writes back the output data to the data buffer. The data buffer reading unit shown in the figure refers to that the data buffer reading unit acquires padding data from the output sub-graph cached in the data buffer and stores the acquired padding data to the external memory.

[0110] Figure 10 FIG. 9 is a flow chart of the data processing method provided by an embodiment of the present application. The data processing method provided by the embodiment of the present application is applied to a computer system, for example, the computer system 10 shown in FIG. 10. Figure 1

[0111] ​​​1001, The controller controls the data buffer write unit to read input data from external memory and write the read input data into the data buffer for buffering.

[0112] The input data stored in the external memory can be a feature map or a sub-map. If the input data stored in the external memory is a sub-map, the data buffer write unit reads from the first sub-map.

[0113] The controller can send a first control command to the data buffer writing unit. The data buffer writing unit reads input data (e.g., the first sub-graph) from the external memory according to the first control command and writes the read input data into the data buffer.

[0114] 1002, write the convolution parameters from external memory to the parameter buffer for caching.

[0115] The computer system may include a DMA controller, which can write convolution parameters to a parameter buffer via DMA.

[0116] 1003. If the input data is a subgraph, the controller controls the data buffer writing unit to read the filling data from the external memory and write the read filling data into the data buffer for caching.

[0117] Since the loading of padding data is not dependent on the depth convolution of the current layer of the subgraph (e.g., the depth convolution of subgraph [0,i]), the data buffer write unit can issue a read request for padding data at the beginning and wait until it is completed before starting the depth convolution of the next layer of the subgraph (e.g., the depth convolution of subgraph [1,i]).

[0118] If the input data read is not a subgraph, a 904 error will be executed.

[0119] The controller can send a second control command to the data buffer write unit. The data buffer write unit reads the fill data from the external memory according to the second control command and generates a fill data write request, and writes the read fill data into the data buffer.

[0120] 1004, The controller controls the data supply unit to send an input data read request to the data buffer read unit.

[0121] In one embodiment of this application, the controller may send a third control command to the data supply unit, and the data supply unit may send an input data read request to the data buffer read unit according to the third control command.

[0122] 1005, The data buffer reading unit reads input data from the data buffer according to the input data reading request, and transmits the input data to the matrix calculation unit through the data supply unit.

[0123] In an embodiment of the present application, the data buffer reading unit reads one sub-data block of the input data from the data buffer each time, and transmits the sub-data block to the matrix calculation unit.

[0124] For example, as shown in FIG. 1, the sub-data block can include (m+k-1)*n data points, and the data buffer reading unit reads one sub-data block from the data buffer each time. Figure 7

[0125] 1006, the controller controls the matrix calculation unit to read the convolution parameter from the parameter buffer, performs convolution operation on the input data according to the convolution parameter to obtain output data, and caches the output data to the output result buffer.

[0126] In an embodiment of the present application, the controller can send a fourth control instruction to the matrix calculation unit, and the matrix calculation unit performs convolution operation according to the fourth control instruction to obtain output data, and stores the output data to the output result buffer.

[0127] In an embodiment of the present application, the matrix calculation unit performs convolution operation on one sub-data block of the input data each clock cycle to obtain the operation result corresponding to the sub-data block.

[0128] 1007, the controller controls the data write-back unit to read the output data cached in the output result buffer, sends the read output data to the data buffer writing unit, and sends an output data writing request to the data buffer writing unit.

[0129] In an embodiment of the present application, the controller can send a fifth control instruction to the data write-back unit, and the data write-back unit reads the output data cached in the output result buffer according to the fifth control instruction, sends the read output data to the data buffer writing unit, and sends an output data writing request to the data buffer writing unit.

[0130] In an embodiment of the present application, the controller can write the operation result of each data block of the input data to the data buffer after performing convolution operation on the data block. Each data block includes a plurality of sub-data blocks.

[0131] 1008, the data buffer writing unit writes the output data to the data buffer according to the output data writing request.

[0132] 1009, if the input data is a sub-graph, the controller controls the data buffer reading unit to generate a padding data reading request, reads new padding data from the output data cached in the data buffer according to the padding data reading request, and stores the new padding data from the data buffer to the external memory.

[0133] ​In an embodiment of the present application, the controller can send a sixth control instruction to the data buffer reading unit, the data buffer reading unit generates a padding data reading request according to the sixth control instruction, reads new padding data from the output data cached in the data buffer according to the padding data reading request, and stores the new padding data from the data buffer to the external memory.

[0134] The embodiment of the present application uses a two-dimensional matrix composed of a matrix calculation unit to perform convolution operation, and the width and channel of the feature map are respectively mapped to two dimensions of the matrix. The embodiment of the present application solves the storage bandwidth problem of convolution operation, improves the utilization rate of the multiplier in the calculation node, and saves the input subgraph and the output subgraph in the data buffer, which can greatly relieve the bandwidth pressure and release the hardware computing power.

[0135] Figure 11 is a structural schematic diagram of the computer system 10 in the embodiment of the present application. As shown in the figure, the electronic device 110 can include: radio frequency (RF) circuit 1101, memory 1102, input unit 1103, display unit 1104, sensor 1105, audio circuit 1106, Wi-Fi module 1107, processor 1108 and power supply 1109 and the like. Those skilled in the art can understand that the structure shown in the embodiment of the present application does not constitute a limitation on the electronic device, and can include more or less components than the figure, or combine certain components, or different component arrangements. Figure 1 Figure 11 The electronic device 110 can include: radio frequency (RF) circuit 1101, memory 1102, input unit 1103, display unit 1104, sensor 1105, audio circuit 1106, Wi-Fi module 1107, processor 1108 and power supply 1109 and the like. Those skilled in the art can understand that the structure shown in the embodiment of the present application does not constitute a limitation on the electronic device, and can include more or less components than the figure, or combine certain components, or different component arrangements. Figure 11

[0136] The RF circuit 1101 can be used to receive and send information or signals during the call process, and in particular, after receiving the downlink information of the base station, it is given to the processor 1108 for processing; in addition, the data related to the uplink is sent to the base station. Usually, the RF circuit 1101 includes, but is not limited to: an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc.

[0137] ​​The memory 1102 can be used to store software programs and modules, and the processor 1108 can execute various function applications and data processing of the electronic device by running the software programs and modules stored in the memory 1102. The memory 1102 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the electronic device (such as audio data, a phone book, etc.), and the like. In addition, the memory 1102 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device.

[0138] The input unit 1103 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device. Specifically, the input unit 1103 can include a touch panel 11031 and other input devices 11032. The touch panel 11031, also known as a touch screen, can collect user touch operations (such as user operations using a finger, a stylus, or any suitable object or accessory near the touch panel 11031) on or near it, and drive the corresponding connection device according to the pre-set program. Optionally, the touch panel 11031 can include two parts of a touch detection device and a touch controller. Among them, the touch detection device detects the touch position of the user and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, and converts it into touch coordinates and sends it to the processor 1108, and receives the command from the processor 1108 and executes it. In addition, the touch panel 11031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 11031, the input unit 1103 can also include other input devices 11032. Specifically, the other input devices 11032 can include one or more of a physical keyboard, a function key (such as a volume control key, an on-off key, etc.), a trackball, a mouse, a joystick, etc.

[0139] The display unit 1104 can be used to display information input by a user or information provided to the user, as well as various menus of the electronic device. The display unit 1104 can include a display panel 11041, which can be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like, optionally. Further, a touch panel 11031 can cover the display panel 11041, which, when detecting a touch operation thereon or adjacent thereto, transmits to the processor 1108 to determine the type of touch event, and then the processor 1108 provides a corresponding visual output on the display panel 11041 according to the type of touch event. Although in the Figure 11 in the above, the touch panel 11031 and the display panel 11041 are implemented as two independent components to realize the input and output functions of the electronic device, but in some embodiments, the touch panel 11031 and the display panel 11041 can be integrated to realize the input and output functions of the electronic device.

[0140] The electronic device can also include at least one sensor 1105, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor can include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 11041 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 11041 and / or the backlight when the electronic device is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and when at rest, it can detect the magnitude and direction of gravity, which can be used for applications that identify the posture of the electronic device (such as landscape / portrait screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), and the like. In addition, the electronic device can also be configured with a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and other sensors, which will not be described here.

[0141] The audio circuit 1106, the speaker 11061, and the microphone 11062 can provide an audio interface between the user and the electronic device. The audio circuit 1106 can convert received audio data into an electrical signal, which is transmitted to the speaker 11061 to be converted into a sound signal for output. On the other hand, the microphone 11062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1106 to be converted into audio data. The audio data is then output to the processor 1108 for processing, and then transmitted to another electronic device via the RF circuit 1101, or output to the memory 1102 for further processing.

[0142] Wi-Fi belongs to short-range wireless transmission technology, and the electronic device 110 can help users send and receive emails, browse web pages, and access streaming media, etc. through the Wi-Fi module 1107, which provides users with wireless broadband Internet access. Although Figure 11 The Wi-Fi module 1107 is shown, but it can be understood that it does not belong to the essential components of the electronic device, and can be omitted as needed without changing the essence of the application.

[0143] The processor 1108 is the control center of the electronic device, which connects all parts of the electronic device through various interfaces and lines, executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 1102 and calling data stored in the memory 1102, thereby overall monitoring the electronic device. Optionally, the processor 1108 can include one or more processing units; preferably, the processor 1108 can integrate an application processor and a modem, wherein the application processor mainly processes the operating system, user interface and application program, etc., and the modem mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 1108.

[0144] The electronic device also includes a power supply 1109 (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 1108 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management, etc. through the power management system.

[0145] Although not shown, the electronic device can also include a camera, a Bluetooth module, etc., which will not be described here.

[0146] Figure 11 The electronic device described in the foregoing embodiments can be used to implement the data processing method introduced in the embodiments of the present application, which can be referred to the related description in the foregoing embodiments, and will not be described here.

[0147] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any change or replacement within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A computer system, characterized by The matrix calculation unit includes two-dimensional calculation nodes, and is used for: performing convolution operation on the input data according to the convolution parameters, and each calculation node completes calculation of a single convolution window; the computer system further includes an external memory used for storing the input data and the convolution parameters; the computer system further includes a controller, a data buffer write unit, a data buffer read unit, a data supply unit, a data write-back unit, a data buffer, a parameter buffer, an output result buffer, the parameter buffer is connected with the external memory and used for caching the convolution parameters; the controller is used for controlling the data buffer write unit to read the input data from the external memory and write the read input data into the data buffer for caching; the controller is further used for controlling the data supply unit to send an input data read request to the data buffer read unit; the data buffer read unit is used for reading the input data from the data buffer according to the input data read request and transmitting the input data to the matrix calculation unit through the data supply unit; the controller is further used for controlling the matrix calculation unit to read the convolution parameters from the parameter buffer, performing convolution operation on the input data according to the convolution parameters to obtain output data, and caching the output data to the output result buffer; the controller is further used for controlling the data write-back unit to read the output data cached in the output result buffer, sending the read output data to the data buffer write unit, and sending an output data write request to the data buffer write unit; the data buffer write unit is further used for writing the output data into the data buffer according to the output data write request.

2. The computer system of claim 1, wherein, The number of columns of the matrix calculation unit is equal to the number of channels of the input data.

3. The computer system of claim 1, wherein, The input data includes subgraphs obtained by splitting feature maps, the external memory is further used for storing padding data; the controller is further used for controlling the data buffer write unit to read the padding data from the external memory and generate a padding data write request, and writing the read padding data into the data buffer according to the padding data write request; the controller is further used for controlling the data buffer read unit to generate a padding data read request and read new padding data from the output data cached in the data buffer according to the padding data read request, and storing the new padding data from the data buffer to the external memory.

4. The computer system of claim 3, wherein, The data buffer write unit is further used for performing write conflict checking on the output data write request and the padding data write request.

5. The computer system of claim 4, wherein, The priority of the output data write request is higher than that of the padding data write request.

6. The computer system of claim 3, wherein, The data buffer write unit is further used for performing read conflict checking on the input data read request and the padding data read request.

7. The computer system of claim 6, wherein, The priority of the input data read request is higher than that of the padding data read request.

8. The computer system of claim 3, wherein, The sub-graphs are split along the height direction or the width direction of the feature map.

9. The computer system of any one of claims 1 to 8, wherein, The computer system performs deep convolution on the input data in a pipeline manner.

10. The computer system of claim 9, wherein, The input data is split into data blocks along the width direction or the height direction, and the data blocks are split into sub-data blocks along the height direction or the width direction, and the matrix calculation unit performs convolution operation on one sub-data block per clock cycle.

11. The computer system of claim 10, wherein, The height of the sub-data block is 1.

12. A data processing method applied to a computer system, said computer system comprising a matrix computing unit composed of two-dimensional computing nodes, characterized in that, The method comprises: The matrix calculation unit performs convolution operation on the input data according to the convolution parameters, and each calculation node completes the calculation of a single convolution window; The computer system further comprises an external memory for storing the input data and the convolution parameters; The computer system further comprises a controller, a data buffer write unit, a data buffer read unit, a data supply unit, a data write-back unit, a data buffer, a parameter buffer, an output result buffer, the parameter buffer is connected with the external memory and used for caching the convolution parameters, and the method further comprises: The controller controls the data buffer write unit to read the input data from the external memory and write the read input data into the data buffer for caching; The controller controls the data supply unit to send an input data read request to the data buffer read unit; The data buffer read unit reads the input data from the data buffer according to the input data read request and transmits the input data to the matrix calculation unit through the data supply unit; The controller controls the matrix calculation unit to read the convolution parameters from the parameter buffer, performs convolution operation on the input data according to the convolution parameters to obtain output data, and caches the output data to the output result buffer; The controller controls the data write-back unit to read the output data cached in the output result buffer, sends the read output data to the data buffer write unit, and sends an output data write request to the data buffer write unit; The data buffer write unit writes the output data into the data buffer for caching according to the output data write request.

13. The data processing method of claim 12, wherein, The number of columns of the matrix calculation unit is equal to the number of channels of the input data.

14. The data processing method of claim 12, wherein, The input data comprises sub-graphs split from a feature map, and the external memory is further used for storing padding data. The method further comprises: The controller controls the data buffer write unit to read the padding data from the external memory and generate a padding data write request, and writes the read padding data into the data buffer according to the padding data write request; The controller controls the data buffer read unit to generate a padding data read request, reads new padding data from the output data cached in the data buffer according to the padding data read request, and stores the new padding data from the data buffer to the external memory.

15. The data processing method of claim 14, wherein, The method further comprises: The data buffer write unit is further used for write conflict checking on the output data write request and the padding data write request.

16. The data processing method of claim 15, wherein, The priority of the output data write request is higher than the priority of the padding data write request.

17. The data processing method of claim 14, wherein, The method further comprises: The data buffer write unit performs a read conflict check on the input data read request and the padding data read request.

18. The data processing method of claim 17, wherein, The priority of the input data read request is higher than the priority of the padding data read request.

19. The data processing method of claim 14, wherein, The sub-graph is obtained by cutting along the height direction or the width direction of the feature map.

20. The data processing method according to any one of claims 12 to 19, wherein, The data processing method performs deep convolution on the input data in a pipeline manner.

21. The data processing method of claim 20, wherein, The input data is cut into data blocks along the width direction or the height direction, and the data blocks are cut into sub-data blocks along the height direction or the width direction, and the matrix calculation unit performs convolution operation on one sub-data block per clock cycle.

22. The data processing method of claim 21, wherein, The height of the sub-data block is 1.

Images