A data processing method, device and storage medium
By directly reading data from off-chip memory units and performing deep convolution operations in registers, combining data slicing and cache unit filling, the problem of being limited by shared memory in deep convolution operations is solved, and performance and efficiency are improved.
Patent Information
- Application Number
- CN202410964622.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-07-18
AI Technical Summary
The prior art is limited by the size of the shared memory when performing deep convolution operations, resulting in an increase in the time for repeatedly reading input data from the shared memory to the register, and if the input data is too large, it is susceptible to the size of the shared memory.
By directly reading the input data from the off-chip memory unit to the register and performing convolution operations in the register, avoiding the size of shared memory, reducing the number of times data is read from shared memory, and dividing the data to Adapt to the register size, data filling is used with the cache unit to reduce the number of reads from off-chip storage.
It significantly improves the performance of the operator, avoids shared memory limitations, reduces the number of data reads, and improves data processing efficiency.
Smart Images

Figure CN118673260B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate to the field of data processing technology, and in particular, to a data processing method, a data processing device, and a storage medium. Background Art
[0002] Deep Convolutional Neural Networks (DCNN) is a deep learning model widely used in the field of computer vision. Through multi-layer convolution and pooling operations, it can automatically learn the abstract features of images and achieve excellent performance in tasks such as image classification, object detection, and semantic segmentation. With the continuous development of artificial intelligence technology, deep convolutional neural networks will play an important role in more fields and promote the further development of computer vision.
[0003] Deep convolution is a convolution method that performs convolution operations on each channel separately, and the number of channels of the output feature map is the same as the input. Deep convolution is an extreme case of group convolution, which divides each channel of the input feature into a group, that is, the number of groups is equal to the number of input channels. Deep convolution can construct a deep convolutional neural network through local receptive field and weight sharing. Summary of the invention
[0004] At least one embodiment of the present disclosure provides a data processing method for a convolution operation, wherein the size of a convolution kernel of the convolution operation is (dw, dh), dw represents the number of columns of the convolution kernel, and dh represents the number of rows of the convolution kernel, and the method comprises: reading m columns of input data from M columns of input data of an off-chip storage unit into a register, wherein M is an integer greater than 0, m is an integer greater than 0 and less than or equal to M, and m>= dw; performing a convolution operation on an i-th column of input data among the m columns of input data, and after the convolution operation on the i-th column of input data is completed, deleting k columns of input data corresponding to the i-th column of input data from the register, wherein i is an integer greater than 0 and less than or equal to m, and k is determined according to the step size of the convolution operation and the size of data processed in parallel by a single instruction of the convolution operation; rereading k columns of input data from the off-chip storage unit into the register, and multiplexing the mk columns of input data remaining after the convolution operation is performed on the i-th column of input data, and performing the convolution operation on the i+1-th column of input data.
[0005] For example, at least one embodiment of the present disclosure provides a data processing method, further comprising: judging whether the input data needs to be segmented according to the size of the input data and the size of the register, wherein the size of the input data is (WI, HI), wherein WI represents the number of columns of the input data and WI=M, and HI represents the number of rows of the input data; if the size of the register is greater than or equal to HI dw, then there is no need to segment the input data; if the size of the register is smaller than HI dw, the input data is divided into T data blocks so that the size of the register is greater than or equal to (HI / T) dw, wherein the size of each data block of the input data is (WI, (HI / T)); wherein T is an integer greater than 1.
[0006] For example, in the data processing method provided by at least one embodiment of the present disclosure, after the input data is divided into T data blocks, the method further includes: storing adjacent j rows of input data of two adjacent data blocks in the T data blocks into a cache unit, wherein j is an integer greater than 0 and is determined according to the step size of the convolution operation; and performing the convolution operation on the T data blocks in sequence.
[0007] For example, the data processing method provided by at least one embodiment of the present disclosure also includes: when the convolution kernel moves to the upper boundary of the tth data block, the upper boundary corresponding to the tth data block is read from the cache unit and filled into the register; when the convolution kernel moves to the lower boundary of the tth data block, the lower boundary corresponding to the tth data block is read from the cache unit and filled into the register; wherein, for the tth data block, the j rows of input data adjacent to the tth data block in the t-1th data block are used as the upper boundary filling of the tth data block; the j rows of input data adjacent to the tth data block in the t+1th data block are used as the lower boundary filling of the tth data block, wherein t is an integer greater than 1 and less than T.
[0008] For example, in the data processing method provided by at least one embodiment of the present disclosure, when a convolution operation is performed on the tth data block among the T data blocks, the method includes: reading m columns of input data in the tth data block from the off-chip storage unit to the register; performing the convolution operation in the column direction on the i-th column of input data of the m columns of input data in the tth data block; when the convolution kernel moves to the lower boundary of the i-th column of input data in the column direction, reading the lower boundary corresponding to the tth data block from the cache unit and filling it into the register to perform the convolution operation on the i-th column of input data; after the convolution operation on the i-th column of input data is completed, deleting the k columns of input data corresponding to the i-th column of input data from the register, re-reading the k columns of input data in the tth data block from the off-chip storage unit to the register, and reusing the mk columns of input data remaining after the convolution operation on the i-th column of input data, and performing the convolution operation on the i+1-th column of input data.
[0009] For example, the data processing method provided by at least one embodiment of the present disclosure also includes: when the convolution kernel moves to the upper boundary of the i-th column of input data, the upper boundary corresponding to the t-th data block is read from the cache unit and filled into the register to perform the convolution operation of the i-th column of input data.
[0010] For example, in the data processing method provided by at least one embodiment of the present disclosure, the convolution operation is a depth-wise convolution operation.
[0011] At least one embodiment of the present disclosure further provides a data processing device for a convolution operation, wherein the size of a convolution kernel of the convolution operation is (dw, dh), wherein dw represents the number of columns of the convolution kernel, and dh represents the number of rows of the convolution kernel, and the device comprises: a reading unit configured to read m columns of input data from M columns of input data of an off-chip storage unit into a register, wherein M is an integer greater than 0, m is an integer greater than 0 and less than or equal to M, and m> =dw; a first processing unit, configured to perform a convolution operation on the i-th column of input data among the m columns of input data, and after the convolution operation on the i-th column of input data is completed, delete the k-th column of input data corresponding to the i-th column of input data from the register, wherein i is an integer greater than 0 and less than or equal to m, and k is determined according to the step size of the convolution operation and the size of data processed in parallel by a single convolution operation instruction; a second processing unit, configured to re-read the k-th column of input data from the off-chip storage unit into the register, and reuse the mk-th column of input data remaining after the convolution operation on the i-th column of input data, and perform the convolution operation on the i+1-th column of input data.
[0012] For example, at least one embodiment of the present disclosure provides a data processing device, further comprising a judgment unit, configured to: judge whether the input data needs to be segmented according to the size of the input data and the size of the register, wherein the size of the input data is (WI, HI), wherein WI represents the number of columns of the input data and WI=M, and HI represents the number of rows of the input data; if the size of the register is greater than or equal to HI dw, then there is no need to segment the input data; if the size of the register is smaller than HI dw, the input data is divided into T data blocks so that the size of the register is greater than or equal to (HI / T) dw, wherein the size of each data block of the input data is (WI , (HI / T)); wherein T is an integer greater than 1.
[0013] For example, the data processing device provided by at least one embodiment of the present disclosure also includes a cache unit, which is configured to: store adjacent j rows of input data of two adjacent data blocks in the T data blocks into the cache unit, where j is an integer greater than 0 and is determined according to the step size of the convolution operation; and perform the convolution operation on the T data blocks in sequence.
[0014] For example, in the data processing device provided by at least one embodiment of the present disclosure, the convolution operation is a depth-wise convolution operation.
[0015] At least one embodiment of the present disclosure further provides a data processing device, comprising: a processor; a memory; and one or more computer program modules, wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include instructions for executing the data processing method provided by any embodiment of the present disclosure.
[0016] At least one embodiment of the present disclosure further provides a storage medium that non-temporarily stores computer-readable instructions, and when the computer-readable instructions are executed by a computer, the data processing method provided by any embodiment of the present disclosure is executed. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.
[0018] Figure 1 A schematic diagram of input data reuse based on shared memory is shown.
[0019] Figure 2A schematic diagram of a data processing method provided by at least one embodiment of the present disclosure is shown.
[0020] Figure 3 A flowchart of a data processing method provided for at least one embodiment of the present disclosure.
[0021] Figure 4 A schematic diagram of data segmentation provided by at least one embodiment of the present disclosure is shown.
[0022] Figure 5 A schematic diagram of repeated loading of data near a split point provided in at least one embodiment of the present disclosure.
[0023] Fig. 6A At least one embodiment of the present disclosure provides a method for processing 3 columns of input data and a convolution kernel of 3 3. Schematic diagram of a deep convolution operation with a convolution operation step size of 1 and a single convolution operation instruction processing one column of data in parallel in the row direction.
[0024] Figure 6B At least one embodiment of the present disclosure provides a method for processing 3 columns of input data and a convolution kernel of 3 3. Schematic diagram of a deep convolution operation with a convolution operation step size of 2 and a single convolution operation instruction processing one column of data in parallel in the row direction.
[0025] Figure 6C At least one embodiment of the present disclosure provides a method for 5 columns of input data with a convolution kernel of 5 5. Schematic diagram of a deep convolution operation with a convolution operation step size of 2 and a single convolution operation instruction processing one column of data in parallel in the row direction.
[0026] Fig.6D At least one embodiment of the present disclosure provides a method for processing 4 columns of input data and a convolution kernel of 3 3. Schematic diagram of a deep convolution operation with a convolution operation step size of 1 and a single convolution operation instruction processing 2 columns of data in parallel in the row direction.
[0027] Fig. 6E A schematic block diagram of a data processing device provided for at least one embodiment of the present disclosure.
[0028] Figure 7 A schematic block diagram of another data processing device provided for at least one embodiment of the present disclosure.
[0029] Figure 8 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.
[0030] Fig. 9 A schematic diagram of a storage medium provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0032] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0033] In the related art, since the shared memory is located on the chip, the reading speed and storage speed are much faster than those outside the chip. Therefore, the data that needs to be loaded repeatedly is usually placed in the shared memory first, and then the required data is read from the shared memory into the register to perform the corresponding calculation operation. Figure 1 A schematic diagram of input data reuse based on shared memory is shown, Figure 1 As shown in the figure, for a depthwise convolution operation, it is usually necessary to first load n (n is an integer greater than 0) rows of input data into the shared memory. After consuming the first q (q is an integer greater than 0) rows of input data of the n rows of input data, the next q rows of input data are loaded into the shared memory for register reading and subsequent depthwise convolution operations.
[0034] The inventors have noticed that, on the one hand, the above method increases the time for repeatedly reading input data from the shared memory to the register; on the other hand, if the input data is too large, the above method is easily limited by the size of the shared memory.
[0035] At least one embodiment of the present disclosure provides a data processing method for a convolution operation, wherein the size of a convolution kernel of the convolution operation is (dw, dh), wherein dw represents the number of columns of the convolution kernel, and dh represents the number of rows of the convolution kernel. The method comprises: reading m columns of input data from M columns of input data of an off-chip storage unit into a register, wherein M is an integer greater than 0, m is an integer greater than 0 and less than or equal to M, and m> dw; performing a convolution operation on an i-th column of input data among the m columns of input data, and after the convolution operation on the i-th column of input data is completed, deleting k columns of input data corresponding to the i-th column of input data from the register, wherein i is an integer greater than 0 and less than or equal to m, and k is determined according to the step size of the convolution operation and the size of data processed in parallel by a single convolution operation instruction; rereading k columns of input data from the off-chip storage unit into the register, and multiplexing the mk columns of input data remaining after the convolution operation is performed on the i-th column of input data, and performing the convolution operation on the i+1-th column of input data.
[0036] At least one embodiment of the present disclosure further provides a data processing device, an electronic device, and a storage medium corresponding to the above-mentioned data processing method.
[0037] The data processing method provided by at least one embodiment of the present disclosure directly reads input data from an off-chip storage unit through a register to perform a convolution operation, thereby avoiding the problem of being limited by the size of the shared memory and reducing the time for repeatedly reading input data from the shared memory to the register. In addition, the read input data can be reused to reduce the number of times data is read from the off-chip storage, which can significantly improve the performance of the operator.
[0038] The embodiments of the present disclosure and some examples thereof are described in detail below with reference to the accompanying drawings.
[0039] At least one embodiment of the present disclosure provides a data processing method. For example, input data can be directly read from an off-chip storage unit through a register to perform a convolution operation. In addition, the input data can be multiplexed to reduce the number of times data is read from the off-chip storage, thereby significantly improving the performance of the operator.
[0040] For example, the data processing method can be implemented in the form of software, hardware, firmware or any combination thereof, and loaded and executed by a processor in a device such as a mobile phone, a digital camera, a tablet computer, a laptop computer, a desktop computer, a network server, etc. to improve the performance of the operator.
[0041] For example, the data processing method is applicable to a computing device, which includes any electronic device with computing functions, such as a mobile phone, a digital camera, a laptop computer, a tablet computer, a desktop computer, a network server, etc., which can load and execute the data processing method, and the embodiments of the present disclosure are not limited to this. For example, the computing device may include a central processing unit (CPU) or a graphics processing unit (GPU), a digital signal processor (DSP), and other processing units with data processing capabilities and / or instruction execution capabilities, storage units, etc., and the computing device is also installed with an operating system, an application programming interface (for example, OpenGL (Open Graphics Library), Metal, etc.), etc., and the data processing method provided by the embodiments of the present disclosure is implemented by running codes or instructions.
[0042] Figure 2 A schematic diagram of a data processing method provided by at least one embodiment of the present disclosure is shown. For example, the data processing method is used for a convolution operation, and the size of the convolution kernel of the convolution operation is (dw, dh), for example, dw represents the number of columns of the convolution kernel (i.e., the width of the convolution kernel), and dh represents the number of rows of the convolution kernel (i.e., the height of the convolution kernel).
[0043] For example, in some examples, the convolution operation in the embodiments of the present disclosure is a deep convolution operation, and of course it can also be other neural network convolution operations, which can be determined according to the actual situation, and the embodiments of the present disclosure do not limit this. The following is an introduction using the convolution operation as a deep convolution operation as an example, and the embodiments of the present disclosure do not limit this.
[0044] For example, Figure 2 As shown, the data processing method includes steps S110 to S130.
[0045] Step S110: Read m columns of input data from M columns of input data of the off-chip storage unit into the register, wherein M is an integer greater than 0, m is an integer greater than 0 and less than or equal to M, and m>= dw.
[0046] Step S120: perform a convolution operation on the i-th column of input data among the m-column input data, and after the convolution operation on the i-th column of input data is completed, delete the k-th column of input data corresponding to the i-th column of input data from the register, where i is an integer greater than 0 and less than or equal to m, and k is determined according to the step size of the convolution operation and the size of data processed in parallel by a single convolution operation instruction.
[0047] Step S130: re-read k columns of input data from the off-chip storage unit into the register, reuse the mk columns of input data remaining after the convolution operation is performed on the i-th column of input data, and perform the convolution operation on the i+1-th column of input data.
[0048] For step S110, for example, the total input data (M columns of input data) is stored in an off-chip storage unit, and m columns of input data in the M columns of input data are read from the off-chip storage unit to the register unit. In some examples of the present disclosure, the register unit may be an extended thread local register (Thread-Local Register, TLR for short), which is closest to the computing unit and has a smaller capacity; the cache unit may be a global shared memory (Group-Shared Memory, GSM for short), which is farther from the computing unit than the TLR is from the computing unit. Of course, it may also be other units that can realize data storage, and the embodiments of the present disclosure are not limited to this.
[0049] The following description is made by taking the register unit as an extended thread local register TLR and the cache unit as a global shared memory GSM as an example, but the embodiments of the present disclosure are not limited to this.
[0050] For example, in an embodiment of the present disclosure, the value of m is greater than or equal to the width dw of the convolution kernel, thereby ensuring that a depthwise convolution operation can be performed based on m columns of input data read into the register.
[0051] For example, in an embodiment of the present disclosure, by directly reading m columns of input data from an off-chip storage unit into a register, the problem of being limited by the size of the shared memory can be avoided and the time for repeatedly reading input data from the shared memory into the register can be reduced.
[0052] For step S120, i=1, that is, the first column of input data is used as an example for description. For example, a depthwise convolution operation is performed on the first column of input data in the m columns of input data, and after the depthwise convolution operation on the first column of input data is completed, the k columns of input data corresponding to the first column of input data are deleted from the register.
[0053] Fig.6D At least one embodiment of the present disclosure provides a method for processing 4 columns of input data and a convolution kernel of 3 3. Schematic diagram of a deep convolution operation with a convolution operation step size of 1 and a single convolution operation instruction processing 2 columns of data in parallel in the row direction. For example, refer to Fig.6D , when the step size of the depthwise convolution operation is 1, and a single depthwise convolution operation instruction processes two columns of data in parallel in the row direction, after completing the first depthwise convolution operation in the column direction, the two columns of input data (i.e. Fig.6DThat is, the depth convolution operation of the first two columns of input data will be completed at the same time. Therefore, the first and second columns of data that complete the depth convolution operation have been consumed and will not be used again in subsequent depth convolutions. Therefore, the two columns of input data corresponding to the first column of input data can be deleted from the register, that is, the first and second columns of input data that complete the depth convolution operation are deleted.
[0054] Fig. 6A At least one embodiment of the present disclosure provides a method for processing 3 columns of input data and a convolution kernel of 3 3. Schematic diagram of a deep convolution operation with a convolution operation step size of 1 and a single convolution operation instruction processing 1 column of data in parallel in the row direction. For example, refer to Fig. 6A , when the step size of the depthwise convolution operation is 1, and a single depthwise convolution operation instruction processes one column of data in parallel in the row direction, then after completing one depthwise convolution operation in the column direction, the depthwise convolution operation on one column of input data (i.e., the first column) can be implemented, that is, the first column of data that completes the depthwise convolution operation has been consumed and will no longer be used in subsequent depthwise convolutions, so that the one column of input data corresponding to the first column of input data can be deleted from the register, that is, the first column of input data that completes the convolution operation is deleted.
[0055] Therefore, the k columns of input data corresponding to the i-th column of input data that need to be deleted from the register are determined according to the step size of the convolution operation and the data size processed in parallel by a single convolution operation instruction (ie, the number of data columns processed in parallel).
[0056] For step S130, for example, after deleting the first k columns of input data from the register, the remaining input data in the register, i.e., the remaining mk columns of input data, are retained and reused, and new k columns of data that are continuous with the previous m columns of data are read from the off-chip storage into the register to start performing convolution of a new column. The above steps are repeated until all depthwise convolutions are completed.
[0057] It should be noted that in the embodiments of the present disclosure, for example, if the data size (for example, the number of data columns) processed in parallel by a single convolution operation instruction is Y, and Y is an integer greater than 1, the i-th column of input data is subjected to a convolution operation, that is, multiple columns of data (i-th to i+(Y-1)-th columns of input data) subjected to convolution operations at the same time are taken as the i-th column (because when the data size processed in parallel by a single convolution operation instruction is Y, the i-th to i+(Y-1)-th columns will simultaneously undergo a depth convolution operation), and the i+1-th column of input data is the i+Y-th to i+2Y-1-th columns of data. Therefore, the i-th column here corresponds to multiple columns of data that are currently being executed in parallel for a depth convolution operation, and the i+1-th column corresponds to multiple columns of data that will be executed in parallel for the next depth convolution operation.
[0058] Figure 3 A flowchart of a data processing method provided by at least one embodiment of the present disclosure. Figure 3 As shown, the data processing method also includes the following steps.
[0059] Step S210: Determine whether the input data needs to be segmented according to the size of the input data and the size of the register.
[0060] For example, the size of the input data is (WI, HI), where WI represents the number of columns of the input data (ie, the width of the input data) and WI=M, and HI represents the number of rows of the input data (ie, the height of the input data).
[0061] Step S220: If the size of the register is greater than or equal to HI dw, there is no need to split the input data.
[0062] For example, HI dw indicates the input data that needs to be loaded into the register to complete the deep convolution operation of one column or several columns of input data. The input data of size dw can all be put into the register without overflow, that is, the size of the register can ensure that the input data that needs to be read in when performing the depthwise convolution operation in each column does not need to be split. Otherwise, the input data needs to be split into T data blocks, so that (HI / T) All data of size dw can be put into the register without overflow to ensure the execution of the depthwise convolution operation.
[0063] Step S230: If the register size is smaller than HI dw, the input data is divided into T data blocks so that the register size is greater than or equal to (HI / T) dw.
[0064] For example, the size of each block of the input data is (WI , (HI / T)); where T is an integer greater than 1.
[0065] For example, if the register size is smaller than HI dw, that is, when the size of the register cannot store the data required for performing the convolution operation of one column or several columns of input data in parallel, the input data needs to be split into T data blocks: data block 1, data block 2, ..., data block T, and the split point is recorded, for example, the split point is the number of rows at the split position, etc., to determine the upper boundary padding and lower boundary padding corresponding to the data block. Figure 4 As shown in the figure, the width and height of each data block after segmentation are WI and (HI / T) respectively.
[0066] For example, based on the split data blocks, registers can be loaded from off-chip storage (HI / T) dw size data to perform depth convolution operations.
[0067] For example, after the input data is divided into T data blocks, the method also includes: storing adjacent j rows of input data (i.e., input data near the dividing point) of two adjacent data blocks in the T data blocks as upper boundary filling or lower boundary filling in a cache unit, where j is an integer greater than 0 and is determined according to the step size of the convolution operation; and performing the convolution operation on the T data blocks in sequence.
[0068] Figure 5 A schematic diagram of repeated loading of data near a split point provided in at least one embodiment of the present disclosure. Figure 5 As shown, for the t-th data block, the j rows of input data adjacent to the t-1-th data block are used as the upper boundary padding of the t-th data block; the j rows of input data adjacent to the t+1-th data block are used as the lower boundary padding of the t-th data block. For example, the upper boundary padding and the lower boundary padding corresponding to the t-th data block are stored in the cache unit, and when needed in the deep convolution operation, they are loaded from the cache unit to the register to implement the execution of the deep convolution operation.
[0069] For example, a depth convolution operation is performed on each data block in turn. The method of performing the depth convolution operation on each data block is described in detail below for the tth data block. For example, if the size of the register is smaller than HI dw, the input data needs to be divided into T data blocks, and there is a split point, which will cause the repeated loading of several lines of input data above and below the split point (for example, the j lines mentioned above). Therefore, in the embodiment of the present disclosure, the data that needs to be repeatedly loaded (for example, the upper boundary padding and lower boundary padding corresponding to the t-th data block, and the j lines of input data corresponding to the t-th data block and the data blocks adjacent to it) are stored in the cache unit in advance, such as Figure 5 As shown in the figure, when the 3x3 convolution kernel traverses to the lower boundary of the t-th data block, it is necessary to load the upper two rows of the t+1-th data block from the cache unit (that is, fill the lower boundary of the t-th data block) to the register to complete the convolution operation of the bottom two rows of the t-th data block; when the 3x3 convolution kernel traverses to the upper boundary of the t-th data block, it is necessary to load the upper two rows of the t-1-th data block from the cache unit (that is, fill the upper boundary of the t-th data block) to the register to complete the convolution operation of the top two rows of the t-th data block.
[0070] In this example, by storing only the data near the split point in the cache unit, the memory requirements of the cache unit can be reduced. At the same time, by effectively utilizing the space and bandwidth advantages of the cache unit, the number and time of reading data from off-chip storage are reduced, thereby improving operator performance.
[0071] For example, Figure 3 As shown, when a convolution operation is performed on the tth data block among T data blocks, the method includes the following steps.
[0072] Step S240: reading m columns of input data in the tth data block from the off-chip storage unit to the register.
[0073] For example, in this example, when the value of m is the width dw of the convolution kernel, the size of the input data read into the register is (HI / T) The following description will be made by taking m=dw as an example, but m may also take other values greater than dw, which is not limited in the embodiments of the present disclosure.
[0074] Fig. 6A At least one embodiment of the present disclosure provides a method for processing 3 columns of input data and a convolution kernel of 3 3. Schematic diagram of a deep convolution operation with a convolution operation step size of 1 and a single convolution operation instruction processing one column of data in parallel in the row direction. For example, Fig. 6A As shown, the following takes the first data block, i.e. t=1; the convolution kernel size is 3x3 (i.e. dw=3, dh=3), and the step size of the convolution operation is 1 as an example. Therefore, Fig. 6A In the example shown, three columns of input data are loaded (ie, read) from an off-chip storage into a register to perform a depthwise convolution operation, but the embodiments of the present disclosure are not limited to this.
[0075] Step S250: performing the convolution operation in the column direction on the i-th column of input data of the m-column input data in the t-th data block.
[0076] Step S260: Determine whether the convolution kernel has moved to the lower boundary of the t-th data block. When the convolution kernel moves to the lower boundary of the i-th column input data in the column direction, read the lower boundary corresponding to the t-th data block from the cache unit and fill it into the register to perform the convolution operation of the i-th column input data; after the convolution operation of the i-th column input data is completed, delete the k-th column input data corresponding to the i-th column input data from the register.
[0077] For example, Fig. 6AAs shown, the convolution kernel is moved in the column direction for the first data block to perform a depthwise convolution operation until the convolution kernel has moved to the lower boundary of the first data block. When the convolution kernel moves in the column direction to the lower boundary of the first column of input data, the lower boundary corresponding to the first data block is read from the cache unit and filled into the register to perform a convolution operation on the first column of input data. For example, Figure 5 As shown, the lower boundary padding is read from the global shared memory GSM, that is, the upper 2 rows of the second data block adjacent to it (that is, the adjacent 2 rows of the second data block adjacent to the first data block, that is, j=2) are read into the register to complete the depth convolution operations of the last 2 rows and the last row of data of the first data block (that is, the data near the split point) respectively.
[0078] At this point, the first column of data in the first data block in the register has been consumed (that is, the deep convolution operation is completed), and subsequent deep convolution operations will no longer use the first column of data. Therefore, after the convolution operation on the first column of input data is completed, the first column of input data is deleted from the register to free up space in the register to continue the deep convolution operation on subsequent data.
[0079] Step S270: The convolution kernel moves k columns of data in the row direction.
[0080] For example, Figures 6A-6D As shown, after the deep convolution operation of the first column of data of the first data block is performed, the convolution kernel moves k columns of data in the row direction and discards the first k columns of data in the register, for example, Fig. 6A In the scenario, since the convolution operation step size is 1 and a single convolution operation instruction processes 1 column of data in parallel in the row direction, the value of k is equal to 1, so the 1st column of data is discarded from the register, and the new k columns of input data are read into the register to continue the depthwise convolution operation of the next column of data. Fig. 6A As shown, the value of k is equal to 1, so a new column of data, that is, the 4th column of data, is read sequentially into the register. At this time, the data is moved 1 column in the row direction, that is, the depth convolution operation is continued on the 2nd column of data based on the 2nd, 3rd, and 4th columns of data.
[0081] Step S280: Determine whether the convolution kernel has moved to the right boundary of the tth data block.
[0082] Step S290: If no, discard the first k columns of data in the register, re-read the k columns of input data in the tth data block from the off-chip storage unit into the register, and reuse the remaining mk columns of input data (i.e., the second column of data and the third column of data) after the convolution operation on the i-th column of input data, and perform the convolution operation on the i+1-th column of input data; if yes, the convolution operation is completed.
[0083] For example, Fig. 6A As shown, a new column of data, namely the 4th column of data, is sequentially read into the register, and the 2nd column of data and the 3rd column of data are multiplexed, that is, the depth convolution operation is continued to be performed on the 2nd column of data based on the 2nd, 3rd, and 4th column of data.
[0084] For example, the above steps S230-S270 are repeated until the convolution kernel has moved to the right boundary of the t-th data block, at which point the depth convolution of the t-th data block has been completed.
[0085] It should be noted that if the input data does not need to be split, the size of the data in the register for performing convolution calculation is HI dw, and only needs to read data from off-chip storage to ensure that the size of the data currently performing the depth convolution operation in the register is HI dw is sufficient. Since it has no split point, it does not need to read the lower boundary fill or the upper boundary fill from the cache unit.
[0086] In addition, for the multiple data blocks that are segmented, it is not necessary to read the upper boundary padding for the first data block, and it is not necessary to read the lower boundary padding for the last data block.
[0087] Figure 6B At least one embodiment of the present disclosure provides a method for processing 3 columns of input data and a convolution kernel of 3 3. Schematic diagram of a deep convolution operation with a convolution operation step size of 2 and a single convolution operation instruction processing one column of data in parallel in the row direction. For example, Figure 6B As shown in the figure, when the convolution kernel size is 3x3 (ie, dw=3, dh=3), and the step size of the convolution operation is 2, after moving the convolution kernel in the column direction to the lower boundary, the 1st and 2nd columns of input data are consumed at the same time (ie, the 1st and 2nd columns of input data complete the depth convolution operation at the same time), the convolution kernel moves 2 columns of data in the row direction, that is, Figure 6B In the example shown, since the convolution operation stride is 2 and a single convolution operation instruction processes 1 column of data in parallel in the row direction, k=2. Figure 6B In the example shown, after the depth convolution operation of the first column and the second column of the first data block is performed, the convolution kernel moves k columns of data in the row direction and discards the first k columns of data in the register, for example, Figure 6BIn the scenario, the value of k is equal to 2, so the 1st and 2nd columns of data are discarded, and the new k columns of input data are read into the register to continue the deep convolution operation of the next column of data. Since the value of k is equal to 2, the new k columns of data, that is, the 4th and 5th columns of data are read into the register in sequence. At this time, 2 columns of data are moved in the row direction, that is, the deep convolution operation is continued for the 3rd and 4th columns of data based on the 3rd, 4th, and 5th columns of data.
[0088] Figure 6C At least one embodiment of the present disclosure provides a method for 5 columns of input data with a convolution kernel of 5 5. Schematic diagram of a deep convolution operation with a convolution operation step size of 2 and a single convolution operation instruction processing one column of data in parallel in the row direction. For example, Figure 6C As shown in the figure, when the convolution kernel size is 5x5 (ie, dw=5, dh=5), and the step size of the convolution operation is 2, after moving the convolution kernel in the column direction to the lower boundary, the 1st and 2nd columns of input data are consumed at the same time (ie, the 1st and 2nd columns of input data complete the depth convolution operation at the same time), the convolution kernel moves 2 columns of data in the row direction, that is, Figure 6C In the example shown, since the convolution operation stride is 2 and a single convolution operation instruction processes 1 column of data in parallel in the row direction, k=2. Figure 6C In the example shown, after the depth convolution operation of the first column and the second column of the first data block is performed, the convolution kernel moves k columns of data in the row direction and discards the first k columns of data in the register, for example, Figure 6C In the scenario, the value of k is equal to 2, so the 1st and 2nd columns of data are discarded, and the new k columns of input data are read into the register to continue the deep convolution operation of the next column of data. Since the value of k is equal to 2, the new k columns of data, that is, the 6th and 7th columns of data are read into the register in sequence. At this time, 2 columns of data are moved in the row direction, that is, the deep convolution operation is continued on the 3rd and 4th columns of data based on the 3rd, 4th, 5th, 6th, and 7th columns of data.
[0089] For example, in this example, when the convolution kernel moves to the lower boundary of the first column of input data in the column direction, the lower boundary corresponding to the first data block is read from the cache unit and filled into the register to perform the convolution operation of the first column of input data. Figure 5 and 6C As shown, the lower boundary padding, that is, the upper 4 rows of the second data block adjacent to it (that is, the 4 rows in the second data block adjacent to the first data block, that is, j=4) are read from the global shared memory GSM into the register to complete the depth convolution operation of the last 4 rows of data of the first data block (that is, the data near the split point).
[0090] For example, Fig.6DAs shown in the figure, 4 columns of data are read into the register for depth convolution. When the convolution kernel size is 3x3 (i.e. dw=3, dh=3), the step size of the convolution operation is 1, and the data size processed in parallel by a single convolution operation instruction (i.e. processed in parallel in the row direction) is two columns of data (i.e. the depth convolution operation of the first two columns of input data is completed in parallel by two 3x3 convolution kernels), after moving the convolution kernel in the column direction to the lower boundary, the 1st and 2nd columns of input data are consumed at the same time (i.e. the 1st and 2nd columns of input data complete the depth convolution operation at the same time). At this time, the convolution kernel moves 2 columns of data in the row direction, i.e. Fig.6D In the example shown, since the stride of the convolution operation is 1 and the data size processed in parallel by a single convolution operation instruction (i.e., processed in parallel in the row direction) is two columns of data, k=2. Fig.6D In the example shown, after the depth convolution operation of the first column and the second column of the first data block is performed, the convolution kernel moves k columns of data in the row direction and discards the first k columns of data in the register, for example, Fig.6D In the scenario, the value of k is equal to 2, so the 1st and 2nd columns of data are discarded, and the new k columns of input data are read into the register to continue the deep convolution operation of the next column of data. Since the value of k is equal to 2, the new k columns of data, that is, the 5th and 6th columns of data are read into the register in sequence. At this time, 2 columns of data are moved in the row direction, that is, the deep convolution operation is continued on the 3rd and 4th columns of data based on the 3rd, 4th, 5th, and 6th columns of data.
[0091] For example, in this example, when the convolution kernel moves to the lower boundary of the first column of input data in the column direction, the lower boundary corresponding to the first data block is read from the cache unit and filled into the register to perform the convolution operation of the first column of input data. Figure 5 and 6D As shown, the lower boundary padding, that is, the upper 4 rows of the second data block adjacent to it (that is, the 4 rows in the second data block adjacent to the first data block, that is, j=4) are read from the global shared memory GSM into the register to complete the depth convolution operation of the last 4 rows of data of the first data block (that is, the data near the split point).
[0092] It should be noted that the above example is based on the first data block. The depthwise convolution operation of other data blocks is similar, but also includes upper boundary padding. For example, if Figures 6A-6D The data blocks in the data processing method are for other data blocks except the first data block, and the data processing method also includes: when the convolution kernel moves to the upper boundary of the i-th column of input data, the upper boundary corresponding to the t-th data block is read from the cache unit and filled into the register to perform the convolution operation of the i-th column of input data.
[0093] For example, Figure 5 As shown, the upper boundary padding of the tth data block is read from the global shared memory GSM, for example, Figures 6A-6D The data block in the example is the second data block, but of course the embodiments of the present disclosure are not limited to this.
[0094] For example, for Figure 6A-6B , read the lower 2 rows of the first data block adjacent to the second data block from the global shared memory GSM (i.e., the adjacent 2 rows of the first data block adjacent to the second data block, i.e., j=2) into the register to complete the depth convolution operation of the upper 2 rows of data of the second data block (i.e., the data near the dividing point between the first data block and the second data block); read the upper 2 rows of the third data block adjacent to the second data block (i.e., the adjacent 2 rows of the third data block adjacent to the second data block, i.e., j=2) into the register from the global shared memory GSM to complete the depth convolution operation of the lower 2 rows of data of the second data block (i.e., the data near the dividing point between the second data block and the third data block).
[0095] For example, for Figure 6C , read the lower 4 rows of the first data block adjacent to the second data block from the global shared memory GSM (i.e., the adjacent 4 rows of the first data block adjacent to the second data block, i.e., j=4) into the register to complete the depth convolution operation of the upper 4 rows of data of the second data block (i.e., the data near the dividing point between the first data block and the second data block); read the upper 4 rows of the third data block adjacent to the second data block (i.e., the adjacent 4 rows of the third data block adjacent to the second data block, i.e., j=4) into the register from the global shared memory GSM to complete the depth convolution operation of the lower 4 rows of data of the second data block (i.e., the data near the dividing point between the second data block and the third data block).
[0096] For example, for Fig.6D , read the lower 2 rows of the first data block adjacent to the second data block (i.e., the adjacent 2 rows of the first data block adjacent to the second data block, i.e., j=2) from the global shared memory GSM into the register to respectively complete the depth convolution operation of the upper 2 rows of data of the second data block (i.e., the data near the dividing point between the first data block and the second data block); read the upper 2 rows of the third data block adjacent to the second data block (i.e., the adjacent 2 rows of the third data block adjacent to the second data block, i.e., j=2) from the global shared memory GSM into the register to respectively complete the depth convolution operation of the lower 2 rows of data of the second data block (i.e., the data near the dividing point between the second data block and the third data block).
[0097] It should be noted that the size of the convolution kernel is not limited to 3 3 and 5 5. The convolution kernel may also be of other sizes, depending on the actual situation, and the embodiments of the present disclosure do not limit this.
[0098] It should be noted that in an embodiment of the present disclosure, for example, the lower boundary padding of the tth data block is the same as the upper boundary padding of the t+1th data block, and the upper boundary padding of the tth data block is the same as the upper boundary padding of the t-1th data block.
[0099] It should be noted that, in the embodiments of the present disclosure, the process of the data processing method provided by the above-mentioned various embodiments of the present disclosure may include more or fewer operations, and these operations may be performed sequentially or in parallel. Although the process of the data processing method described above includes multiple operations that appear in a specific order, it should be clearly understood that the order of multiple operations is not limited. The data processing method described above can be executed once, or it can be executed multiple times according to predetermined conditions.
[0100] The data processing method provided by the above-mentioned embodiment of the present disclosure, on the one hand, reads input data directly from the off-chip storage unit through the register to perform convolution operation, thereby avoiding the problem of being limited by the size of the shared memory and reducing the time of repeatedly reading the input data from the shared memory to the register; on the other hand, the space of the register can be reused to realize the multiplexing of the deep convolution input data. Combined with the splitting method, it is not limited by the size of the input data, and the number of times data is read from the off-chip storage is greatly reduced. At the same time, by storing only the data near the split point in the cache unit, the memory requirements of the cache unit can be reduced. At the same time, by effectively utilizing the space and bandwidth advantages of the cache unit, the number and time of reading data from the off-chip storage are reduced, thereby improving the operator performance.
[0101] Fig. 6E A schematic block diagram of a data processing device provided in at least one embodiment of the present disclosure. Fig. 6E In the example shown, the data processing device 100 is used for a convolution operation. For example, the size of the convolution kernel of the convolution operation is (dw, dh). For example, dw represents the number of columns of the convolution kernel (i.e., the width of the convolution kernel), and dh represents the number of rows of the convolution kernel (i.e., the height of the convolution kernel). For example, in some examples, the convolution operation in the embodiments of the present disclosure is a depth convolution operation. Of course, it can also be other neural network operations, which can be determined according to actual conditions. The embodiments of the present disclosure do not limit this. The following is an introduction using the convolution operation as a depth convolution operation as an example, and the embodiments of the present disclosure do not limit this.
[0102] For example, in some examples, the data processing device 100 includes a reading unit 110, a first processing unit 120 and a second processing unit 130; in other examples, a judgment unit and a cache unit (not shown in the figure) are also included. For example, these units can be implemented by hardware (such as circuit) modules or software modules, etc. The following embodiments are the same and will not be repeated. For example, these units can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a field programmable gate array (FPGA) or other forms of processing units with data processing capabilities and / or instruction execution capabilities and corresponding computer instructions.
[0103] The reading unit 110 is configured to read m columns of input data from M columns of input data of the off-chip storage unit into the register, where M is an integer greater than 0, m is an integer greater than 0 and less than or equal to M, and m>=dw. For example, the reading unit 110 can implement step S110, and its specific implementation method can refer to the relevant description of step S110, which will not be repeated here.
[0104] The first processing unit 120 is configured to perform a convolution operation on the i-th column of input data in the m-column input data, and after the convolution operation on the i-th column of input data is completed, delete the k-th column of input data corresponding to the i-th column of input data from the register, where i is an integer greater than 0 and less than or equal to m, and k is determined according to the step size of the convolution operation and the size of data processed in parallel by a single convolution operation instruction. For example, the first processing unit 120 can implement step S120, and its specific implementation method can refer to the relevant description of step S120, which will not be repeated here.
[0105] The second processing unit 130 is configured to re-read k columns of input data from the off-chip storage unit into the register, and reuse the remaining mk columns of input data after the convolution operation is performed on the i-th column of input data, and perform the convolution operation on the i+1-th column of input data. For example, the second processing unit 130 can implement step S130, and its specific implementation method can refer to the relevant description of step S130, which will not be repeated here.
[0106] For example, in some other examples of the present disclosure, the data processing device further includes a judgment unit configured to: judge whether the input data needs to be segmented according to the size of the input data and the size of the register, wherein the size of the input data is (WI, HI); if the size of the register is greater than or equal to HI dw, then there is no need to segment the input data; if the size of the register is smaller than HI dw, the input data is divided into T data blocks so that the size of the register is greater than or equal to (HI / T) dw, wherein the size of each block of the input data is (WI, (HI / T)); wherein T is an integer greater than 1.
[0107] For example, in some other examples of the present disclosure, the data processing device also includes a cache unit, configured to: after dividing the input data into T data blocks, store j adjacent rows of data of two adjacent blocks in the T data blocks, where j is an integer greater than 0 and is determined according to the step size of the convolution operation; and perform the convolution operation on the T data blocks in turn.
[0108] For example, in some other examples of the present disclosure, the data processing device can also be configured as: when the convolution kernel moves to the upper boundary of the tth data block, the upper boundary corresponding to the tth data block is read from the cache unit and filled into the register; when the convolution kernel moves to the lower boundary of the tth data block, the lower boundary corresponding to the tth data block is read from the cache unit and filled into the register; wherein, for the tth data block, the j rows of input data adjacent to the tth data block in the t-1th data block are used as the upper boundary filling of the tth data block; the j rows of input data adjacent to the tth data block in the t+1th data block are used as the lower boundary filling of the tth data block, wherein t is an integer greater than 1 and less than T.
[0109] For example, in some other examples of the present disclosure, the data processing device may also be configured as follows: when the convolution operation is performed on the t-th data block among the T data blocks, the method includes: reading m columns of input data in the t-th data block from the off-chip storage unit to the register; performing the convolution operation on the i-th column of input data of the m columns of input data in the t-th data block in the column direction; when the convolution kernel moves to the lower boundary of the i-th column of input data in the column direction, reading the t-th column of input data from the cache unit; The lower boundary corresponding to the ith data block is filled into the register to perform the convolution operation on the i-th column input data; after the convolution operation on the i-th column input data is completed, the k-column input data corresponding to the i-th column input data is deleted from the register, the k-column input data in the t-th data block is re-read from the off-chip storage unit into the register, and the mk-column input data remaining after the convolution operation on the i-th column input data is reused, and the convolution operation is performed on the i+1-th column input data of the t-th data block.
[0110] For example, in some other examples of the present disclosure, the data processing device can also be configured as: when the convolution kernel moves to the upper boundary of the i-th column of input data, the upper boundary corresponding to the t-th data block is read from the cache unit and filled into the register to perform the convolution operation of the i-th column of input data.
[0111] It should be noted that in the embodiments of the present disclosure, the data processing device 100 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit is not limited and can be composed of analog devices according to circuit principles, or can be composed of digital chips, or can be composed in other applicable ways.
[0112] The data processing device provided by the above-mentioned embodiment of the present disclosure, on the one hand, reads input data directly from the off-chip storage unit through the register to perform a deep convolution operation, thereby avoiding the problem of being limited by the size of the shared memory and reducing the time for repeatedly reading the input data from the shared memory to the register; on the other hand, the space of the register can be reused to realize the multiplexing of the deep convolution input data. Combined with the segmentation method, it is not limited by the size of the input data, and the number of times data is read from the off-chip storage is greatly reduced. At the same time, by storing only the data near the segmentation point in the cache unit, the memory requirements of the cache unit can be reduced. At the same time, by effectively utilizing the space and bandwidth advantages of the cache unit, the number and time of reading data from the off-chip storage are reduced, thereby improving the operator performance.
[0113] Figure 7 A schematic block diagram of another data processing device provided by at least one embodiment of the present disclosure. Figure 7 As shown, the data processing device 200 includes a processor 210 , a memory 220 , and one or more computer program modules 221 .
[0114] For example, the processor 210 is connected to the memory 220 via a bus system 230. For example, one or more computer program modules 221 are stored in the memory 220. For example, one or more computer program modules 221 include instructions for executing the data processing method provided by any embodiment of the present disclosure. For example, the instructions in one or more computer program modules 221 can be executed by the processor 210. For example, the bus system 230 can be a commonly used serial or parallel communication bus, etc., and the embodiments of the present disclosure are not limited to this.
[0115] For example, the processor 210 may be a central processing unit (CPU), a digital signal processor (DSP), a graphics processor (GPU), a general-purpose computing on graphics processing units (GPGPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may be a general-purpose processor or a dedicated processor, and may control other components in the data processing device 100 to perform desired functions. For example, in the embodiments of the present disclosure, the processor is taken as a general-purpose graphics processor (GPGPU) for example.
[0116] The memory 220 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 210 may run the program instructions to implement the functions (implemented by the processor 210) in the embodiment of the present disclosure and / or other desired functions, such as data processing methods, etc. Various applications and various data, such as various data used and / or generated by applications such as input data and data blocks, etc., may also be stored in the computer-readable storage medium.
[0117] It should be noted that, for the sake of clarity and brevity, the embodiments of the present disclosure do not provide all the components of the data processing device 200. To implement the necessary functions of the data processing device 200, those skilled in the art may provide and set other components not shown according to specific needs, and the embodiments of the present disclosure are not limited to this.
[0118] The data processing method or device according to the embodiment of the present disclosure can also be used as follows Figure 8 The exemplary electronic device 3000 shown in FIG. Figure 8As shown, the electronic device 3000 may include a bus 3010, one or more central processing units (CPU) or graphics processing units (GPU) or general purpose graphics processing units (GPGPU) 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the electronic device 3000, such as the ROM 3030 or the hard disk 3070, can store various data or files required for the processing and / or communication of the method provided by the present disclosure and the program instructions executed by the CPU or GPU. The electronic device 3000 may also include a user interface 3080. Of course, Figure 8 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 8 One or more components of an electronic device are shown.
[0119] At least one embodiment of the present disclosure further provides a storage medium. Fig. 9 A schematic diagram of a storage medium provided by at least one embodiment of the present disclosure. Fig. 9 As shown, the storage medium 400 non-temporarily stores computer-readable instructions 401 , and when the non-temporary computer-readable instructions are executed by a computer (including a processor), the data processing method provided by any embodiment of the present disclosure can be executed.
[0120] For example, the storage medium may be any combination of one or more computer-readable storage media, such as a computer-readable storage medium containing a computer-readable program code for reading m columns of input data from an off-chip storage unit into a register, another computer-readable storage medium containing a computer-readable program code for performing a convolution operation on the i-th column of input data among the m columns of input data, and deleting the k-th column of input data corresponding to the i-th column of input data from the register after the convolution operation on the i-th column of input data is completed, and another computer-readable storage medium containing a computer-readable program code for re-reading the k-th column of input data from the off-chip storage unit into the register, and reusing the mk-th column of input data remaining after the convolution operation on the i-th column of input data, and performing the convolution operation on the i+1-th column of input data. For example, when the program code is read by a computer, the computer may execute the program code stored in the computer storage medium, and execute, for example, the data processing method provided in any embodiment of the present disclosure.
[0121] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other applicable storage media.
[0122] Regarding the above public content, there are a few points that need to be explained.
[0123] (1) The drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure. Other structures may refer to the general design.
[0124] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.
[0125] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.
Claims
1. A data processing method for convolution operation, characterized in that: The size of the convolution kernel of the convolution operation is (dw, dh), where dw represents the number of columns of the convolution kernel, and dh represents the number of rows of the convolution kernel. The method includes: Read m columns of input data from M columns of input data of an off-chip storage unit into a register, where M is an integer greater than 0, m is an integer greater than 0 and less than or equal to M, and m>= dw; Perform the convolution operation on the i-th column of input data among the m-columns of input data, and after the convolution operation on the i-th column of input data is completed, delete the k-columns of input data corresponding to the i-th column of input data from the register, wherein i is an integer greater than 0 and less than or equal to m, and k is determined according to the step size of the convolution operation and the size of data processed in parallel by a single instruction of the convolution operation; Reread k columns of input data from the off-chip storage unit into the register, reuse the mk columns of input data remaining after the convolution operation is performed on the i-th column of input data, and perform the convolution operation on the i+1-th column of input data.
2. The data processing method according to claim 1, further comprising: According to the size of the input data and the size of the register, determining whether the M columns of input data need to be segmented, wherein the size of the M columns of input data is (WI, HI), wherein WI represents the number of columns of the input data and WI=M, and HI represents the number of rows of the input data; If the size of the register is greater than or equal to HI dw, then there is no need to segment the input data; If the size of the register is smaller than HI dw, the input data is divided into T data blocks so that the size of the register is greater than or equal to (HI / T) dw, wherein the size of each data block of the input data is (WI, (HI / T)); Wherein, T is an integer greater than 1.
3. The data processing method according to claim 2, wherein: After dividing the input data into T data blocks, the method further includes: Storing adjacent j rows of input data of two adjacent data blocks in the T data blocks into a cache unit, where j is an integer greater than 0 and is determined according to the step size of the convolution operation; The convolution operation is performed on the T data blocks in sequence.
4. The data processing method according to claim 3, further comprising: When the convolution kernel moves to the upper boundary of the t-th data block, the upper boundary corresponding to the t-th data block is read from the cache unit and filled into the register; When the convolution kernel moves to the lower boundary of the t-th data block, the lower boundary corresponding to the t-th data block is read from the cache unit and filled into the register; Wherein, for the t-th data block, the j rows of input data adjacent to the t-th data block in the t-1th data block are used as the upper boundary filling of the t-th data block; the j rows of input data adjacent to the t-th data block in the t+1th data block are used as the lower boundary filling of the t-th data block, Here, t is an integer greater than 1 and less than T.
5. The data processing method according to claim 3, wherein: When performing the convolution operation on the tth data block among the T data blocks, the method includes: Reading m columns of input data in the t-th data block from the off-chip storage unit to the register; Performing the convolution operation on the i-th column of input data of the m columns of input data in the t-th data block in a column direction; When the convolution kernel moves in the column direction to the lower boundary of the i-th column of input data, the lower boundary corresponding to the t-th data block is read from the cache unit and filled into the register to perform the convolution operation of the i-th column of input data; After the convolution operation of the i-th column of input data is completed, the k-th column of input data corresponding to the i-th column of input data is deleted from the register, Re-read the k columns of input data in the t-th data block from the off-chip storage unit into the register, reuse the mk columns of input data remaining after the convolution operation is performed on the i-th column of input data, and perform the convolution operation on the i+1-th column of input data in the t-th data block.
6. The data processing method according to claim 5, further comprising: When the convolution kernel moves to the upper boundary of the i-th column of input data, the upper boundary corresponding to the t-th data block is read from the cache unit and filled into the register to perform the convolution operation of the i-th column of input data.
7. The data processing method according to any one of claims 1 to 6, wherein: The convolution operation is a depth-wise convolution operation.
8. A data processing device for convolution operation, characterized in that: The size of the convolution kernel of the convolution operation is (dw, dh), where dw represents the number of columns of the convolution kernel, and dh represents the number of rows of the convolution kernel. The device includes: A reading unit is configured to read m columns of input data from M columns of input data of an off-chip storage unit into a register, wherein M is an integer greater than 0, m is an integer greater than 0 and less than or equal to M, and m>=dw; a first processing unit, configured to perform the convolution operation on the i-th column of input data among the m columns of input data, and after the convolution operation on the i-th column of input data is completed, delete the k-th column of input data corresponding to the i-th column of input data from the register, wherein i is an integer greater than 0 and less than or equal to m, and k is determined according to the step size of the convolution operation and the size of data processed in parallel by a single instruction of the convolution operation; The second processing unit is configured to re-read k columns of input data from the off-chip storage unit into the register, and reuse the mk columns of input data remaining after the convolution operation is performed on the i-th column of input data, and perform the convolution operation on the i+1-th column of input data.
9. A data processing device, characterized in that: The data processing device comprises: processor; Memory; One or more computer program modules, wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include instructions for executing the data processing method described in any one of claims 1-7.
10. A storage medium that non-temporarily stores computer-readable instructions, characterized in that: When the computer-readable instructions are executed by a computer, the data processing method according to any one of claims 1 to 7 is performed.
