Data storage and reading method and device of neural network

Abstract

The embodiment of the invention relates to a data storage and reading method and device of a neural network, and is used for storing to-be-convolved data of a neural network convolution layer to an RAM module, and the to-be-convolved data is used for carrying out convolution by adopting a 3 * 3 convolution kernel. The data storage method comprises the following steps: sequentially storing the Ath row of data of to-be-convolved data into each RAM of the first row of RAMs; storing the Bth row of data of the to-be-convolved data in each RAM of the second row of RAM in sequence; and storing the C-th row of data of the to-be-convolved data in each RAM of the third row of RAM in sequence. According to the technical scheme provided by the embodiment of the invention, the to-be-convolved data of the convolutional layer of the neural network is stored in different row and column RAMs according to the idea of row and column separation, and the storage data addresses are allocated based on the 3 * 3 convolution kernel rule, so that the storage and reading speeds of the to-be-convolved data are effectively improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of neural network technology, and more particularly to a method and device for storing and reading data of a neural network. Background Art

[0002] Convolutional neural network is a deep learning model that is widely used in image recognition, video analysis, and natural language processing. The convolution layer is the core component of the convolutional neural network, which extracts the features of the input data through convolution operations.

[0003] The input data of the convolution layer is usually a data matrix, which is stored in RAM in row-first or column-first order. However, this storage method cannot achieve simultaneous update of the convolved data, and the data storage and reading speed needs to be improved. Summary of the invention

[0004] Based on the above situation of the prior art, the purpose of the embodiments of the present invention is to provide a method and device for storing and reading data of a neural network, which adopts a row-column separation storage method to effectively improve the storage and reading speed of the data to be convolved.

[0005] To achieve the above object, according to a first aspect of the present invention, a method for storing data of a neural network is provided, wherein data to be convolved of a convolutional layer of a neural network is stored in a RAM module, wherein the data to be convolved is used for convolution using a 3×3 convolution kernel, and the method comprises the steps of:

[0006] The Ath row of data to be convolved is stored sequentially in each RAM of the first row of RAM;

[0007] The Bth row of data to be convolved is stored in each RAM of the second row of RAM in sequence;

[0008] The Cth row of data to be convolved is sequentially stored in each RAM of the third row of RAM;

[0009] A=1+3i, i=0,1,……and i is an integer;

[0010] B=2+3i, i=0,1,… and i is an integer;

[0011] C=3+3i, i=0,1,…and i is an integer.

[0012] Further, the first row of RAM includes a first row of first RAM, a first row of second RAM, and a first row of third RAM; the second row of RAM includes a second row of first RAM, a second row of second RAM, and a second row of third RAM; the third row of RAM includes a third row of first RAM, a third row of second RAM, and a third row of third RAM;

[0013] Wherein, the address of each RAM is independently controlled.

[0014] Further, the Ath row of data to be convolved is sequentially stored in each RAM of the first row of RAM, including:

[0015] Store the data of the Ath row and the Dth column of the data to be convolved into the first RAM of the first row;

[0016] Store the data of the Ath row and the Eth column of the data to be convolved into the second RAM of the first row;

[0017] Store the data of the Ath row and the Fth column of the data to be convolved into the third RAM of the first row;

[0018] D=1+3j, j=0,1,……and j is an integer;

[0019] E=2+3j, j=0,1,……and j is an integer;

[0020] F=3+3j, j=0,1,……and j is an integer.

[0021] Further, the Bth row of data to be convolved is sequentially stored in each RAM of the second row of RAM, including:

[0022] Store the data of the Bth row and the Dth column of the data to be convolved into the first RAM of the second row;

[0023] Store the data of the Bth row and the Eth column of the data to be convolved into the second RAM of the second row;

[0024] Store the data of the Bth row and the Fth column of the data to be convolved into the third RAM of the second row;

[0025] The Cth row of data to be convolved is sequentially stored in each RAM of the third row of RAM, including:

[0026] Store the data of the Cth row and the Dth column of the data to be convolved into the first RAM of the third row;

[0027] Store the data of the Cth row and Eth column of the data to be convolved into the second RAM of the third row;

[0028] The Cth row and Fth column data of the data to be convolved are stored in the third RAM of the third row.

[0029] According to a second aspect of the present invention, there is provided a data storage device for a neural network, the data storage device being used to store data to be convolved of a convolutional layer of a neural network using the method described in the first aspect of the present invention, the data to be convolved being used to perform convolution using a 3×3 convolution kernel;

[0030] The data storage device includes a RAM module, and the RAM module includes a 3-row × 3-column RAM matrix.

[0031] According to the third aspect of the present invention, a method for reading data of a neural network is provided. Based on the storage method provided in the first aspect of the present invention, the data to be convolved is read out from the RAM module. The method includes the steps of:

[0032] Setting a reading order based on the number of sliding window times of convolution and the step value of convolution;

[0033] Reading each RAM in the RAM module according to the reading order.

[0034] Further, setting the reading order based on the number of sliding window times of convolution and the step value of convolution includes: when str = 1,

[0035] M(3k - 3) ≤ w < M(3k - 2), reading the first-row RAM, the second-row RAM, and the third-row RAM in sequence;

[0036] M(3k - 2) ≤ w < M(3k - 1), reading the second-row RAM, the third-row RAM, and the first-row RAM in sequence;

[0037] M(3k - 1) ≤ w < M(3k), reading the third-row RAM, the first-row RAM, and the second-row RAM in sequence;

[0038] where w represents the number of sliding window times, str represents the step value, k is a positive integer, M = N - 2, and N represents the number of columns of the data to be convolved.

[0039] Further, setting the reading order based on the number of sliding window times of convolution and the step value of convolution includes: when str = 2,

[0040] p(3k - 3) ≤ w < p(3k - 2), reading the first-row RAM, the second-row RAM, and the third-row RAM in sequence;

[0041] p(3k - 2) ≤ w < p(3k - 1), reading the third-row RAM, the first-row RAM, and the second-row RAM in sequence;

[0042] p(3k - 1) ≤ w < p(3k), reading the second-row RAM, the third-row RAM, and the first-row RAM in sequence;

[0043] where, FLOOR() represents taking the integer part of the quotient of division; w represents the number of sliding window times, str represents the step value, k is a positive integer, and N represents the number of columns of the data to be convolved.

[0044] Furthermore, the reading order is set based on the number of sliding windows of the convolution and the step value of the convolution, and further includes: when w=0,

[0045] In each row of RAM, the first RAM, the second RAM and the third RAM of the row are read in sequence;

[0046] Further, the reading order is set based on the number of sliding windows of the convolution and the step value of the convolution, and further includes: in each row of RAM,

[0047] When str=1 and w mod 3=1, or str=2 and w mod 3=2, the second RAM, the third RAM and the first RAM of the row are read in sequence;

[0048] When str=1 and w mod 3=2, or str=2 and w mod 3=1, the third RAM, the first RAM, and the second RAM of the row are read in sequence;

[0049] When str=1 and w mod 3=3, or str=2 and w mod 3=3, the first RAM, the second RAM and the third RAM of the row are read in sequence;

[0050] Here, a mod b means the remainder when a is divided by b.

[0051] In summary, the embodiment of the present invention provides a method and device for storing and reading data of a neural network, which is used to store the data to be convolved of the convolution layer of the neural network into a RAM module, wherein the data to be convolved is used for convolution using a 3×3 convolution kernel, and the data storage method includes the steps of: storing the Ath row of data of the data to be convolved in each RAM of the first row of RAM in sequence; storing the Bth row of data of the data to be convolved in each RAM of the second row of RAM in sequence; storing the Cth row of data of the data to be convolved in each RAM of the third row of RAM in sequence. The technical solution provided by the embodiment of the present invention stores the data to be convolved of the convolution layer of the neural network into different row and column RAMs according to the idea of ​​row and column separation, and allocates the storage data address based on the 3×3 convolution kernel rule, thereby effectively improving the storage and reading speed of the data to be convolved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the sliding window when the convolution data is convolved with a 3×3 convolution kernel;

[0053] Figure 2 It is a schematic diagram of the actual storage of the data to be convolved in the prior art;

[0054] Figure 3 is a flow chart of a data storage method for a neural network provided by an embodiment of the present invention;

[0055] Figure 4 1 is a schematic diagram of the storage order of the data to be convolved in the row-column separation RAM according to an embodiment of the present invention;

[0056] Figure 5 is a schematic diagram of the structure of a data storage device provided by an embodiment of the present invention;

[0057] Figure 6 2 is a schematic diagram of row-column separation storage when a 1×1 convolution kernel is used in an embodiment of the present invention;

[0058] Figure 7 It is a schematic diagram of the corresponding relationship between the read data and the weight when the convolution data is a 3×8 data matrix and the number of sliding windows is 0;

[0059] Figure 8 This is a schematic diagram of the corresponding relationship between the read data and the weight when the convolution data is a 3×8 data matrix, the number of sliding windows is 1, and the step length is 1;

[0060] Fig. 9 This is a schematic diagram of the corresponding relationship between the read data and the weight when the convolution data is a 3×8 data matrix, the number of sliding windows is 1, and the step length is 2;

[0061] Fig.10 is a counting flow chart of a data writing precision counter according to an embodiment of the present invention;

[0062] Fig.11 is a counting flow chart of a write column counter according to an embodiment of the present invention;

[0063] Fig.12 is a counting flow chart of a write row counter according to an embodiment of the present invention;

[0064] Fig.13 It is a counting flow chart of the write column RAM counter according to an embodiment of the present invention;

[0065] Fig.14 It is a counting flow chart of a write row RAM counter according to an embodiment of the present invention;

[0066] Fig.15 is a counting flow chart of a write column base address counter according to an embodiment of the present invention;

[0067] Fig.16 is a counting flow chart of a data reading precision counter according to an embodiment of the present invention;

[0068] Fig.17 is a counting flow chart of a read column counter according to an embodiment of the present invention;

[0069] Fig.18 is a counting flow chart of a read line counter according to an embodiment of the present invention;

[0070] Fig.19is a counting flow chart of a RAM counter read in an embodiment of the present invention;

[0071] Fig. 20 is a counting flow chart of a read row RAM counter according to an embodiment of the present invention;

[0072] Fig.21 It is a counting flow chart of the column base address counter according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0074] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in one or more embodiments of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0075] In a convolutional neural network, for a 3×3×128=1152 (bit) convolution calculation unit (i.e., data to be convolved) near-memory RAM design, if you want 1152 inputs to be updated simultaneously, you can design the RAM bit width to be 1152 (bit). When this storage method is used, the sliding window diagram of the convolution calculation unit (i.e., data to be convolved) using a 3×3 convolution kernel for convolution is as follows: Figure 1 As shown in the figure, without considering padding, the first convolution (red box) overlaps with the second convolution (yellow box) by 6 pixels, overlaps with the third convolution (purple box) by 3 pixels, and has no overlap with the fourth convolution (blue box). When the convolution calculation unit is stored in RAM, since the on-chip storage of the processor is very valuable, it is impossible to store the overlapping part multiple times, otherwise the space occupied will be 3 times that of the entire convolution calculation unit. Therefore, the actual storage of the convolution calculation unit (i.e., the data to be convolved) is usually as follows: Figure 2As shown, only two addresses (addr_0 and addr_dp) are used to store the pixels required for the red and blue convolutions. Figure 2 Under the storage mode, the data of the second convolution (yellow box) needs to be fetched from two addresses of the RAM, which obviously violates the premise of updating 1152 data at the same time.

[0076] In response to the above technical problems, an embodiment of the present invention provides a data storage method for a neural network, which uses multiple low-bit-width RAMs to be spliced ​​into 1152 (bits) and adopts a row-column separation storage method to achieve simultaneous data update, effectively improving the storage and reading speed of the data to be convolved. The technical solution of the present invention is described in detail below with reference to the accompanying drawings. An embodiment of the present invention provides a data storage method for a neural network, which is used to store the data to be convolved of the convolution layer of the neural network into a RAM module, and the data to be convolved is used to perform convolution using a 3×3 convolution kernel. Figure 3 A flow chart of the method is shown in Figure 3 As shown, the method comprises the following steps:

[0077] S202, sequentially store the Ath row of data to be convolved into each RAM of the first row of RAM, the first row of RAM including the first row of first RAM, the first row of second RAM and the first row of third RAM. When storing data into each RAM of the first row of RAM, store the Ath row of Dth column of data to be convolved into the first row of first RAM; store the Ath row of Eth column of data to be convolved into the first row of second RAM; store the Ath row of Fth column of data to be convolved into the first row of third RAM. Wherein, the values ​​of A, D, E and F are determined according to the following formula:

[0078] A=1+3i, i=0,1,……and i is an integer;

[0079] D=1+3j, j=0,1,……and j is an integer;

[0080] E=2+3j, j=0,1,……and j is an integer;

[0081] F=3+3j, j=0,1,……and j is an integer.

[0082] S204, sequentially store the Bth row of data to be convolved into each RAM of the second row of RAM, the second row of RAM including the second row of first RAM, the second row of second RAM and the second row of third RAM. When storing data into each RAM of the second row of RAM, store the Bth row of Dth column of data to be convolved into the second row of first RAM; store the Bth row of Eth column of data to be convolved into the second row of second RAM; store the Bth row of Fth column of data to be convolved into the second row of third RAM. Among them, the method of taking the values ​​of D, E and F is the same as in the above step S202, and the value of B is determined according to the following formula:

[0083] B=2+3i, i=0,1,…and i is an integer.

[0084] S206. Store the Cth row of data to be convolved in each RAM of the third row of RAM in sequence, and the third row of RAM includes the third row of first RAM, the third row of second RAM, and the third row of third RAM. When storing data in each RAM of the third row of RAM, store the Cth row of Dth column of data to be convolved in the third row of first RAM; store the Cth row of Eth column of data to be convolved in the third row of second RAM; store the Cth row of Fth column of data to be convolved in the third row of third RAM. Among them, the method of taking the values ​​of D, E, and F is the same as in the above step S202, and the value of C is determined according to the following formula:

[0085] C=3+3i, i=0,1,…and i is an integer.

[0086] In the storage process of the above steps and the subsequent reading process, the address of each RAM is independently controlled.

[0087] In the embodiment provided by the present invention, 9 RAMs are used for splicing to realize the storage of the data to be convolved. These 9 RAMs are designed based on the idea of ​​row and column separation of 9 pixels using a 3×3 convolution kernel. Considering the example above, for 3×3×128=1152 (bit) of data to be convolved, using the embodiment of the present invention, 9 128 (bit) RAMs can be used to splice into 1152 (bit). Figure 4 FIG. 1 shows a schematic diagram of the storage order of the row and column separation RAM of the convolution data using the storage method of the embodiment of the present invention. Figure 1 The 3×3×128=1152 (bit) data to be convolved is shown in Figure 4As shown, the first row of the data to be convolved is stored in ram1, ram2, and ram3, wherein the first column is stored in ram1, the second column is stored in ram2, the third column is stored in ram3, the fourth column is stored in ram1, the fifth column is stored in ram2, and the sixth column is stored in ram3, and so on; the second row of the data to be convolved is stored in ram4, ram5, and ram6, wherein the first column is stored in ram4, the second column is stored in ram5, the third column is stored in ram6, the fourth column is stored in ram4, the fifth column is stored in ram5, and the sixth column is stored in ram6, and so on. Loop; store the third row of data to be convolved in ram7, ram8, and ram9, where the first column is stored in ram7, the second column is stored in ram8, the third column is stored in ram9, the fourth column is stored in ram7, the fifth column is stored in ram8, and the sixth column is stored in ram9, and so on; store the fourth row of data to be convolved in ram1, ram2, and ram3, where the first column is stored in ram1, the second column is stored in ram2, the third column is stored in ram3, the fourth column is stored in ram1, the fifth column is stored in ram2, and the sixth column is stored in ram3, and so on.

[0088] According to the storage method provided by the embodiment of the present invention, every three rows and every three columns of the data to be convolved are stored separately in the RAM. Every three RAMs store a row of the data to be convolved, which is called a row RAM. The three RAMs in the row RAM are respectively called column RAMs in the row. The three column RAMs store the pixel points of a whole row of the data to be convolved in sequence. After storing a whole row, the next row to be stored is the row after two rows of the data to be convolved. For example, after the first row RAM stores the first row of the data to be convolved, the fourth row of the data to be convolved is stored.

[0089] An embodiment of the present invention further provides a data storage device for a neural network, the data storage device is used to store the data to be convolved of the convolution layer of the neural network using the method described in the above embodiment of the present invention, the data to be convolved is used to perform convolution using a 3×3 convolution kernel. The data storage device provided by the present invention includes a RAM module, and the RAM module includes a 3-row×3-column RAM matrix. Figure 5 The structural diagram of the data storage device is shown in FIG. Figure 5 As shown, the three RAMs ram1, ram2 and ram3 are collectively referred to as rRAM0 (row ram0), and the three column RAMs in row ram0 are respectively referred to as cRAM0, cRAM1 and cRAM2. Similarly, the three RAMs ram4, ram5 and ram6 are collectively referred to as rRAM1 (row ram1), and the three RAMs ram7, ram8 and ram9 are collectively referred to as rRAM2 (row ram2).

[0090] In the above embodiments, the data to be convolved is for the case of using a 3×3 convolution kernel. According to some alternative embodiments, the data to be convolved can also be convolved using a 1×1 convolution kernel. In the case of using a 1×1 convolution kernel, since most of the input channels of the 1×1 convolution kernel are relatively large, such as 256 / 512 / 1024, etc., when storing in a row-column separated RAM, the input channels can be divided into multiple 128 (bit) and placed in ram1 - ram9. Figure 6 The schematic diagram of row-column separated storage when using a 1×1 convolution kernel is shown in Figure 6 As shown, the 9 RAMs can support a maximum of 128×9 input channels. Since the number of channels of the convolution kernel is generally a power of 2, the maximum input channels that the RAM structure shown in Figure 5 can support in a 1×1 convolution is 1024. In a 1×1 convolution, the RAM is used according to "channel" separation. When writing to the RAM, if there are several 128 in the input channels, write to several RAMs, and do not write those not used. For example, when the number of input channels is 512, only ram1, ram2, ram3, and ram4 are used. All rows and columns of the first 128 channels (channels 1 - 128) are stored in ram1, all rows and columns of the second 128 channels (channels 129 - 256) are stored in ram2, all rows and columns of the third 128 channels (channels 257 - 384) are stored in ram3, and all rows and columns of the fourth 128 channels (channels 385 - 512) are stored in ram4. When reading, the address is incremented by 1 successively and read directly.

[0091] An embodiment of the present invention also provides a method for reading data of a neural network. When performing a convolution operation on the data read from the RAM, the order of the data read from the 9 RAMs needs to be adjusted according to different numbers of sliding window times and different step values. This data reading method is based on the storage method provided in the above embodiments of the present invention, and reads the data to be convolved from the RAM module. The method includes the following steps:

[0092] S302. Set the reading order based on the number of sliding window times of the convolution and the step value of the convolution.

[0093] S3021. Set the reading order based on the number of sliding window times of the convolution and the step value of the convolution. Consider two cases where the step value is 1 and the step value is 2, and address the row RAM based on the value of the sliding window times w:

[0094] When the step value str = 1,

[0095] M(3k - 3) ≤ w < M(3k - 2), read the first row RAM, the second row RAM, and the third row RAM in sequence;

[0096] When \(M(3k - 2)\leq w\lt M(3k - 1)\), read the second-row RAM, the third-row RAM, and the first-row RAM in sequence;

[0097] When \(M(3k - 1)\leq w\lt M(3k)\), read the third-row RAM, the first-row RAM, and the second-row RAM in sequence;

[0098] Among them, \(w\) represents the number of sliding window times, \(str\) represents the step value, \(k\) is a positive integer, \(M = N - 2\), and \(N\) represents the number of columns of the data to be convolved.

[0099] When the step value \(str = 2\),

[0100] When \(p(3k - 3)\leq w\lt p(3k - 2)\), read the first-row RAM, the second-row RAM, and the third-row RAM in sequence;

[0101] When \(p(3k - 2)\leq w\lt p(3k - 1)\), read the third-row RAM, the first-row RAM, and the second-row RAM in sequence;

[0102] When \(p(3k - 1)\leq w\lt p(3k)\), read the second-row RAM, the third-row RAM, and the first-row RAM in sequence;

[0103] Among them, \(FLOOR()\) represents taking the integer part of the quotient of the division; \(w\) represents the number of sliding window times, \(str\) represents the step value, \(k\) is a positive integer, and \(N\) represents the number of columns of the data to be convolved.

[0104] S3022. Set the reading order based on the number of sliding window times of convolution and the step value of convolution. Considering the values of the number of sliding window times \(w\) and the step value \(str\), address the column RAM in the specific row RAM:

[0105] S3023. When \(w = 0\), read the first RAM, the second RAM, and the third RAM in sequence in each row of RAM.

[0106] S3024. In each row of RAM,

[0107] When \(str = 1\) and \(w\bmod 3 = 1\), or \(str = 2\) and \(w\bmod 3 = 2\), read the second RAM, the third RAM, and the first RAM in sequence in this row;

[0108] When \(str = 1\) and \(w\bmod 3 = 2\), or \(str = 2\) and \(w\bmod 3 = 1\), read the third RAM, the first RAM, and the second RAM in sequence in this row;

[0109] When str=1 and w mod 3=3, or str=2 and w mod 3=3, the first RAM, the second RAM and the third RAM of the row are read in sequence;

[0110] Here, a mod b means the remainder when a is divided by b.

[0111] S304, reading each RAM in the RAM module according to the reading order.

[0112] In the above reading method provided by the embodiment of the present invention, addressing is performed according to the storage rule of the data to be convolved in the RAM and the step value and the number of sliding windows when the convolution layer performs convolution, so that the data reading in the convolution operation can be updated simultaneously. For example, when the data to be convolved is a 3×8 data matrix, the above reading method is used for addressing. Figure 7 The figure shows the corresponding relationship between the read data and the weight when the convolution data is 3×8 and the number of sliding windows is 0. Figure 7 As shown, the convolution step value is 1, that is, str = 1. For the pixels (1, 2) and (1, 3) that need to be reused, the corresponding weights are w2 and w3 in the first convolution (that is, the number of sliding windows is 0), and the corresponding weights are w1 and w2 in the second convolution. In this application, weights refer to the parameters in the convolution kernel. Therefore, according to the method provided in the above embodiment, when the row RAM is addressed, the first row RAM, the second row RAM, and the third row RAM are read in sequence; when the column RAM is addressed in each row, the first RAM, the second RAM, and the third RAM of each row RAM are read in sequence. Therefore, the order of the weights corresponding to the data read out from the 9 RAMs is {ram1, ram2, ram3, ram4, ram5, ram6, ram7, ram8, ram9}. Figure 8 The figure shows the corresponding relationship between the read data and the weight when the convolution data is a 3×8 data matrix, the number of sliding windows is 1, and the step length is 1. According to the method provided in the above embodiment, when the row RAM is addressed, the first row RAM, the second row RAM, and the third row RAM are read in sequence; when the column RAM is addressed in each row, the second RAM, the third RAM, and the first RAM of the row are read in sequence. Therefore, in order to correspond one-to-one with the weights, the data read from the 9 RAMs should be in the order of {ram2,ram3,ram1,ram5,ram6,ram4,ram8,ram9,ram7}. For the read address, Figure 7 In the example, the read addresses of the 9 RAMs are all the same, and in Figure 8In the above figure, the address of the data read in RAM1 has one more dp than the base address, and the addresses of the data read in RAM2 and RAM3 remain unchanged; the address of the data read in RAM4 has one more dp than the base address, and the addresses of the data read in RAM5 and RAM6 remain unchanged; the address of the data read in RAM7 has one more dp than the base address, and the addresses of the data read in RAM8 and RAM9 remain unchanged. Fig. 9 The figure shows the corresponding relationship between the read data and the weight when the convolution data is a 3×8 data matrix, the number of sliding windows is 1, and the step length is 2. According to the method provided in the above embodiment, when the row RAM is addressed, the first row RAM, the second row RAM, and the third row RAM are read in sequence; when the column RAM is addressed in each row, the third RAM, the first RAM, and the second RAM of the row are read in sequence. Therefore, if Fig. 9 As shown, the order of the weights of the data read from the 9 RAMs is {ram3,ram1,ram2,ram6,ram4,ram5,ram9,ram7,ram8}.

[0113] The data to be convolved is stored bit by bit in RAM. If the data precision of the data to be convolved is dp, then it needs to be written dp times to write the value of one pixel of the data to be convolved, and it needs to be read dp times to read the value of one pixel of the data to be convolved.

[0114] When writing data to the RAM using the data storage method provided by the above embodiment of the present invention, the following counters may be set, and each counter has a linkage relationship:

[0115] Write data precision counter wr_dp_cnt: used to count the precision of write data. When there is available data, the write data precision counter is incremented by 1 while writing data to RAM at the rising edge of each clock cycle, and is cleared when the count reaches dp-1. Fig.10 The counting process of the write data precision counter is shown in FIG.

[0116] Write column counter wr_col_cnt: used to count the number of columns of write data. When the write data precision counter counts to dp-1, the write column counter is incremented by 1. When the write column counter counts to col-1 (col represents the number of columns of data to be convolved) and the write data precision counter counts to dp-1, the write column counter is cleared. Fig.11 The counting process of the write column counter is shown in FIG.

[0117] Write row counter wr_row_cnt: used to count the number of rows of write data. When the write column counter counts to col-1, the write row counter is incremented by 1. When the write row counter counts to row-1 (row represents the number of rows of data to be convolved) and the write column counter counts to col-1 and the write data precision counter counts to dp-1, the write row counter is cleared. Fig.12 . The counting process of the write row counter is shown in FIG.

[0118] The write row counter and write column counter are set outside the 9 RAMs. Figure 1 The data to be convolved input into the 9 RAMs are counted in rows and columns, and stored in the 9 RAMs according to the storage method provided in the above embodiment according to the different numbers of rows and columns of the data to be convolved.

[0119] Write column ram counter wr_col_ram_cnt and write row ram counter wr_row_ram_cnt: used to indicate which of ram1-ram9 a data is written to, thereby controlling the write enable of the 9 RAMs. Because the row-column separated memory is used to store 3 rows and 3 columns of data, the count range of the write row ram counter and the write column ram counter are both 0-2, and the counting rules are the same as the write row counter and the write column counter, but the counting range is different. Different combinations of write column ram counters and write row ram counters are written into different RAMs. When the write column ram counters and write row ram counters are the following values, the enabling status of the 9 RAMs is: (0,0), ram1 is enabled; (0,1), ram2 is enabled; (0,2), ram3 is enabled; (1,0), ram4 is enabled; (1,1), ram5 is enabled; (1,2), ram6 is enabled; (2,0), ram7 is enabled; (2,1), ram8 is enabled; (2,2), ram9 is enabled. Fig.13 The counting process of the write column RAM counter is shown in Fig.14 The counting process of the write row RAM counter is shown in FIG.

[0120] Write column base address counter wr_col_base: used to calculate the address of write data. When the write column RAM counter counts to 2 and the write data precision counter counts to dp-1, the write column base address counter adds dp, so the address of the write RAM is wr_col_base+dp. Fig.15 . The counting process of the write column base address counter is shown in FIG.

[0121] The above embodiment of the present invention provides an addressing method for data reading. At the same time, a counter similar to the above-mentioned write data is also required when reading RAM. The counter used for reading data and the counter used for writing data are two separate sets of counters. Similar to the above-mentioned write data counter, the read data counter includes a read data precision counter rd_dp_cnt, a read column counter rd_col_cnt, a read row counter rd_row_cnt, a read column ram counter rd_col_ram_cnt, a read row ram counter rd_row_ram_cnt and a read column base address counter rd_col_base. The functions of the above-mentioned read data counters are similar to those of the write data counters, except that the read and write directions are different. Fig.16 The counting process of the read data precision counter is shown in FIG. Fig.17 The counting process of the read column counter is shown in FIG. Fig.18 The counting process of the read line counter is shown in FIG. Fig.19 The counting process of the read column RAM counter is shown in FIG. Fig. 20 The counting process of the read row ram counter is shown in Fig.21 . The counting process of the read column base address counter is shown in FIG.

[0122] In summary, the embodiments of the present invention relate to a method and device for storing and reading data of a neural network, which is used to store the data to be convolved of the convolution layer of the neural network into a RAM module, wherein the data to be convolved is used for convolution using a 3×3 convolution kernel, and the data storage method includes the steps of: storing the Ath row of data of the data to be convolved in each RAM of the first row of RAM in sequence; storing the Bth row of data of the data to be convolved in each RAM of the second row of RAM in sequence; storing the Cth row of data of the data to be convolved in each RAM of the third row of RAM in sequence. The technical solution provided by the embodiments of the present invention stores the data to be convolved of the convolution layer of the neural network into different row and column RAMs according to the idea of ​​row and column separation, and allocates the storage data address based on the 3×3 convolution kernel rule, thereby effectively improving the storage and reading speed of the data to be convolved.

[0123] It should be understood that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present invention (including claims) is limited to these examples; under the idea of ​​the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of one or more embodiments of the present invention as described above, which are not provided in detail for the sake of simplicity. The above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the scope of protection of the present invention. In addition, the claims attached to the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the attached claims, or the equivalent forms of such scope and boundaries.

Claims

1. A method for storing data of a neural network, characterized in that: The data to be convolved of the convolution layer of the neural network is stored in a RAM module, wherein the data to be convolved is used for convolution using a 3×3 convolution kernel, and the method comprises the steps of: The Ath row of data to be convolved is stored sequentially in each RAM of the first row of RAM; The Bth row of data to be convolved is stored in each RAM of the second row of RAM in sequence; The Cth row of data to be convolved is sequentially stored in each RAM of the third row of RAM; A=1+3i, i=0,1,……and i is an integer; B=2+3i, i=0,1,… and i is an integer; C=3+3i, i=0,1,…and i is an integer.

2. The method according to claim 1, characterized in that The first row of RAM includes a first row of first RAM, a first row of second RAM, and a first row of third RAM; the second row of RAM includes a second row of first RAM, a second row of second RAM, and a second row of third RAM; the third row of RAM includes a third row of first RAM, a third row of second RAM, and a third row of third RAM; Wherein, the address of each RAM is independently controlled.

3. The method according to claim 2, characterized in that The Ath row of data to be convolved is sequentially stored in each RAM of the first row of RAM, including: Store the data of the Ath row and the Dth column of the data to be convolved into the first RAM of the first row; Store the data of the Ath row and the Eth column of the data to be convolved into the second RAM of the first row; Store the data of the Ath row and the Fth column of the data to be convolved into the third RAM of the first row; D=1+3j, j=0,1,……and j is an integer; E=2+3j, j=0,1,……and j is an integer; F=3+3j, j=0,1,……and j is an integer.

4. The method according to claim 3, characterized in that The Bth row of data to be convolved is sequentially stored in each RAM of the second row of RAM, including: Store the data of the Bth row and the Dth column of the data to be convolved into the first RAM of the second row; Store the data of the Bth row and the Eth column of the data to be convolved into the second RAM of the second row; Store the data of the Bth row and the Fth column of the data to be convolved into the third RAM of the second row; The Cth row of data to be convolved is sequentially stored in each RAM of the third row of RAM, including: Store the data of the Cth row and the Dth column of the data to be convolved into the first RAM of the third row; Store the data of the Cth row and Eth column of the data to be convolved into the second RAM of the third row; The Cth row and Fth column data of the data to be convolved are stored in the third RAM of the third row.

5. A data storage device for a neural network, characterized in that: The data storage device is used to store the data to be convolved of the convolution layer of the neural network using the method according to any one of claims 1 to 4, and the data to be convolved is used to perform convolution using a 3×3 convolution kernel; The data storage device includes a RAM module, and the RAM module includes a 3-row×3-column RAM matrix.

6. A method for reading data of a neural network, characterized in that: Based on the storage method of any one of claims 1 to 4, the data to be convolved is read out from the RAM module, the method comprising the steps of: Set the reading order based on the number of sliding windows of the convolution and the step value of the convolution; Each RAM in the RAM module is read according to the reading order.

7. The method according to claim 6, characterized in that The reading order is set based on the number of sliding windows of the convolution and the step value of the convolution, including: when str=1, When \(M(3k - 3)\leq w\lt M(3k - 2)\), read the first row of RAM, the second row of RAM, and the third row of RAM in sequence; When \(M(3k - 2)\leq w\lt M(3k - 1)\), read the second row of RAM, the third row of RAM, and the first row of RAM in sequence; When \(M(3k - 1)\leq w\lt M(3k)\), read the third row of RAM, the first row of RAM, and the second row of RAM in sequence; where \(w\) represents the number of sliding window times, \(str\) represents the step value, \(k\) is a positive integer, \(M = N - 2\), and \(N\) represents the number of columns of the data to be convolved.

8. The method according to claim 6, characterized in that Setting the reading order based on the number of sliding window times of convolution and the step value of convolution includes: when \(str = 2\), When \(p(3k - 3)\leq w\lt p(3k - 2)\), read the first row of RAM, the second row of RAM, and the third row of RAM in sequence; When \(p(3k - 2)\leq w\lt p(3k - 1)\), read the third row of RAM, the first row of RAM, and the second row of RAM in sequence; When \(p(3k - 1)\leq w\lt p(3k)\), read the second row of RAM, the third row of RAM, and the first row of RAM in sequence; in, FLOOR() means taking the integer part of the quotient in the division method; w means the number of sliding windows, str means the step value, k is a positive integer, and N means the number of columns of data to be convolved.

9. The method according to claim 7 or 8, characterized in that: Setting the reading order based on the number of sliding window times of convolution and the step value of convolution further includes: when \(w = 0\), Read the first RAM, the second RAM, and the third RAM of each row in sequence in each row of RAM.

10. The method according to claim 7 or 8, characterized in that: Setting the reading order based on the number of sliding window times of convolution and the step value of convolution further includes: in each row of RAM, When \(str = 1\) and \(w\bmod 3 = 1\), or \(str = 2\) and \(w\bmod 3 = 2\), read the second RAM, the third RAM, and the first RAM of that row in sequence; When \(str = 1\) and \(w\bmod 3 = 2\), or \(str = 2\) and \(w\bmod 3 = 1\), read the third RAM, the first RAM, and the second RAM of that row in sequence; When \(str = 1\) and \(w\bmod 3 = 3\), or \(str = 2\) and \(w\bmod 3 = 3\), read the first RAM, the second RAM, and the third RAM of that row in sequence; where \(a\bmod b\) represents the remainder of \(a\) divided by \(b\).

Images