CNN data processing method, device and chip based on sensing, storage and computing integrated architecture

Abstract

The present invention relates to the technical field of convolutional neural networks, and discloses a CNN data processing method, device and chip based on a sensing, storage and computing integrated architecture, the method comprising the following steps: splitting convolution to form N convolution groups, constructing N reorganization weights, forming data to be processed according to the N reorganization weights, and multiplying and accumulating the data to be processed with each reorganization weight. The method, device and chip of the present invention can perform parallel convolution processing on input data in the analog domain, improve computing efficiency, and improve resource utilization.

Description

Technical Field

[0001] The present invention relates to a convolutional neural network (CNN), and in particular to a CNN data processing method, device and chip based on a sensing-storage-computing integrated architecture. Background Art

[0002] In recent years, artificial intelligence algorithms represented by Convolutional Neural Networks (CNN) have been widely deployed on tens of billions of cloud and edge devices, and have achieved breakthrough success in security monitoring, autonomous driving, medical diagnosis and other fields. However, the execution process of the CNN algorithm requires the processing of a large amount of unstructured data; in addition, better algorithm performance usually requires greater network depth and more parameter capacity, which poses great challenges to existing hardware systems.

[0003] In the field of machine vision, the following processing is usually used: first, the visual sensor acquires analog image signals, and obtains digital image information through an analog-to-digital converter; then, the digital image information completes the training / inference process on computing chips such as graphics processing units (GPUs) and neural network processing units (NPUs). The analog-to-digital conversion process requires a large number of highly parallel and high-precision ADCs, which will bring huge power consumption and area overhead, known as the "conversion wall". At the same time, due to the basic characteristics of the von Neumann architecture that storage and computing are separated from each other, data is frequently moved between the processor and the memory, which greatly increases the memory access latency and power consumption, known as the "storage wall". At present, the "conversion wall" and "storage wall" problems have become important challenges faced in the hardware deployment of CNN algorithms, especially in edge application scenarios with limited resources and extreme power consumption, where the "conversion wall" and "storage wall" problems are further highlighted.

[0004] The integrated sensing, storage and computing technology provides an effective solution to the "conversion wall" and "storage wall" problems. The integrated storage and computing technology deeply integrates computing and storage units, which can significantly reduce data access latency and overhead and realize large-scale parallel computing. The integrated sensing, storage and computing technology further integrates sensing, computing and storage units to solve the "storage wall" problem caused by frequent data movement at the source of the data. According to the hardware architecture definition, the integrated sensing, storage and computing technology can be divided into a full analog domain solution and an analog-to-digital hybrid domain solution. The full analog domain solution executes the target algorithm in the analog domain, completely eliminating the "conversion wall" problem. The disadvantage is that it cannot meet the processing requirements of complex tasks. The analog-to-digital hybrid domain solution converts the compressed analog signal to the digital domain through the ADC and then completes subsequent processing, which can greatly reduce the area and power consumption of the ADC.

[0005] Convolution operation is the most important operator form in the CNN algorithm. However, the current solution makes it difficult to implement highly parallel convolution operations in the analog domain within the sensor. This is mainly because the convolution operation needs to be performed slidingly on the same feature map, and the analog signal read out by the sensor is not friendly to the multiplexed input brought by sliding, resulting in low resource utilization or limited frame rate. Summary of the invention

[0006] In order to solve the above problems, the present invention provides a CNN data processing method, a CNN data processing device and a sensing-storage-computing integrated chip based on a sensing-storage-computing integrated architecture, which can realize parallel convolution operations and improve computing efficiency.

[0007] The first aspect of the present disclosure provides a CNN data processing method based on a sensing, storage and computing integrated architecture, with a convolution kernel size of K and a step size of S, comprising the following steps: splitting the convolution to form N convolution groups, constructing N reorganization weights to form data to be processed, and multiplying and accumulating the data to be processed with each reorganization weight.

[0008] In the step of splitting convolution to form N convolution groups, the convolution kernel is split into N convolution groups in sequence by row. When K / S is an integer, N=K / S, and each convolution group is S rows; when K / S has a remainder, N=int(K / S)+1. Except for the last convolution group which is less than S rows, the remaining convolution groups are all S rows. In the step of constructing N reorganization weights, when the convolution group is S rows, the corresponding reorganization weights are constructed with the convolution group as a repeating unit; when the convolution group is less than S rows, the S rows are supplemented with blank rows to form a repeating unit to construct the corresponding reorganization weights. In the step of forming the data to be processed, the input data (X+K-1)*K is obtained to form the data to be processed, and X≥1. In the step of multiplying and accumulating the data to be processed and each reorganization weight, the data to be processed and each reorganization weight are multiplied and added in turn, and the results of multiple multiplications and additions are added to obtain int((X-1) / S+1) convolution results in the column direction. Among them, the results of multiplication and addition of the data to be processed and the first reorganization weight are added according to the output storage group to obtain the first intermediate convolution result, and the results of multiplication and addition of the data to be processed and the second reorganization weight are added according to the output storage group and the first intermediate convolution result to obtain the second intermediate convolution result; and so on, the results of multiplication and addition of the data to be processed and the Nth reorganization weight are added according to the output storage group and the N-1th intermediate convolution result to obtain int ((X-1) / S+1) convolution results in the column direction.

[0009] Specifically, in some embodiments, constructing N recombination weights specifically includes that the number of rows of the nth recombination weight is (int((X-1) / S)+n)×S, the nth recombination weight has (n-1)×S rows of blank rows, and the nth recombination weight is constructed with the nth repeating unit after the blank row; wherein 1≤n≤N.

[0010] In some other embodiments, constructing N reorganization weights specifically includes constructing an nth reorganization weight with a row number of (int((X-1) / S)+1)×S rows using an nth repetitive unit, wherein 1≤n≤N; the data to be processed is constructed into N data units to be processed, when 1≤n≤N-1, the number of rows of the nth data unit to be processed is (int((X-1) / S)+1)×S, and the nth data unit to be processed is obtained by removing the first (n-1)×S rows from the input data (X+K-1)*K; when n=N, the nth data unit to be processed is obtained by removing the first (N-1)×S rows from the input data (X+K-1)*K; the first data unit to be processed is constructed into N data units to be processed, when 1≤n≤N-1, the number of rows of the nth data unit to be processed is (int((X-1) / S)+1)×S, and the nth data unit to be processed is constructed by removing the first (n-1)×S rows from the input data (X+K-1)*K; when n=N, the nth data unit to be processed is constructed by removing the first (N-1)×S rows from the input data (X+K-1)*K; The results of multiplication and addition of the processed data unit and the first reorganization weight are added according to the output storage group to obtain the first intermediate convolution result, and the results of multiplication and addition of the second data unit to be processed and the second reorganization weight are added according to the output storage group and the first intermediate convolution result to obtain the second intermediate convolution result; the results of multiplication and addition of the third data unit to be processed and the third reorganization weight are added according to the output storage group and the second intermediate multiplication and addition result to obtain the third intermediate convolution result; and so on, the results of multiplication and addition of the Nth data unit to be processed and the Nth reorganization weight are added according to the output storage group and the N-1th intermediate convolution result to obtain int ((X-1) / S+1) convolution results in the column direction.

[0011] The second aspect of the present disclosure provides a CNN data processing device based on a sensing, storage and computing integrated architecture, the device comprising a first acquisition module, a second acquisition module, a processing module and a data flow control unit. The first acquisition module is used to acquire input data (X+K-1)*K; the second acquisition module is used to acquire the weight value of the convolution kernel, the size of the convolution kernel is K, and the step length is S; the second acquisition module is electrically connected to the processing module. The data flow control unit constructs N repetition units according to the size and step size of the convolution kernel, and the repetition unit is S rows; the convolution kernel is split into N convolution groups in sequence by row, when K / S is an integer, N=K / S, and each convolution group is S rows; when K / S has a remainder, N=int(K / S)+1, except for the last convolution group which is less than S rows, the remaining convolution groups are all S rows; when the convolution group is S rows, the repetition unit is the convolution group; when the convolution group is less than S rows, the repetition unit includes the convolution group and the empty row; the data flow control unit is used to control the convolution groups stored in the second acquisition module to cyclically transmit the weight values ​​to the processing module in turn, thereby forming N reorganization weights. Among them, the processing module is used to multiply the input data (X+K-1)*K with the N reorganization weights in turn, and add the results of multiple multiplications to obtain int((X-1) / S+1) convolution results in the column direction. Among them, the result of multiplying the input data with the second reorganization weight and the result of multiplying the to-be-processed data with the first reorganization weight are added together according to the output storage group to obtain the first intermediate cumulative result; the result of multiplying the input data with the third reorganization weight and the first intermediate cumulative result are added together according to the output storage group to obtain the second intermediate cumulative result; and so on, the result of multiplying the input data with the Nth reorganization weight and the N-1th intermediate cumulative result are added together according to the output storage group to obtain int ((X-1) / S+1) convolution results in the column direction.

[0012] Specifically, the processing module of the CNN data processing device includes multiple row multipliers (row_PE) and int ((X-1) / S+1) capacitors, row_PE is used to process the multiplication and addition operations of input data and reorganization weights, and each capacitor is used to record a convolution result.

[0013] More specifically, in some embodiments, the number of rows of the nth reorganized weight of the CNN data processing device is (int((X-1) / S)+n)×S, the nth reorganized weight has (n-1)×S rows of empty rows, and the nth reorganized weight is constructed with the nth repeating unit after the empty row, wherein 1≤n≤N. The processing module further includes a first switch array, and the first switch array is used to accumulate the result of multiplying and adding the input data and the nth reorganized weight in row_PE to the corresponding capacitor.

[0014] In some other embodiments, the CNN data processing device also includes a second switch array, which is used to transmit the input data stored in the first acquisition module to multiple rows of row_PE and multiply them with the nth reorganization weight, wherein 1≤n≤N; the multiple rows of row_PE are fixedly connected to corresponding capacitors; the nth reorganization weight is specifically constructed as a matrix with a row number of int((X-1) / S+1)×S rows constructed with the nth repeating unit.

[0015] Optionally, multiple rows of row_PE are divided into multiple groups in sequence according to S rows, and row_PE in the same group are connected to the same capacitor.

[0016] Furthermore, the CNN data processing device is used to process the convolution calculation of the total input data (X+K-1)*M, where M is an integer greater than or equal to K. The first acquisition module is constructed as an analog register, which sequentially obtains multiple input data (X+K-1)*K from the total input data (X+K-1)*M for parallel convolution calculation. After processing one parallel convolution calculation, the analog register moves S columns to the right from the total input data (X+K-1)*M to obtain the next group (X+K-1)*K; and so on, until the convolution result P*R is obtained, P=int((X-1) / S+1), R=int((MK) / S+1). The CNN data processing device also includes a pooling unit, and the convolution result P*R is processed by the pooling unit to obtain a feature map. Specifically, the second acquisition module is constructed as a SARM array.

[0017] The third aspect of the present disclosure provides a sensing, storage and computing integrated chip, including the CNN data processing device of the second aspect.

[0018] Features and advantages of the present disclosure include:

[0019] The CNN data processing method based on the sensor-storage-computation integrated architecture provided by the present disclosure distributes each convolution calculation of the input data into N sub-cycles, so that after the multiplication and addition operations of N sub-cycles, the results are accumulated according to the output storage group, and the convolution value of the input data (X+K-1)*K can be obtained. The CNN data processing method can perform parallel convolution processing on the input data in the analog domain, improve the calculation efficiency, and improve the resource utilization.

[0020] The CNN data processing device based on the integrated sensing, storage and computing architecture provided by the present disclosure adopts the aforementioned CNN data processing method to disperse the convolution kernel into multiple repeated units, so that the convolution operation is dispersed in multiple sub-cycles. The data processing device is suitable for high-parallelism and high-speed sliding convolution hardware calculations in sensing, storage and computing scenarios, and can effectively solve the problem of slow convolution and matrix multiplication operators in the current integrated sensing, storage and computing chips in processing deep learning convolution and full connection operations, and has the advantages of high parallelism, low power consumption, and area friendliness. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A schematic diagram of a sensing, storage and computing integrated architecture is shown;

[0023] FIG. 2A to FIG. 2C The calculation process of the CNN data processing method based on the sensor-storage-computation integrated architecture disclosed in the present invention in a parallel computing cycle is shown;

[0024] Figure 3A A schematic diagram showing a specific implementation of the CNN data processing method disclosed in the present invention;

[0025] Figure 3B A schematic diagram showing another specific implementation of the CNN data processing method disclosed in the present invention;

[0026] Figure 4A Shows the use of Figure 3A Schematic diagram of the calculation process and resource allocation of the implementation method;

[0027] Figure 4B Shows the use of Figure 3B Schematic diagram of the calculation process and resource allocation of the implementation method;

[0028] Figure 5 The overall structural diagram of the sensing, storage and computing integrated chip disclosed in the present invention is shown;

[0029] Figure 6 A schematic diagram of a CNN data processing device based on a sensing-storage-computing integrated architecture disclosed in the present invention is shown;

[0030] Figure 7 Shows the use of Figure 3A Another computing process and resource allocation diagram of an implementation method. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0032] To facilitate understanding of the contents of this disclosure, the sensor storage and computing integrated system is first introduced. Figure 1 , taking the integrated sensing, storage and computing system used to collect and process image information as an example for explanation. The integrated sensing, storage and computing system includes a sensor array, an analog domain storage and computing integrated computing core, an analog-to-digital conversion module, and all or part of a digital domain storage and computing integrated or other computing core. The sensor array collects image information and converts the optical signal into an electrical signal in the analog domain, such as an analog value of voltage or current; after the analog value is processed in the analog domain storage and computing integrated computing core, it is converted into digital data through the analog-to-digital conversion module, so as to facilitate entry into the subsequent digital domain computing core.

[0033] Convolutional neural networks are often used in computing kernels to process analog values ​​or digital values. The specific operation process of convolutional neural networks is that the convolution kernel slides on the input data, performs convolution operations and generates feature maps. For example, the convolution kernel can first slide from left to right, and the distance of each slide is the stride. If it slides to the rightmost side of the input data, it returns horizontally to the leftmost side and slides down by one stride, and then slides from left to right again until it slides to the lower right corner of the input data. The convolution kernel can also first slide from top to bottom, and the distance of each slide is the stride. If it slides to the bottom side of the input data, it returns vertically to the top side and slides to the right by one stride, and then slides from top to bottom again until it slides to the lower right corner of the input data. At the current sliding position, each element in the convolution kernel is multiplied and added with the input matrix element at the corresponding position. Each sliding calculation obtains a result, and the final result matrix is ​​the result of the convolution operation, that is, the feature map.

[0034] The computing core is usually composed of multiple processing units (PEs), and the input data and the convolution kernel are multiplied and added in the PE. In the analog domain, the results of the multiplication and addition operations can be stored in the form of charge on the output unit with a capacitor as the basic structure to obtain the convolution result of a single slide. In the existing convolutional neural network methods, the parallelism of a single convolution calculation is poor, and the processing unit cannot achieve full parallel processing, which is inefficient.

[0035] The present invention provides a CNN data processing method based on a sensor-storage-computation integrated architecture, which is used to perform parallel convolution processing on input data to improve computing efficiency. The input data may be at least one of image data, video data and voice data. In the specific embodiment, image data is used as an example for illustration. The size of the convolution kernel is K and the step size is S. The weight matrix of the convolution kernel is K*K, and Wn represents the row vector of the weight matrix. The convolution kernel can be expressed as [W1, W2, ...W K] The convolution kernel is split into multiple convolution groups according to the step size S, and the reorganization weights are constructed based on the convolution groups. The input data and each reorganization weight are multiplied and added in turn, and the multiple calculation results are added accordingly to obtain multiple convolution results in the column direction.

[0036] Specifically, the CNN data processing method based on the integrated sensing, storage and computing architecture of the present invention can be used to process input data (X+K-1)*M, where X+K-1 is the number of rows of the input data, and M is the number of columns of the input data, wherein X is an integer greater than or equal to 1, and M is an integer greater than or equal to K. The input data can be an analog signal or a digital signal, and the analog signal is used as an example for explanation below. Specifically, in a complete parallel computing cycle, the input data (X+K-1)*K and all the reorganization weights can be multiplied and added in sequence, and parallel processing is performed to obtain int ((X-1) / S+1) convolution results in the column direction. After processing a parallel computing cycle, S columns are moved to the right in the input data (X+K-1)*M, and the next set of input data (X+K-1)*K is read, and multiple convolution results of the new set of input data in the column direction are calculated in parallel. And so on, the convolution calculation of the input data (X+K-1)*M is completed.

[0037] Refer to the following FIG. 2A to FIG. 2C , the calculation process of a parallel calculation cycle is introduced in detail. The data processing method of the present invention comprises the following steps: splitting the convolution kernel to form N convolution groups; constructing N reorganization weights; forming data to be processed; multiplying and accumulating the data to be processed and the N reorganization weights.

[0038] In the step of splitting convolution to form convolution groups, the convolution kernel is sequentially split into N convolution groups according to the step size S. When K / S is an integer, N=K / S, and each convolution group is S rows; when K / S has a remainder, N=int(K / S)+1, except for the last convolution group which is less than S rows, the remaining convolution groups are all S rows. In the step of constructing N reorganization weights, when the convolution group is S rows, the corresponding reorganization weights are constructed with the convolution group as a repeating unit; when the convolution group is less than S rows, blank rows are used to fill S rows to form a repeating unit to construct the corresponding reorganization weights.

[0039] In the step of forming the data to be processed, input data (X+K-1)*K is obtained, and the data to be processed includes at least part of the input data. In some embodiments, the data to be processed is the input data (X+K-1)*K; in other embodiments, the data to be processed is constructed into a plurality of data units to be processed, and each data unit to be processed is a part of the input data (X+K-1)*K.

[0040] In the step of multiplying and accumulating the data to be processed and the N reorganization weights, after N sub-cycles, the data to be processed and each reorganization weight are multiplied and added in turn, and the results of multiple calculations are added correspondingly to obtain int ((X-1) / S+1) convolution results in the column direction. In other words, in each sub-cycle of parallel calculation, a part of the convolution result is calculated separately, and the convolution results of each part are accumulated to obtain the final convolution result.

[0041] Specifically, in some embodiments, in the first sub-cycle of the parallel computing cycle, the results of multiplication and addition of the data to be processed and the first reorganization weight (a total of N reorganization weights) are added according to the output storage group to obtain the first intermediate convolution result; in the second sub-cycle of the parallel computing cycle, the results of multiplication and addition of the data to be processed and the second reorganization weight are added according to the output storage group and the first intermediate convolution result to obtain the second intermediate convolution result; and so on, in the Nth sub-cycle of the parallel computing cycle, the results of multiplication and addition of the data to be processed and the Nth reorganization weight are added according to the output storage group and the N-1th intermediate convolution result to obtain int ((X-1) / S+1) convolution results in the column direction. The results of multiplication and addition are grouped according to the step size S, and the results of multiplication and addition in each group are added separately. This type of embodiment will be described in detail later.

[0042] It should be added that, in some other embodiments, in each sub-cycle of the parallel computing cycle, the data to be processed is multiplied and added correspondingly with each reorganization weight, and finally the results of the multiplication and addition are added according to the output storage group to obtain int ((X-1) / S+1) convolution results in the column direction.

[0043] In the CNN data processing method based on the sensor-storage-computation integrated architecture disclosed in the present invention, each convolution calculation of the input data is dispersed into N sub-cycles, so that after the multiplication and addition operations of N sub-cycles, the results are accumulated according to the output storage group, and the convolution value of the input data (X+K-1)*K can be obtained. According to various embodiments disclosed in the present invention, the parts related to the same convolution value are divided into one output storage group. For example, C related to the convolution value Co1 11 , C 12 , C 13 , C 14 , C 15 , C 1L1 , C 1L2 is an output storage group, where C 11 , C 12 , C 13 , C 14 , C 15 are the results of row multiplication and addition corresponding to the convolution kernel, C 1L1 is the first intermediate convolution result, C 1L2 This is the second intermediate convolution result, which will be introduced in detail later.

[0044] Reference FIG. 2A to FIG. 2C , the following takes K=5, S=2 as an example for detailed description. The convolution kernel is [W1, W2, W3, W4, W5]. The convolution kernel is divided into 3 convolution groups with a step size S of 2, that is, N is 3, the first convolution group is [W1, W2], the second convolution group is [W3, W4], and the third convolution group is [W5]. The first convolution group is 2 rows, and the first reorganization weight is constructed with [W1, W2] as the repeating unit, see Figure 2A , the first reorganization weight is [W1, W2, W1, ..., W2]. The second convolution group is 2 rows, and the second reorganization weight is constructed with [W3, W4] as a repeating unit, see Figure 2B , the second reorganization weight is [W3, W4, W3, ..., W4]. The third convolution group is 1 row, and a blank row is added to the third convolution group to form a repeating unit [W5, _ ], see Figure 2C , the third recombination weight is [W5, _, W5, ..., _ ], where "_" represents a blank line. More specifically, in each recombination weight, the number of rows constructed by the repeating unit is (int((X-1) / S)+1)×S. For the convenience of explanation, (X-1) / S can be set as an integer, and the number of rows constructed by the repeating unit is (int((X-1) / S)+1)×S=X-1+S=X-1+2=X+1. Since K=5, X+1 corresponds to X+K-4.

[0045] Continue to see FIG. 2A to FIG. 2C , the input data is a matrix (X+K-1)*K, with A i Represents the row vector of the input matrix. The matrix of input data can be expressed as [A1, A2, ..., A X+K-1 ]. For the convenience of introduction, the letter P will be used to replace int((X-1) / S+1) in the following.

[0046] Define the convolution result of matrix [A1, A2, A3, A4, A5] and matrix [W1, W2, W3, W4, W5] as Co1, Co1=C 11 +C 12 +C 13 +C 14 +C 15 Among them, C 11 It is obtained by multiplying the corresponding elements of A1 and W1 and then accumulating them. It can be obtained by C 11 =A1×W1. Similarly, C 12 It is obtained by multiplying the corresponding elements of A2 and W2 and then accumulating them. 12 =A2×W2;C 13 It is obtained by multiplying the corresponding elements of A3 and W3 and then accumulating them. 13 =A3×W3;C 14It is obtained by multiplying the corresponding elements of A4 and W4 and then accumulating them. 14 =A4×W4;C 15 It is obtained by multiplying the corresponding elements of A5 and W5 and then accumulating them. 15 =A5×W5. In the present disclosure, C is calculated in the first sub-period 11 , C 12 , C is calculated in the second sub-period 13 , C 14 , C is calculated in the third sub-period 15 , the multiplication and addition results in the three sub-cycles are accumulated to obtain the convolution result Co1.

[0047] See also Figure 2A , in the first sub-cycle of the parallel computing cycle, the input data [A1, A2, ..., A X+K-4 ] and each row of the first regrouping weight [W1, W2, W1, ..., W2] are multiplied and added to obtain C 11 , C 12 , C 21 , C 22 , ..., C P1 , C P2 . 11 , C 12 , C 21 , C 22 , ..., C P1 , C P2 Divide into multiple groups according to the step size S, and add the multiplication and addition values ​​of each group to obtain the first intermediate convolution result [C 1L1 , C 2L1 , ..., C PL1 ]. The step size S is 2, C 11 , C 12 As a group, C 11 Add C 12 The sum is C 1L1 ; C 21 , C 22 As a group, C 21 Add C 22 The sum is C 2L1 ; and so on, C P1 , C P2 As a group, C P1 Add C P2 The sum is C PL1 .

[0048] See also Figure 2B , in the second sub-cycle of the parallel computing cycle, the input data [A3, A4, ..., A X+K-2] and each row of the second regrouping weight [W3, W4, W3, ..., W4] are multiplied and added to obtain C 13 , C 14 , C 23 , C 24 , ..., C P3 , C P4 . 13 , C 14 , C 23 , C 24 , ..., C P3 , C P4 Divide into multiple groups according to the step size S, and add the multiplication and addition operation value of each group to the first intermediate convolution result to obtain [C 1L2 , C 2L2 , ..., C PL2 ]. The step size S is 2, C 13 , C 14 As a group, C 13 , C 14 With C 1L1 The sum is C 1L2 ; C 23 , C 24 As a group, C 23 , C 24 With C 2L1 The sum is C 2L2 ; and so on, C P3 , C P4 As a group, C P3 , C P4 With C PL1 The sum is C PL2 .

[0049] See also Figure 2C , in the third sub-cycle of the parallel computing cycle, the input data [A5, A6, ..., A X+K-1 ] and each row of the third regroup weight [W5, _, W5, ..., W5, _] to obtain C 15 ,_,C 25 ,_,...,C P5 , _. Among them, C 15 It is obtained by multiplying the corresponding elements of A5 and W5 and then accumulating them. 25 It is obtained by multiplying the corresponding elements of A7 and W5 and then accumulating them. Similarly, C P5 By A X+K-1 Multiply each element of W5 and add them together. 15 ,_,C 25 ,_,...,C P5, _ are divided into multiple groups according to the step size S, and the multiplication and addition operation values ​​of each group are added to the corresponding second intermediate convolution result to obtain the final convolution result [Co1, Co2, ..., Co P ]. The step size S is 2, C 15 Grouped with adjacent empty rows, C 15 With C 1L2 The sum is Co1; C 25 Grouped with adjacent empty rows, C 25 With C 2L2 The sum of is Co2; and so on, C P5 Grouped with adjacent empty rows, C P5 With C PL2 The sum is Co P .

[0050] Specifically, in the CNN data processing method based on the sensor-storage-computation integrated architecture of the present invention, the multiplication and addition operation can be realized by setting a number of row_PEs, and the length of row_PE is K, that is, each row contains K PEs. More specifically, in the entire parallel computing cycle, the number of rows of row_PE used for the operation can be (X+K-1) or (int((X-1) / S+1))×S (that is, P×S). Input data [A1, A2, ..., A X+K-1 ] is stored in the simulation register. The input data stored in the simulation register corresponds to the reorganized weights and performs multiplication and addition operations in row_PE to obtain k0×C 11 , k0×C 12 , k0×C 13 , k0×C 14 , k0×C 15 , where k0 is a circuit constant. The accumulation of multiplication and addition results is achieved by setting int ((X-1) / S+1) capacitors C. The total charge change on each capacitor C is linearly related to the convolution result. Specifically, the multiplication and addition result k0×C 11 , k0×C 12 , k0×C 13 , k0×C 14 , k0×C 15 All are input to the same capacitor C, such as capacitor C1. Preferably, row_PE can be grouped according to step size S, and in each sub-cycle of the parallel computing cycle, row_PE in the same group is connected to the same capacitor C. According to the computing requirements, each row_PE can be connected to a fixed capacitor C or to different capacitors C.

[0051] More specifically, the analog registers store input data [A1, A2, ..., A X+K-1 ], in some embodiments, see Figure 3A, the number of rows of row_PE is (X+K-1), the number of rows of analog registers is the same as the number of rows of row_PE, each row of analog registers is directly connected to each row of row_PE, and each group of row_PE can be connected to at least one capacitor C through the first switch array. The row_PE rows are divided into multiple groups according to the step size S, and the same group of row_PE can be connected to the same capacitor C through the first switch array. In some embodiments, in order to realize that S is any integer from 1 to K, the pth capacitor C can be connected to the pth to Kth capacitors C through the first switch array.

[0052] row_PE of p×K rows, where 1≤p≤P. For example, when K is 7, the first capacitor C can be connected to row_PE of rows 1 to 7, the second capacitor C can be connected to row_PE of rows 2 to 14, and so on.

[0053] In each sub-cycle of the parallel computing cycle, P groups of row_PE are respectively connected to P capacitors C through the first switch array, and each capacitor C is used to accumulate to obtain a convolution result. In different sub-cycles of the parallel computing cycle, the row_PE groups used to calculate the multiplication and addition operations related to the same convolution result are connected to the same capacitor C through the first switch array.

[0054] See also Figure 4A For example, taking the step size S as 2 and K=5 as an example, in the first sub-period, multiplication and addition are performed in row_PE1 to row_PE4 to obtain k0×C 11 , k0×C 12 , k0×C 21 , k0×C 22 , row_PE1 and row_PE2 are connected to capacitor C1 through the first switch array, and row_PE3 and row_PE4 are connected to capacitor C2 through the first switch array. In the second sub-period, k0×C is obtained by multiplication and addition in row_PE3 to row_PE6. 13 , k0×C 14 , k0×C 23 , k0×C 24 , row_PE3 and row_PE4 are connected to capacitor C1 through the first switch array, and row_PE5 and row_PE6 are connected to capacitor C2 through the first switch array, that is, the first switch array moves the capacitor C down two rows as a whole and connects it to the row_PE group currently used for operation. In the third sub-cycle, k0×C is obtained by multiplication and addition in row_PE5 to row_PE8 respectively. 15 , _ , k0×C 25, _ , row_PE5 is connected to capacitor C1 through the first switch array, row_PE7 is connected to capacitor C2 through the first switch array, row_PE6 and row_PE8 are closed because there is no weight input and can be regarded as not connected to capacitors C1 and C2 respectively.

[0055] Taking capacitor C1 as an example, in the first sub-period, the charge on capacitor C1 changes for:

[0056]

[0057] In the second and third sub-periods, the charge change on capacitor C1 is:

[0058]

[0059]

[0060] After three sub-cycles, the total charge on capacitor C1 changes to:

[0061]

[0062] k0 is a circuit constant. Therefore, after one parallel calculation cycle, the total charge change on capacitor C1 is a linear mapping of the actual convolution operation result, and is reflected as a voltage change on the plate of capacitor C1. The result can be read out through ADC later.

[0063] In other embodiments, see Figure 3B , the number of rows of row_PE is P×S, when (X-1) / S is an integer and S is 2, P×S=X+1). Each row of the analog register is connected to each row of row_PE through the second switch array, and each row_PE is connected to the corresponding capacitor C through the switch array according to the calculation needs. Optionally, since the number of rows of row_PE used for calculation in the entire parallel calculation cycle is (int((X-1) / S+1))×S (i.e., P×S rows), only P×S rows of row_PE may be set. In particular, each row_PE row is connected to each capacitor C in groups through the capacitor configuration matrix.

[0064] In each sub-cycle of the parallel computing cycle, the P×S rows in the analog register are connected to the P×S rows of row_PE through the second switch array. Each group of row_PE is a multiplexing unit, and the convolution calculations related to the same convolution result are multiplied and added in the same row_PE group so as to be accumulated in the same capacitor C. More specifically, in order to satisfy the convolution calculation that can be performed for any value of the step size S, for example, the step size S is 1 and the number of capacitors C is X. Specifically, in some embodiments, each row_PE can be connected to all capacitors C through a capacitor configuration matrix. In particular, in some embodiments, the 1st to Kth rows of row_PE can be connected to capacitor C1 through a capacitor configuration matrix, the 2nd to 2Kth rows of row_PE can be connected to capacitor C2 through a capacitor configuration matrix, and so on.

[0065] See also Figure 4B For example, if the step size S is 2, K=5, and (X-1) / S is an integer, the input data [A1, A2, ..., A X+K-1 ] are stored in rows of the analog register, and each row_PE group is connected to a capacitor C in sequence. The first row_PE group is connected to capacitor C1, and so on. In the first sub-cycle, rows 1 to P×S rows (i.e., to row X+1) of the analog register are connected to P×S rows of row_PE in sequence through the second switch array, and multiplication and addition operations are performed respectively. k0×C is obtained in row_PE1 and row_PE2 of the first group respectively. 11 and k0×C 12 Accumulated into capacitor C1. In the second sub-period, the 3rd to (P×S+2)th rows (i.e., to the X+3th row) of the analog register are sequentially connected to the P×S rows row_PE through the second switch array, i.e., the second switch array moves the analog register up two rows as a whole and then connects it to the row_PE group accordingly, and performs multiplication and addition operations respectively, and obtains k0×C in row_PE1 and row_PE2 of the first group respectively. 13 and k0×C 14 In the third step, the 5th row to the (P×S+3)th row (i.e., to the X+4th row) of the analog register are sequentially connected to the P×S rows row_PE through the second switch array, and multiplication and addition operations are performed respectively. k0×C is obtained in row_PE1 of the first group. 15 , accumulated in capacitor C1, row_PE2 in the first group is closed due to no weight input, which can be regarded as no output of row_PE2.

[0066] It should be added that, in some embodiments, the number of capacitors C can be other, for example, different number configurations can be selected according to the acceptable number of calculation cycles and delays, such as S / 2. Specifically, for example, if S / 2 is selected, the number of capacitors C is increased to twice the original number on the basis of the original S configuration. The input matrix stored in the analog register is controlled to flow through the second switch array, and the number of rows of the second switch array is less than the number of rows of the input matrix. After each sub-cycle, the input index of the input matrix on the row increases by S, and the input convolution kernel weight of each sub-cycle takes S rows for circulation.

[0067] The present invention also discloses a CNN data processing device based on a sensing-storage-computing architecture and a sensing-storage-computing integrated chip including the device, which is used to process highly parallel convolution calculations in the analog signal domain, solving the problem of low parallelism and slow speed of the convolution operator in the sensing-storage-computing integrated chip. The data processing device disclosed in the present invention adopts the aforementioned CNN data processing method to disperse the convolution kernel into multiple repeating units, so that the convolution operation is dispersed in multiple sub-cycles.

[0068] For ease of understanding, the overall structure of the sensing, storage and computing integrated chip disclosed in the present invention is first introduced. Figure 5 In the integrated sensing, storage and computing chip, a typical solution is that the sensing pixel array first collects the light signal of the external environment and converts the light signal into an analog voltage or current signal. These analog signals then continue to pass through the preprocessing circuit for operations such as correlated double sampling. The preprocessed analog voltage or current signal is input into the data processing device for high-parallel convolution calculation. The total charge change on each capacitor C is read out through the ADC, which is continued for subsequent calculation and processing in the digital domain, and finally the feature map is output.

[0069] Specifically, see Figure 6, the CNN data processing device includes an analog register, a parallel convolution calculation array, an SRAM array and a data flow control unit. The analog register can realize the temporary storage of the voltage or current signal transmitted by the previous circuit, and is used to store the image information (i.e., input data) preprocessed by the sensing end circuit, such as image analog voltage information. The SRAM array is used to store the weight value of the convolution kernel, and the scale of the SRAM array is related to the convolution kernel size, weight accuracy and number of channels. The parallel convolution calculation array specifically includes multiple rows of row_PE, and more specifically, the multiple rows of row_PE can be composed of an analog-to-digital hybrid multiplication-addition unit array and a control switch array. The analog-to-digital hybrid multiplication-addition unit has a variety of structural options, and a typical solution is a current domain multiplication-addition structure based on a current mirror and a digital switch. In the parallel calculation cycle, the data flow control unit is used to control the input data stored in the analog register and the weight value of the SRAM array to perform multiplication and addition operations in steps in the row_PE row in the parallel convolution calculation array, and accumulate the multiplication and addition operation results in groups to obtain the convolution result. In each step of the parallel calculation cycle, some or all of the row_PE can participate in the operation at the same time.

[0070] Specifically, the data flow control unit reads the size K and step size S of the convolution kernel, splits the convolution kernel stored in the SRAM array into N groups according to K and S, and completes the convolution operation in N sub-cycle calculations. Each row of the SRAM array is electrically connected to each row of the parallel sliding convolution calculation array, and the weight value of the convolution kernel stored in the SRAM array can be transmitted to each row of the parallel sliding convolution calculation array through the control of the data flow control unit. More specifically, in each sub-cycle, the data flow control unit controls the SRAM array to cyclically transmit the weight of the convolution group to the parallel sliding convolution calculation array, that is, the data flow control unit controls the SRAM array to output the reorganized weights to the parallel sliding convolution calculation array.

[0071] For example, in the first sub-cycle, the data flow control unit controls the input data stored in the analog register to be transmitted to the parallel convolution calculation array, and at the same time, the data flow control unit cyclically transmits the weight of the first convolution group to the parallel convolution calculation array for multiplication and addition operations. By analogy, N-step calculations are completed. In the N-step calculations, the multiplication and addition results of the same convolution are accumulated to finally obtain the convolution result. The data processing device also includes a plurality of capacitors C for accumulating the multiplication and addition results of the same convolution.

[0072] The data processing device disclosed in the present invention adopts a parallel convolution calculation array based on analog voltage or current (from a perception pixel array) and digital weights, a group weight reuse and a data flow control module, and has the advantages of high parallelism, low power consumption, and area friendliness.

[0073] Compared with the existing single convolution operation, the reuse of grouped weights can realize the deployment and operation of sliding algorithms with high parallelism and high resource utilization. The number of rows of row_PE is the same as the number of rows of input data. Taking the 5*5 convolution kernel as an example, when performing a single convolution operation, 5 rows of row_PE are used at a time, and many row_PEs are idle, with low utilization and low computational efficiency. Although the computational efficiency is high when the 5*5 convolution kernel is convolved in columns, a large number of row_PEs need to be added on the basis of the original row_PE, which is not conducive to the arrangement of components. The data processing device disclosed in the present invention can split the 5*5 convolution kernel into 2*5+2*5+1*5 forms according to the step size S of 2, and perform multiplication and addition operations in three steps. When only the existing row_PE is used, the computational efficiency can be improved, and the number of row_PEs required for the 5*5 convolution kernel to be convolved in columns is greatly reduced, which is more conducive to the deployment and operation of sliding algorithms with high parallelism and high resource utilization.

[0074] In order to maximize the circuit parallel computing efficiency and resource utilization, the present disclosure provides two data processing devices, an output flow type and an input flow type. Specifically, see Figure 3A In the output flow type data processing device, the number of rows of the analog register and row_PE is (X+K-1), each row of the analog register is directly connected to each row of row_PE, and the data processing device also includes a first switch array, each row of row_PE is divided into groups according to the step length S, and each group of row_PE can be connected to at least one capacitor C through the first switch array. For details of the output flow type data processing device, please refer to the relevant description in the previous method, which will not be repeated here.

[0075] In the data processing device, only N steps of calculation are needed to complete the sliding of the convolution on the column and obtain P convolution results. The convolution slides on the column to obtain a column of convolution results. On this basis, repeating the sliding in the row direction and performing R column convolution operations can obtain a P*R sliding convolution result matrix, realizing the convolution calculation of the input data (X+K-1)*M, where R=int((MK) / S+1).

[0076] Continue to see Figure 6 In some embodiments, the data processing device further includes a pooling unit, and the convolution result array P*R is processed by the pooling unit to obtain a feature map.

[0077] It should be noted that the above is only one or some specific examples of a data processing device. The device can be applied to convolution kernels of any size and convolution operations of any step length (the step length is less than or equal to the convolution kernel size). The size K of the convolution kernel and the step length S determine the number N of convolution groups. When K / S is an integer, N=K / S, and each convolution group is S rows; when K / S has a remainder, N=int(K / S)+1, except that the last convolution group is less than S rows, the remaining convolution groups are all S rows. At the same time, in a parallel computing cycle, N steps of multiplication and addition operations are required. For example, for a 3*3 convolution kernel, the step length is 3, and the three rows of sliding convolution in each sub-cycle only need 1 step to complete the operation; for a 7*7 convolution kernel, the step length is 1, and the sliding convolution of one row in each sub-cycle is implemented in 7 steps.

[0078] See also Figure 7 In the output flow type data processing device, the calculation process is explained by taking the step size S as 3 and K=7 as an example. In the first step, multiply and add in row_PE1 to row_PE6 to obtain k0×C 11 , k0×C 12 , k0×C 13 , k0×C 21 , k0×C 22 , k0×C 23 , row_PE1, row_PE2, and row_PE3 are connected to capacitor C1 through the first switch array, and row_PE4, row_PE5, and row_PE6 are connected to capacitor C2 through the first switch array. In the second step, k0×C is obtained by multiplication and addition in row_PE4 to row_PE9. 14 , k0×C 15 , k0×C 16 , k0×C 24 , k0×C 25 , k0×C 26 , row_PE4, row_PE5, and row_PE6 are connected to capacitor C1 through the first switch array, and row_PE7, row_PE8, and row_PE9 are connected to capacitor C2 through the first switch array, that is, the first switch array moves the capacitor C down three rows as a whole and connects it to the row_PE group currently used for operation. In the third step, multiply and add in row_PE7 to row_PE12 to obtain k0×C 17 ,_,_,k0×C 27 , _, _, row_PE7 is connected to capacitor C1 through the first switch array, row_PE10 is connected to capacitor C2 through the first switch array, row_PE8, row_PE9, row_PE11 and row_PE12 are closed because there is no weight input and can be regarded as not connected to capacitors C1 and C2 respectively.

[0079] Specifically, see Figure 3B In the input flow type data processing device, the number of rows of analog registers is (X+K-1), and each row of the analog register is connected to each row of row_PE through the second switch array. According to the calculation requirements, each row_PE is connected to the capacitor C in groups through the capacitor configuration array. In particular, since the number of rows of row_PE used for calculation in the entire parallel calculation cycle is (int((X-1) / S+1))×S (i.e., P×S rows), in order to improve the utilization of row_PE, only P×S rows of row_PE can be set. The rows of row_PE are divided into groups in sequence according to the step size S, and each group of row_PE can be connected to a capacitor C through the capacitor configuration array. In each sub-cycle of the parallel calculation cycle, the P×S rows in the analog register are connected to the P×S rows of row_PE through the second switch array. Each group of row_PE is a multiplexing unit, and the convolution calculations related to the same convolution result are multiplied and added in the same row_PE group so as to be accumulated in the same capacitor C. That is, in each sub-cycle of the parallel computing cycle, each row_PE group can be connected to the same capacitor C through the capacitor configuration array. For details of the input flow type data processing device, please refer to the relevant description in the previous method, which will not be repeated here.

[0080] The CNN data processing device provided by the present invention is a high-parallel, high-speed sliding convolution hardware calculation suitable for sensing, storage and computing scenarios, which can effectively solve the problem that the convolution and matrix multiplication operators in the current sensing, storage and computing integrated chip are slow in processing deep learning convolution and full connection operations. The CNN data processing device realizes the maximum row parallel convolution operation at the sensor signal readout end through the coordinated configuration of analog registers, parallel convolution calculation arrays and SRAM arrays to meet the application requirements of ultra-high frame rate scenarios.

[0081] The above are only several embodiments of the present disclosure. Those skilled in the art may make various changes or modifications to the embodiments of the present disclosure based on the contents disclosed in the application documents without departing from the spirit and scope of the present disclosure.

Claims

1. A CNN data processing method based on a sensor-storage-computation integrated architecture, characterized in that: The size of the convolution kernel is K, the step size is S, and it includes the following steps: Split convolution to form N convolution groups: Split the convolution kernel into N convolution groups in sequence by row. When K / S is an integer, N=K / S, and each convolution group is S rows; when K / S has a remainder, N=int(K / S)+1. Except for the last convolution group which is less than S rows, the remaining convolution groups are all S rows. Construct N reorganization weights: When the convolution group is S rows, the corresponding reorganization weights are constructed with the convolution group as a repeating unit; when the convolution group is less than S rows, blank rows are used to fill S rows to form a repeating unit to construct the corresponding reorganization weights; Forming data to be processed: obtaining input data (X+K-1)*K to form data to be processed, X≥1, and the data to be processed is at least one of image data, video data and voice data in the analog domain; Multiply and accumulate the data to be processed with each reorganization weight: multiply and add the data to be processed with each reorganization weight in turn, and add the results of multiple multiplications and additions to obtain int((X-1) / S+1) convolution results in the column direction; The results of the corresponding multiplication and addition of the data to be processed and the first reorganization weight in the multi-row multiplier-adder are accumulated and stored in int((X-1) / S+1) capacitors according to the output storage group, and then the first intermediate convolution result is obtained; the results of the corresponding multiplication and addition of the data to be processed and the second reorganization weight in the multi-row multiplier-adder are further accumulated and stored in int((X-1) / S+1) capacitors according to the output storage group, and then the second intermediate convolution result is obtained; and so on, the results of the corresponding multiplication and addition of the data to be processed and the Nth reorganization weight in the multi-row multiplier-adder are further accumulated and stored in int((X-1) / S+1) capacitors according to the output storage group, and then int((X-1) / S+1) convolution results in the column direction are obtained; Each multiplier-adder includes K multiplication-addition units, and in N multiplication-addition operations, at least part of the multipliers-adders in multiple rows are reused.

2. The CNN data processing method according to claim 1, characterized in that: Constructing N reorganization weights specifically includes: The number of rows of the nth recombination weight is (int((X-1) / S)+n)×S, and the nth recombination weight has (n-1)×S rows of blank rows, after which the nth repetitive unit is used to construct the nth recombination weight; wherein 1≤n≤N.

3. The CNN data processing method according to claim 1, characterized in that: Constructing N recombination weights specifically includes constructing an nth recombination weight with (int((X-1) / S)+1)×S rows using the nth repeating unit, where 1≤n≤N; The data to be processed is structured into N data units to be processed. When 1≤n≤N-1, the number of rows of the nth data unit to be processed is (int((X-1) / S)+1)×S, and the nth data unit to be processed is obtained by removing the first (n-1)×S rows from the input data (X+K-1)*K; when n=N, the nth data unit to be processed is obtained by removing the first (N-1)×S rows from the input data (X+K-1)*K. The corresponding multiplication and addition results of the first data unit to be processed and the first reorganization weight in the multi-row multiplier-adder are accumulated and stored in int((X-1) / S+1) capacitors according to the output storage group, and then the first intermediate convolution result is obtained. The corresponding multiplication and addition results of the second data unit to be processed and the second reorganization weight in the multi-row multiplier-adder continue to be accumulated and stored in int((X-1) / S+1) capacitors according to the output storage group, and then the second intermediate convolution result is obtained; and so on, the corresponding multiplication and addition results of the Nth data unit to be processed and the Nth reorganization weight in the multi-row multiplier-adder continue to be accumulated and stored in int((X-1) / S+1) capacitors according to the output storage group, and then int((X-1) / S+1) convolution results in the column direction are obtained; In N multiplication-addition operations, all multiplication-addition devices in multiple rows of multiplication-addition devices are reused.

4. A CNN data processing device based on a sensor-storage-computing integrated architecture, characterized in that: The device comprises: A first acquisition module is used to acquire input data (X+K-1)*K, where the input data (X+K-1)*K is at least one of image data, video data and voice data in an analog domain; The second acquisition module is used to obtain the weight value of the convolution kernel, the size of the convolution kernel is K, and the step length is S; A processing module, wherein the second acquisition module is electrically connected to the processing module, and the processing module includes a plurality of rows of multipliers and adders and int((X-1) / S+1) capacitors; A data flow control unit, wherein the data flow control unit constructs N repetition units according to the size and step size of the convolution kernel, and the repetition unit is S rows; the convolution kernel is sequentially split into N convolution groups by row, when K / S is an integer, N=K / S, and each convolution group is S rows; when K / S has a remainder, N=int(K / S)+1, except that the last convolution group is less than S rows, the remaining convolution groups are all S rows; when the convolution group is S rows, the repetition unit is a convolution group; when the convolution group is less than S rows, the repetition unit includes a convolution group and a blank row; the data flow control unit is used to control the convolution groups stored in the second acquisition module to cyclically transmit weight values ​​to the processing module in sequence, thereby forming N reorganized weights; The processing module is used to sequentially multiply and add the input data (X+K-1)*K with N reorganization weights, and to add the results of multiple multiplications and additions to obtain int((X-1) / S+1) convolution results in the column direction; The corresponding multiplication and addition results of the input data and the first reorganization weight in the multi-row multiplier-adder are accumulated and stored in int((X-1) / S+1) capacitors according to the output storage group, and then the first intermediate convolution result is obtained; the corresponding multiplication and addition results of the input data and the second reorganization weight in the multi-row multiplier-adder are continued to be accumulated and stored in int((X-1) / S+1) capacitors according to the output storage group, and then the second intermediate convolution result is obtained; and so on, the corresponding multiplication and addition results of the input data and the Nth reorganization weight in the multi-row multiplier-adder are continued to be accumulated and stored in int((X-1) / S+1) capacitors according to the output storage group, and then int((X-1) / S+1) convolution results in the column direction are obtained; In N multiplication-addition operations, at least some of the multipliers and adders in the plurality of rows are reused.

5. The CNN data processing device according to claim 4, characterized in that Each multiplier-adder includes K PEs, and the number of rows of the multiple groups of multipliers-adders is less than or equal to (int((X-1) / S+1))×S.

6. The CNN data processing device according to claim 5, characterized in that The number of rows of the nth recombination weight is (int((X-1) / S)+n)×S, and the nth recombination weight has (n-1)×S rows of blank rows, after which the nth repeating unit is used to construct the nth recombination weight, where 1≤n≤N; The processing module further includes a first switch array, which is used to accumulate the result of multiplying and adding the input data and the nth reorganization weight in the multiplier to the corresponding capacitor.

7. The CNN data processing device according to claim 5, characterized in that The device further comprises a second switch array, the second switch array being used to transmit the input data stored in the first acquisition module to the multiple rows of multipliers and adders to multiply and add the nth reorganization weights, where 1≤n≤N; The multiple rows of multipliers and adders are fixedly connected to corresponding capacitors; The nth recombination weight is specifically constructed as a matrix with int((X-1) / S+1)×S rows constructed with the nth repeating unit.

8. The CNN data processing device according to claim 6 or 7, characterized in that: The multiple rows of multipliers and adders are divided into multiple groups in sequence according to S rows, and the multipliers and adders in the same group are connected to the same capacitor.

9. The CNN data processing device according to claim 8, characterized in that The CNN data processing device is used to process the convolution calculation of the total input data (X+K-1)*M, where M is an integer greater than or equal to K; The first acquisition module is constructed as an analog register, which sequentially acquires multiple input data (X+K-1)*K from the total input data (X+K-1)*M for parallel convolution calculation; wherein, after processing one parallel convolution calculation, the analog register moves S columns to the right from the total input data (X+K-1)*M to acquire the next group (X+K-1)*K; and so on, until a convolution result P*R is obtained, P=int((X-1) / S+1), R=int((MK) / S+1); The device also includes a pooling unit, and the convolution result P*R is processed by the pooling unit to obtain a feature map.

10. A sensing, storage and computing integrated chip, characterized in that: The chip includes a CNN data processing device as described in any one of claims 4 to 9.

Images