Data processing method, device and chip based on sensing, storage and computing integrated architecture

Abstract

The present invention relates to the technical field of convolutional neural networks, and discloses a data processing method, device and chip based on a sensing-storage-computing integrated architecture, the method is used to process the convolution calculation of the input matrix [A1, A2, ..., AM] of M×N input data, the size of the convolution kernel W is k, the step length is s, and the method includes the following loop steps: constructing an input submatrix Yi, forming a reorganized convolution kernel Wi, Yi and Wi convolution calculation, and judging whether the loop is completed. The method, device and chip of the present invention can effectively reduce the number of analog registers required for convolution operations in a sensing-storage-computing integrated chip by cyclically controlling the input and weights, greatly reducing the area, cost and power consumption of the sensing-storage-computing chip, and improving the calculation speed.

Description

Technical Field

[0001] The present invention relates to a convolutional neural network (CNN), and in particular to a data processing method, device and chip based on a sensing-storage-computing integrated architecture. Background Art

[0002] IoT devices need to be able to perceive information from the physical world, and various types of sensors play an important role in this, including touch, smell, taste, hearing, and vision. Humans obtain more than 80% of information through vision. Therefore, image sensors that collect visual information have naturally become one of the hot research directions in the field of IoT.

[0003] At present, most camera modules use CMOS image sensors (CIS) to realize the perception of optical signals and provide raw image data for subsequent processing. With the continuous development of artificial intelligence, the field of intelligent visual perception has become an important direction in the field of sensors. In traditional intelligent visual perception nodes, the perception nodes and computing nodes are separated at the hardware level due to their different functional requirements and manufacturing technologies. Most visual perception nodes are located in the analog signal domain, while computing nodes usually use the traditional von Neumann computing architecture to execute in the digital signal domain. The electrical signal converted by the CMOS image sensor is converted into a digital image by an analog-to-digital converter (ADC) and then output to an external digital system, using various neural network (NN) algorithm models to train and classify image data. However, this method is affected by high power consumption, high latency, and limited throughput, making it difficult to use in edge and end scenarios with high speed, high real-time performance, and limited resources.

[0004] On the road to the intelligent era, the core basic technology that needs to be solved first is low-power, low-latency, low-cost, and miniaturized intelligent sensing chips. To address this problem, the integrated sensing, storage, and computing chip adds a computing unit inside the traditional sensor to realize data calculation and processing within the sensor. The integrated sensing, storage, and computing chip integrates data perception, storage, and computing at the sensor end, and directly processes the analog signals collected by the sensor, greatly reducing the delay and power consumption caused by data transmission and analog-to-digital conversion, and will have great application value in the future.

[0005] The sensor-storage-computing integrated chip uses an analog-digital convolution circuit array, where the analog input comes from the analog register and the digital weight comes from the on-chip digital memory. In the prior art, the digital convolution kernel of the sensor-storage-computing integrated chip is fixedly connected to the convolution operator. During the calculation process, the sliding position of the convolution kernel is often controlled by switches to address the analog voltage data in the analog register stack. When the image becomes larger and the resolution becomes higher, this method uses a large number of switches, has a large switch area overhead, and has a large number of control signals, and the control logic is complex. Summary of the invention

[0006] In order to solve the above problems, the present invention provides a data processing method, a data processing device and a sensing-storage-computing integrated chip based on a sensing-storage-computing integrated architecture.

[0007] The first aspect of the present disclosure provides a data processing method based on a sensing, storage and computing integrated architecture, which is used to process the convolution calculation of the input matrix [A1, A2, ..., AM] of M×N input data, and the size of the convolution kernel W is k and the step size is s. The data processing method includes the following loop steps: constructing an input sub-matrix Yi, forming a reorganized convolution kernel Wi, Yi and Wi convolution calculation, and judging whether the loop is completed. In the step of constructing the input sub-matrix Yi, part of the input data of the input matrix is ​​obtained to form a k-row input sub-matrix Yi, i is 1, 2, ...; when i=1, the input data A1, A2, ..., Ak are obtained to form an input sub-matrix Y1; when i≥2, the input data Ax+1, ..., Ax+s of S rows are obtained in sequence, x=k+(i-2)×s, and Ax+1-k, ..., Ax+sk in the input sub-matrix Yi-1 are sequentially covered to obtain the input sub-matrix Yi. In forming the reorganized convolution kernel Wi, the weights of the convolution kernel are rearranged through the switch array to form a reorganized convolution kernel Wi that matches the input sub-matrix Yi. In the convolution calculation step, the reorganized convolution kernel Wi is input into the convolution circuit array, and the reorganized convolution kernel Wi slides in the row direction of the input sub-matrix Yi to perform convolution calculation with the input sub-matrix Yi until the convolution calculation of the input sub-matrix Yi is completed. In the step of judging whether the loop is completed, after the convolution calculation of the input sub-matrix Yi and the reorganized convolution kernel Wi is completed, it is judged whether k+(i-1)×s is less than M. When it is less than M, the value of i is incremented by 1; when it is greater than or equal to M, the calculation is terminated.

[0008] Specifically, the weights of the convolution kernel are stored in a digital memory, which includes k rows, each row corresponds to k×Q units, and Q represents the weight accuracy of the convolution kernel.

[0009] Specifically, in some embodiments, the switch array includes k×Q switch groups, each switch group includes P groups of switches distributed in columns, and each group of switches includes switches c1, ..., ck. One end of each switch in each group of switches is correspondingly connected to a row of the same column unit of the digital memory, and the other end is correspondingly connected to a row of the same column unit of the convolution circuit array. By controlling all switches in any identical group of switches in the k×Q switch groups to be turned on, a reorganized convolution kernel Wi matching the input submatrix Yi can be formed. Wherein, P is the quotient of the least common multiple of k and s divided by s.

[0010] In some other embodiments, the switch array is arranged in such a manner that the switch array includes k×Q switch groups, each switch group includes k groups of switches distributed in columns, and each group of switches includes switches c1, ..., ck. One side of each group of switches is sequentially connected to the rows of the same column unit of the digital memory, the other side of the first group of switches is connected to the first row of the same column unit of the convolution circuit array, the other side of the second group of switches is connected to the second row of the same column unit of the convolution circuit array, and the other side of the kth group of switches is connected to the kth row of the same column unit of the convolution circuit array. By controlling one of the switches c1, ..., ck in the k×Q switch groups to be turned on, a reorganized convolution kernel Wi matching the input submatrix Yi can be formed.

[0011] More specifically, from top to bottom, the arrangement order of the first group of switches of each switch group is c1, ..., ck, the arrangement order of the second group of switches is ck, c1, ..., ck-1, the arrangement order of the third group of switches is ck-1, ck, c1, ..., ck-2, and so on. The initial control vector C0 of the switch array is [1, 0, 0, ..., 0], and the control vector Ci corresponding to Yi is the initial control vector C0 cyclically shifted left by (i-1) × s bits.

[0012] Preferably, when k+(i-1)×s is greater than or equal to M, the calculation is terminated, the convolution result matrix is ​​output, and the convolution calculation of the next M×N input data is continued.

[0013] The second aspect of the present disclosure provides a data processing device based on a sensing, storage and computing integrated architecture, which is used to process the convolution calculation of an input matrix [A1, A2, ..., AM] of M×N input data, and the size of the convolution kernel W is k and the step size is s. The data processing device includes an acquisition module, a storage module, a loop control unit and a convolution circuit array. The acquisition module is used to acquire k rows of input data, and form an input sub-matrix Yi in the acquisition module, where i is 1, 2, ..., int ((Mk) / s+1). When i=1, the input sub-matrix Yi is [A1, A2, ..., Ak]; when i≥2, the input data Ax+1, ..., Ax+s of S rows are acquired in sequence, x=k+(i-2)×s, and Ax+1-k, ..., Ax+sk in the input sub-matrix Yi-1 are sequentially covered to obtain the input sub-matrix Yi. The storage module is used to store the convolution kernel. The switch array is coupled to the storage module, and is used to reorder the convolution kernels by row to generate a reorganized convolution kernel Wi externally. The loop control unit is used to control the acquisition module to generate the input sub-matrix Yi, and control the switch array to generate the reorganized convolution kernel Wi. The reorganized convolution kernel Wi determines the row vector order according to the row number order of the input sub-matrix Yi. The convolution circuit array is used to slide the reorganized convolution kernel Wi in the row direction of the input sub-matrix Yi to perform convolution calculation with the input sub-matrix Yi until the convolution calculation of the input sub-matrix Yi is completed. After the convolution calculation of the input sub-matrix Yi is completed, the value of i increases by 1.

[0014] Specifically, the storage module is a digital memory, which includes k rows, each row corresponds to k×Q units, and Q represents the weight accuracy of the convolution kernel.

[0015] Specifically, the switch array includes k×Q switch groups, each switch group includes k groups of switches distributed in columns, and each group of switches includes switches c1, ..., ck. In some embodiments, one end of each switch in each group of switches is connected to the same row of the same column unit of the digital memory and the other end is respectively connected to the rows of the same column unit of the convolution circuit array. In other embodiments, one end of each switch in each group of switches is respectively connected to the rows of the same column unit of the digital memory and the other end is connected to the same row of the same column unit of the convolution circuit array. The cyclic control unit controls one of the switches c1, ..., ck in the k×Q switch groups to be turned on so that the switch array generates a reorganized convolution kernel Wi that matches the input submatrix Yi.

[0016] Preferably, one end of each switch in each group of switches is respectively connected to each row of the same column unit of the digital memory and the other end is connected to the same row of the same column unit of the convolution circuit array. The switch array is arranged from top to bottom, the arrangement order of the first group of switches is c1, ..., ck, the arrangement order of the second group of switches is ck, c1, ..., ck-1, the arrangement order of the third group of switches is ck-1, ck, c1, ..., ck-2, and so on. The initial control signal C0 of the switch array is [1, 0, 0, ..., 0], and the control signal Ci corresponding to Yi is the initial control signal C0 cyclically shifted to the left by (i-1) × s bits. Among them, there is 1 1 and k-1 0s in the control vector, 0 indicates that the switch is disconnected, 1 indicates that the switch is turned on, and the control vector C0 indicates that all switches with a certain number in the switch array are turned on, and the number is the same as the position number of 1 in the control vector.

[0017] The third aspect of the present disclosure provides a sensing, storage and computing integrated chip, including a sensing circuit and a data processing device according to the second aspect of the present disclosure. The sensing circuit captures image information from the outside to generate input data, the acquisition module is an analog register, and the cyclic control unit controls the input data of the sensing circuit to be stored in the analog register in batches, and the input data stored in the analog register is an input sub-matrix Yi. Preferably, the sensing circuit is a rolling shutter image sensing circuit.

[0018] Features and advantages of the present disclosure include:

[0019] The data processing method and data processing device based on the integrated sensing, storage and computing architecture provided by the present invention can effectively reduce the number of analog registers required for convolution operations in the integrated sensing, storage and computing chip through cyclic control of inputs and weights, thereby greatly reducing the area, cost and power consumption of the integrated sensing, storage and computing chip and improving the computing speed.

[0020] The data processing device disclosed in the present invention is suitable for high-speed cyclic data loading of any non-fully parallel sliding convolution operator in the sensing, storage and computing scenario, which can reduce the area overhead of analog registers and has the advantages of high data reuse rate and high resource utilization efficiency. The data processing device does not affect the convolution calculation rate, and can realize high-speed convolution operator calculation, which increases the performance of the sensing, storage and computing chip for neural computing.

[0021] The sensing, storage and computing integrated chip disclosed in the present invention adopts a rolling shutter image sensor circuit and an analog register to simultaneously perform convolution operations on the analog voltage or current signals of the first few rows that have been exposed and stored in the analog register while the latter few rows are exposed. After the calculation is completed, part of the data stored in the analog register is expired and useless data, and the image information of the subsequent exposure rows is stored in the location of these useless data in the analog register, thereby realizing the reuse of register resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A schematic diagram showing the hardware architecture of the sensing, storage and computing integrated chip disclosed in the present invention;

[0024] Figure 2 An input matrix of M×N input data is shown;

[0025] Figure 3 A 5×5 convolution kernel is shown;

[0026] Figure 4 Shows sliding in the input matrix in the three column directions;

[0027] Figure 5 The schematic diagram of the input submatrix and the reorganized convolution kernel in three column-wise slides is shown, where the size k of the convolution kernel is 5 and the step size s is 2;

[0028] Figure 6 The figure shows the schematic diagram of the input submatrix and the row number sorting in six column-wise slides, where the convolution kernel size k is 5 and the step size s is 2;

[0029] Figure 7 The figure shows the input submatrix and row number sorting in five column-wise slides, where the convolution kernel size k is 8 and the step size s is 2;

[0030] Figure 8 A schematic diagram showing a switch array arrangement of the present disclosure is shown;

[0031] Fig. 9 A schematic diagram of the input submatrix, row number sorting, and switch array output is shown, where the convolution kernel size k is 5 and the step size s is 2;

[0032] Fig.10 A schematic diagram showing the overall operation flow of the loop control logic and array. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0034] In existing integrated sensing, storage and computing chips, sliding convolution operations occupy a large number of analog register resources, resulting in a large chip area and high power consumption. Figure 1 The present invention discloses a sensing-storage-computing integrated chip, which includes a sensing circuit and a data processing device based on a sensing-storage-computing integrated architecture. The sensing circuit is used to convert the light signal of the external captured image into an analog voltage or current signal. The data processing device based on the sensing-storage-computing integrated architecture is used to perform convolution calculations on these analog electrical signals (image data) and finally output a feature map. The analog electrical signal from the sensing circuit is the input data of the data processing device. Alternatively, in some embodiments, the sensing circuit can also be used to capture voice signals, etc., and the data processing device of the sensing-storage-computing integrated architecture can also be used to process video data, voice data, etc. In some embodiments, the input data of the data processing device can also be a digital signal.

[0035] The data processing device based on the sensor-storage-computation integrated architecture provided by the present invention comprises an analog register, a digital memory, a convolution circuit array, a loop control unit and a switch array. The analog register is used to store the image analog voltage or current information loaded by the previous stage sensor circuit, and the image information stored in the analog register is used as the input matrix to be convolution processed, and the input matrix is ​​loaded into the convolution circuit array for convolution operation. The size of the convolution kernel is k, and the step length is s.

[0036] Specifically, the number of rows of the analog register is determined by the size k of the convolution kernel W, and the analog register stores k rows of input data from the sensor circuit. After the k rows of input data complete the convolution operation, the loop control unit controls the analog register to load the input data from the sensor circuit once, forming a new group of k rows of input data. Specifically, the analog register retains ks rows of reusable input data, and loads new s rows of input data into the analog register to overwrite the expired s rows of input data in the analog register, thereby forming a new group of k rows of input data.

[0037] The convolution kernel is stored in a digital memory, and each row of the digital memory is connected to each row of the convolution circuit array through a switch array. The loop control unit controls the switch array so that the arrangement order of the weight values ​​of the convolution kernel loaded into the convolution circuit array is consistent with the arrangement order of the k rows of input data to be processed.

[0038] Specifically, in some embodiments, the sensing circuit uses a rolling shutter image sensing circuit, and uses a successive exposure method to store input data in an analog register. The digital memory can use a variety of storage media, such as SRAM, RRAM, MRAM, Flash, DRAM, etc. The convolution kernel stored in the digital memory is a digital convolution kernel, and the weight information of the digital convolution kernel comes from the convolution kernel weight trained by the software end.

[0039] The convolution circuit array is used to implement the convolution operation of the analog voltage or current input and the digital convolution kernel. Specifically, the convolution circuit array includes a number of row_PEs, for example, k row_PEs, i.e., k rows of row_PEs. Among them, the length of each row_PE is k, that is, each row contains k PEs. In a loading process of the analog register, the convolution circuit array is used to implement the sliding calculation of the convolution kernel on the k rows of input data. Sliding on the row can be implemented in parallel or serial mode, and the result of the convolution kernel sliding on the row is obtained and output.

[0040] The switch array is used to control the convolution kernel weights in the digital memory to be cyclically input into the convolution circuit array for operation, and the circulation and cyclic loading of the convolution kernel weights correspond to the input data of the analog register. The advantage of the convolution kernel weight cyclic input convolution array disclosed in the present invention is that while maximizing the reuse of input data and analog registers, the control and circuit complexity are reduced to achieve efficient analog domain convolution calculation.

[0041] The data processing device disclosed in the present invention is suitable for high-speed cyclic data loading of sliding convolution operators in the sensing, storage and computing scenarios, which can reduce the area overhead of analog registers and has the advantages of high data reuse rate and high resource utilization efficiency. The data processing device does not affect the convolution calculation rate, and can realize high-speed convolution operator calculation, which increases the performance of the sensing, storage and computing chip for neural computing.

[0042] Reference below Figures 2 to 10 , and introduces in detail the data processing method and data processing device based on the integrated sensing, storage and computing architecture disclosed in the present invention. Figure 2 An input matrix [A1, A2, ..., AM] consisting of M×N input data is shown, where A1, A2...AM represent row vectors of the input matrix, M represents the number of rows of the input matrix, and N represents the number of columns of the input matrix. Figure 3 A 5×5 convolution kernel [w1, w2, w3, w4, w5] is shown, the size of the convolution kernel is 5, and the step size s is 2, where w1, w2…w5 represent the row vectors of the convolution kernel. Figure 4It shows that in three column-wise sliding, the first calculation is for the first 5 rows of the input matrix (A1, A2, A3, A4, A5), the second calculation is for the 3rd to 7th rows of the input matrix (A3, A4, A5, A6, A7), and the third calculation is for the 5th to 9th rows of the input matrix (A5, A6, A7, A8, A9). After the first calculation is completed, A1 and A2 in the first 5 rows of the input matrix are no longer used in subsequent calculations and are expired input data; A3, A4, and A5 continue to be used in the second calculation and are reusable input data. After the second calculation is completed, A3 and A4 are no longer used in subsequent calculations and are expired input data; A5, A6, and A7 can continue to be used in the third calculation and are reusable input data.

[0043] Figure 5 The data processing method disclosed in the present invention is used to show the input sub-matrix Yi and the reorganized convolution kernel Wi in three column-wise slides, where i=1, 2, and 3. The input sub-matrix Yi refers to the k rows of input data stored in the analog register, where 1≤i≤int((Mk) / s+1). The reorganized convolution kernel Wi refers to a convolution kernel in which the weight values ​​stored in the digital memory are rearranged in rows through the switch array. The row numbering of the weight values ​​in the reorganized convolution kernel Wi is consistent with the row numbering of the input data in the input sub-matrix Yi, so that Yi and Wi can directly perform convolution calculations without changing the calculation results.

[0044] See also Figure 5 , Figure 6 In the first slide, the first five rows of the input matrix A1, A2, A3, A4, A5 are loaded into the 1st to 5th rows of the analog register in sequence to form the input submatrix Y1. The row numbers of the input submatrix Y1 are sorted as 1, 2, 3, 4, 5, and the reorganized convolution kernel W1 has the same row numbering as the input submatrix Y1. The reorganized convolution kernel W1 is [w1, w2, w3, w4, w5]. Y1 and W1 are transmitted to the convolution circuit array, and the convolution calculation of the input submatrix Y1 is completed by sliding in the row direction.

[0045] In the second slide, the reusable input data A3, A4, and A5 stored in the analog register are retained, and the 6th and 7th rows of the input matrix are sequentially loaded into the 1st and 2nd rows of the analog register, so that A6 and A7 are stored in the 1st and 2nd rows of the analog register, respectively, thereby covering A1 and A2 to form the input submatrix Y2. The row numbers of the input submatrix Y2 are sorted as 4, 5, 1, 2, and 3, and the reorganized convolution kernel W2 has the same row numbering as the input submatrix Y2, and the reorganized convolution kernel W2 is [w4, w5, w1, w2, w3]. Y2 and W2 are transmitted to the convolution circuit array, and the convolution calculation of the input submatrix Y2 is completed by sliding in the row direction.

[0046] In the third slide, the reusable input data A5, A6, and A7 stored in the analog register are retained, and the 8th and 9th rows of the input matrix are sequentially loaded into the 3rd and 4th rows of the analog register, so that A8 and A9 are stored in the 3rd and 4th rows of the analog register, respectively, thereby covering A3 and A4 to form the input submatrix Y3. The row numbers of the input submatrix Y3 are sorted as 2, 3, 4, 5, and 1. The reorganized convolution kernel W2 has the same row numbering as the input submatrix Y2, and the reorganized convolution kernel W3 is [w2, w3, w4, w5, w1]. Y3 and W3 are transmitted to the convolution circuit array, and the convolution calculation of the input submatrix Y3 is completed by sliding in the row direction.

[0047] Each sliding of the convolution kernel in the column direction corresponds to a loading of the analog register. In the first loading, 5 rows of input data are added to the analog register; except for the first loading, in subsequent loadings, s rows of input data are added to the analog register each time, and the newly added s rows of input data sequentially overwrite the s rows of expired input data in the analog register, maximizing the reuse of input data and analog registers.

[0048] See also Figure 6 , taking k as 5 and s as 2 as an example, the row numbers of the 6th slide are sorted as 1, 2, 3, 4, 5, which is the same as the row numbers of the 1st slide. It can be seen that the cycle of the simulated register loading is 5. Figure 7 , taking k as 8 and s as 2 as an example, the row numbering sequence of the 5th slide is 1, 2, 3, 4, 5, 6, 7, 8, which is the same as the row numbering sequence of the 1st slide, and the cycle period of simulated register loading is 4. In summary, it can be seen that the cycle period of simulated register loading is related to k and s. Specifically, the cycle period is the quotient of the least common multiple of k and s divided by s.

[0049] Specifically, the digital memory includes k rows, each row includes k×Q units, that is, the digital memory includes a total of k×Q×k units. In terms of columns, the digital memory includes k×Q column units. Among them, Q represents the weight accuracy of the convolution kernel, and each weight value requires Q digital memory units to store. When Q is 1, the weight accuracy is 1 bit, the weight value can be 0 or 1, and each weight value requires 1 digital memory unit to store; when Q is 2, the weight accuracy is 2 bits, the weight value can be 00, 01, 10 or 11, and each weight value requires 2 digital memory units to store. The convolution circuit array includes k rows of row_PE, each row of row_PE includes k PEs, and each PE includes Q weight value input ports. Therefore, each row of the convolution circuit array includes k×Q weight value input ports, and each unit of the digital memory is connected to the weight value input port of the corresponding column of the convolution circuit array through a switch.

[0050] In some embodiments, the switch array includes k×Q switch groups, each switch group includes P groups of switches distributed in columns. The digital memory includes k×Q column units, each column unit of the digital memory is connected to the corresponding column unit of the convolution circuit array through a switch group (i.e., P groups of switches), for example, the first column unit W11, W21...Wk1 of the digital memory is connected to the first column unit of the convolution circuit array through P groups of switches. Each group of switches in a switch group includes k switches, namely switches c1, c2,..., ck. Each switch includes one end connected to the digital memory and the other end connected to the convolution circuit array. One side of each group of switches includes one end of k switches, and the other side of each group of switches includes the other end of k switches. One side of each set of switches is respectively connected to each row of the same column unit of the digital memory (i.e., one end of each switch in each set of switches is correspondingly connected to a row of the same column unit of the digital memory), and the other side of each set of switches is respectively connected to each row of the same column unit of the convolution circuit array (i.e., the other end of each switch in each set of switches is correspondingly connected to a row of the same column unit of the convolution circuit array), so that when any identical set of switches in the switch groups connected to all column units are all closed, the row numbering sequence of the reorganized convolution kernel Wi corresponds to one of the row numbering sequences of the input data cyclically loaded into the analog register. Wherein, P is the quotient of the least common multiple of k and s divided by s. Preferably, one side of each set of switches is respectively and sequentially connected to each row of the same column unit of the digital memory.

[0051] by Figure 5Taking the sliding shown in FIG. 1 as an example, when the size k of the convolution kernel is 5, the step size s is 2, and Q is 1, the switch array is introduced in detail. The switch array includes 5 switch groups, each switch group includes 5 groups of switches, and each switch group has 5 switches. One switch group is used to connect one column unit of the digital memory to one column unit of the convolution circuit array. Since the digital memory and the convolution circuit array have 5 column units, 5 switch groups are required. The switch group connected to the first column unit is introduced below. In the switch group, one side of each group of switches is sequentially connected to a row of the same column unit of the digital memory. For example, in the first group of switches, one end of switches c1, c2, c3, c4, and c5 are respectively connected to the 1st to 5th rows of the first column unit of the digital memory. In the first group of switches, the other ends of switches c1, c2, c3, c4, and c5 are respectively connected to the 1st to 5th rows of the first column units of the convolution circuit array; in the second group of switches, the other ends of switches c1, c2, c3, c4, and c5 are respectively connected to the 4th, 5th, 1st, 2nd, and 3rd rows of the first column units of the convolution circuit array; in the third group of switches, the other ends of switches c1, c2, c3, c4, and c5 are respectively connected to the 2nd, 3rd, 4th, 5th, and 1st rows of the first column units of the convolution circuit array; in the fourth group of switches, the other ends of switches c1, c2, c3, c4, and c5 are respectively connected to the 5th, 1st, 2nd, 3rd, and 4th rows of the first column units of the convolution circuit array; in the fifth group of switches, the other ends of switches c1, c2, c3, c4, and c5 are respectively connected to the 3rd, 4th, 5th, 1st, and 2nd rows of the first column units of the convolution circuit array. In one cycle, the control signals of the switch array are [1, 0, 0, 0, 0], [0, 1, 0, 0, 0], [0, 0, 1, 0, 0], [0, 0, 0, 1, 0], [0, 0, 0, 0, 1], [0, 0, 0, 0, 1], where [1, 0, 0, 0, 0] means that all switches of the first group of the five switch groups are turned on.

[0052] In other embodiments, see Figure 8, the switch array includes k×Q switch groups, each switch group includes k groups of switches distributed in columns. Each column unit of the digital memory is connected to the corresponding column unit of the convolution circuit array through a switch group (i.e., k groups of switches), for example, the first column unit of the digital memory is connected to the first column unit of the convolution circuit array through k groups of switches. Wherein, each group of switches includes k switches, namely switches c1, c2, ..., ck. One side of each group of switches is respectively connected to each row of the same column unit of the digital memory (one end of each switch of each group of switches is connected to a row of the digital memory), and the other side of each group of switches is connected to the same row of the same column unit of the convolution circuit array (the other end of each switch of each group of switches is connected to the same row of the same column unit of the convolution circuit array). In particular, the other end of each switch in each group of switches is connected together as an output end connected to the weight value input port of the convolution circuit array. Alternatively, one side of each group of switches is connected to the same row of the same column unit of the digital memory, and the other side of each group of switches is respectively connected to each row of the same column unit of the convolution circuit array.

[0053] The following description will be made by taking an example where the other side of each set of switches is connected to the same row of units in the same column of the convolution circuit array. Figure 8 An example is given in which one column unit of a digital memory is connected to one column unit of a convolution circuit array through one switch group. Figure 8 , more specifically, the other side of the first group of switches of the switch group is connected to the first row of the same column unit of the convolution circuit array, the other side of the second group of switches is connected to the second row of the same column unit of the convolution circuit array, and the other side of the kth group of switches is connected to the kth row of the same column unit of the convolution circuit array. By controlling one of the switches c1, ..., ck in the k×Q switch groups (i.e., switches with the same number) to be turned on, the row numbering sequence of the reorganized convolution kernel Wi corresponds to one of the row numbering sequences of the input data cyclically loaded into the analog register, and a reorganized convolution kernel Wi matching the input submatrix Yi can be formed.

[0054] Preferably, the switches in each group of switches of each switch group are arranged in a specific order. Specifically, from top to bottom, the arrangement order of the first group of switches is c1, ..., ck, the arrangement order of the second group of switches is ck, c1, ..., ck-1, the arrangement order of the third group of switches is ck-1, ck, c1, ..., ck-2, and so on. The initial control vector C0 of the switch array is [1, 0, 0, ..., 0], and the control vector Ci corresponding to Yi is the initial control vector C0 cyclically shifted to the left by (i-1) × s bits, that is, the control vector Ci+1 is cyclically shifted to the left by s bits based on the control vector Ci. Among them, there is 1 1 and k-1 0s in the control vector, 0 means that the switch is disconnected, 1 means that the switch is turned on, and the control vector C0 indicates that all switches with a certain number in the switch array are turned on, and the number is the same as the position number of 1 in the control vector. At this time, each row of the digital memory corresponds to a row connected to the convolution circuit array. For example, 1 in the initial control vector C0 is at the first position, indicating that all c1 switches in the switch array are turned on and the remaining switches are turned off.

[0055] See below Fig. 9 , continue to take the convolution kernel size k as 5 and the step size s as 2 as an example for detailed description. In the first slide, the input submatrix Y1 is [A1, A2, A3, A4, A5], and the row numbers are arranged as 1, 2, 3, 4, 5. The control vector C1 of the switch array is cyclically shifted left by 0 bits based on C0, that is, the control vector is [1, 0, 0, 0, 0]. The c1 switches in the control switch array are all turned on, so that the weight rows w1, w2, w3, w4, w5 stored in the digital memory are respectively transmitted to D1, D2, D3, D4, D5 of the convolution circuit array. Among them, D1 represents the first row of weight value input terminals of the convolution circuit array, and each row of weight input terminals includes k×Q weight value input ports. In the second slide, the input submatrix Y2 is [A6, A7, A3, A4, A5], the row numbers are 4, 5, 1, 2, 3, the control vector C2 of the switch array is cyclically shifted left by s positions based on C1 (that is, cyclically shifted left by s positions in the initial control vector C0), that is, the control vector is [0, 0, 0, 1, 0], and the c4 switches in the control switch array are all turned on, so that the weight rows w1, w2, w3, w4, w5 stored in the digital memory are respectively transmitted to D3, D4, D5, D1, D2 of the convolution circuit array. After 5 slides, a cycle is completed.

[0056] See also Fig.10, the following is a detailed introduction to the overall operation flow of the loop control logic and array in the data processing method and data processing device of the present disclosure. The loop control unit obtains initial information, which includes i=1, the image size is M×N, the size of the convolution kernel is k, the step size is s, and the initial control vector C0 of the switch array is [1,0,0,...,0] (there are k-1 0s in the initial control vector C0). M and N are specifically determined by the sensor circuit. The sensor circuit captures image information from the outside world and generates M×N analog signal input data, such as M×N voltage data. Expose the 1st to kth rows of pixels (voltage data) one by one, that is, expose the voltage data of the first k rows of the image, and store the k rows of voltage data in sequence to the 1st to kth rows of the analog register of size k×N. Then enter the loop part of 1 frame image processing, that is, the loop part of M×N input data processing.

[0057] The loop part specifically includes first calculating the convolution results of the [(i-1)×s+1]th to [(i-1)×s+k]th rows, and then judging whether i×s+k-1 is less than M. The convolution results of the [(i-1)×s+1]th to [(i-1)×s+k]th rows are calculated by inputting the k-row input data Yi stored in the analog register into the convolution circuit array, and at the same time, the convolution kernel weights stored in the digital memory are transmitted to the convolution circuit array after being configured through the switch array, and the convolution kernel is slid on the row in the convolution circuit array. The switch array is configured by the control vector C, C1 is the same as C0, and C1 is [1,0,0,...,0].

[0058] If i×s+k-1 is greater than or equal to M, that is, i×s+k-1 is less than M, it means that the convolution calculation of M×N input data has been processed, and the feature map, that is, the convolution result matrix, is output. The next frame of image processing begins, and the sensor circuit is returned to capture image information from the outside world. If i×s+k-1 is less than M, it means that the input data in this frame has not been processed. i=i+1, that is, i is increased by 1 to form a new input subarray Yi, obtain a new control vector Ci, and then calculate the new convolution results of the [(i-1)×s+1] to [(i-1)×s+k] rows.

[0059] Specifically, a new input subarray Yi is formed to expose s rows, read the data from the image rows (i-2)×s+1+k to (i-1)×s+k, and store the s rows of voltage data in the analog registers at the locations of the analog registers (i-2)×s+1 to (i-1)×s. A new control vector Ci is formed by cyclically shifting the previous control vector to the left by s bits. For example, when forming a new input subarray Y2, i=2, read the data from the image rows 1+k to s+k, and store the s rows of voltage data in the analog registers at the locations of the analog registers 1 to s. Circularly shift the control vector C1[1,0,0,...,0] to the left by s bits, for example, when s is 2, the control vector C2 is [0,0,...,1,0].

[0060] The row numbering order of the analog register is shifted with each sliding cycle on the column, and the connection relationship between the analog register and the convolution array is fixed. At the same time, the row weights of the digital convolution kernel are output to the convolution circuit array through the switch array, and the control vector C is changed each time to realize the cyclic shift.

[0061] The sensing, storage and computing integrated chip disclosed in the present invention adopts a rolling shutter image sensing circuit and an analog register. While the next few lines are exposed, the analog voltage or current signals of the previous few lines that have been exposed and stored in the analog register are simultaneously convolved. After the calculation is completed, part of the data stored in the analog register is expired and useless data, and the image information of the subsequent exposure lines is stored in the location of these useless data in the analog register, thereby realizing the reuse of register resources. The sensing, storage and computing integrated chip, data processing method and data processing device disclosed in the present invention combine the data reuse characteristics of the image convolution operation itself, and adopt a cyclic sensing data loading logic for control.

[0062] The disclosed data processing method and data processing device based on the integrated sensing, storage and computing architecture can effectively reduce the number of analog registers required for convolution operations in the integrated sensing, storage and computing chip by cyclically controlling the input and weights, greatly reducing the area, cost and power consumption of the integrated sensing, storage and computing chip, and improving the computing speed. This method can be further extended to the input logic of other operator circuits, which can promote the further development and application of the integrated sensing, storage and computing chip.

[0063] The cyclic data loading and weight control method of the data processing method disclosed in the present invention is applicable to sliding convolution operations with different convolution kernel sizes, different step lengths, and different input sizes, and is universal.

[0064] The above are only several embodiments of the present disclosure. Those skilled in the art may make various changes or modifications to the embodiments of the present disclosure based on the contents disclosed in the application documents without departing from the spirit and scope of the present disclosure.

Claims

1. A data processing method based on a sensor-storage-computation integrated architecture is used to process the convolution calculation of an input matrix [A1, A2, ..., AM] of M×N input data, where the size of the convolution kernel W is k and the step length is s, and is characterized in that: The cycle includes the following steps: Construct input sub-matrix Yi: obtain part of the input data of the input matrix to form a k-row input sub-matrix Yi, where i is 1, 2, ...; When i=1, the input data A1, A2, ..., Ak are obtained to form the input sub-matrix Y1. When i≥2, sequentially obtain the input data Ax+1, ..., Ax+s of S rows, x=k+(i-2)×s, and sequentially overwrite Ax+1-k, ..., Ax+sk in the input submatrix Yi-1 to obtain the input submatrix Yi; Forming a reorganized convolution kernel Wi: Rearrange the weights of the convolution kernel through the switch array to form a reorganized convolution kernel Wi that matches the input submatrix Yi; Convolution calculation: The reorganized convolution kernel Wi is input into the convolution circuit array, and the reorganized convolution kernel Wi slides in the row direction of the input sub-matrix Yi to perform convolution calculation with the input sub-matrix Yi until the convolution calculation of the input sub-matrix Yi is completed; After completing the convolution calculation of the input submatrix Yi and the reorganized convolution kernel Wi, determine whether k+(i-1)×s is less than M. When it is less than M, the i value increases by 1; when it is greater than or equal to M, the calculation ends.

2. The data processing method according to claim 1, characterized in that: The weights of the convolution kernel are stored in a digital memory, which includes k rows, each row corresponds to k×Q units, and Q represents the weight accuracy; The switch array includes k×Q switch groups, each switch group includes P groups of switches distributed in columns, each switch group includes switches c1, ..., ck, one end of each switch in each switch group is correspondingly connected to a row of the same column unit of the digital memory, and the other end is correspondingly connected to a row of the same column unit of the convolution circuit array, and a reorganized convolution kernel Wi matching the input submatrix Yi can be formed by controlling all switches in any same group of switches in the k×Q switch groups to be turned on; Wherein, P is the quotient of the least common multiple of k and s divided by s.

3. The data processing method according to claim 1, characterized in that: The weight of the convolution kernel is stored in a digital memory, wherein the digital memory includes k rows, each row corresponds to k×Q units, and Q represents the weight accuracy of the convolution kernel; The switch array is arranged in such a manner that the switch array includes k×Q switch groups, each switch group includes k groups of switches distributed in columns, each group of switches includes switches c1, ..., ck, one side of each group of switches is sequentially connected to each row of the same column unit of the digital memory, the other side of the first group of switches is connected to the first row of the same column unit of the convolution circuit array, the other side of the second group of switches is connected to the second row of the same column unit of the convolution circuit array, and the other side of the kth group of switches is connected to the kth row of the same column unit of the convolution circuit array; By controlling one of the switches c1, ..., ck in the k×Q switch groups to be turned on, a reorganized convolution kernel Wi matching the input sub-matrix Yi can be formed.

4. The data processing method according to claim 3, characterized in that: From top to bottom, the arrangement order of the first group of switches of each switch group is c1, ..., ck, the arrangement order of the second group of switches is ck, c1, ..., ck-1, the arrangement order of the third group of switches is ck-1, ck, c1, ..., ck-2, and so on; The initial control vector C0 of the switch array is [1,0,0, ... ,0], and the control vector Ci corresponding to Yi is the initial control vector C0 cyclically shifted left by (i-1)×s bits.

5. The data processing method according to any one of claims 1 to 4, characterized in that: When k+(i-1)×s is greater than or equal to M, the calculation ends, the convolution result matrix is ​​output, and the convolution calculation of the next M×N input data continues.

6. A data processing device based on a sensor-storage-computation integrated architecture, used for processing the convolution calculation of an input matrix [A1, A2, ..., AM] of M×N input data, where the size of the convolution kernel W is k and the step length is s, characterized in that: include: An acquisition module is used to acquire k rows of input data and form an input sub-matrix Yi in the acquisition module, where i is 1, 2, ..., int ((Mk) / s+1); When i=1, the input submatrix Yi is [A1, A2, ..., Ak]; When i≥2, sequentially obtain the input data Ax+1, ..., Ax+s of S rows, x=k+(i-2)×s, and sequentially overwrite Ax+1-k, ..., Ax+sk in the input submatrix Yi-1 to obtain the input submatrix Yi; A storage module, used to store convolution kernels; A switch array, coupled to the storage module, for reordering the convolution kernels by row to generate a reorganized convolution kernel Wi externally; A loop control unit, used to control the acquisition module to generate an input sub-matrix Yi, and control the switch array to generate a reorganized convolution kernel Wi externally, wherein the reorganized convolution kernel Wi determines the row vector order according to the row number order of the input sub-matrix Yi; A convolution circuit array, used for sliding the reorganized convolution kernel Wi in the row direction of the input sub-matrix Yi to perform convolution calculation with the input sub-matrix Yi until the convolution calculation of the input sub-matrix Yi is completed; Among them, the value of i increases by 1 after the convolution calculation of the input sub-matrix Yi is completed.

7. The data processing device according to claim 6, characterized in that: The storage module is a digital memory, and the digital memory includes k rows, each row corresponds to k×Q units, and Q represents the weight accuracy of the convolution kernel; The switch array includes k×Q switch groups, each switch group includes k groups of switches distributed in columns, each group of switches includes switches c1, ..., ck, one end of each switch in each group of switches is connected to the same row of the same column unit of the digital memory and the other end is respectively connected to the rows of the same column unit of the convolution circuit array, or one end of each switch in each group of switches is respectively connected to the rows of the same column unit of the digital memory and the other end is connected to the same row of the same column unit of the convolution circuit array; The loop control unit controls one of the switches c1, ..., ck in the k×Q switch groups to be turned on so that the switch array generates a reorganized convolution kernel Wi that matches the input sub-matrix Yi.

8. The data processing device according to claim 7, characterized in that: One end of each switch in each group of switches is respectively connected to the rows of the same column unit of the digital memory and the other end is connected to the same row of the same column unit of the convolution circuit array; The arrangement of the switch array is as follows: from top to bottom, the arrangement order of the first group of switches is c1, ..., ck, the arrangement order of the second group of switches is ck, c1, ..., ck-1, the arrangement order of the third group of switches is ck-1, ck, c1, ..., ck-2, and so on; The initial control signal C0 of the switch array is [1, 0, 0, . . . , 0], and the control signal Ci corresponding to Yi is the initial control signal C0 cyclically shifted to the left by (i-1)×s bits.

9. A sensing, storage and computing integrated chip, characterized in that: comprising a sensing circuit and a data processing device as claimed in any one of claims 6 to 8; The sensing circuit captures image information from the outside world to generate input data. The acquisition module is an analog register. The loop control unit controls the input data of the sensing circuit to be stored in the analog register in batches. The input data stored in the analog register is the input sub-matrix Yi.

10. According to the sensing, storage and computing integrated chip of claim 9, the sensing circuit is a rolling shutter image sensing circuit.

Images