A method and apparatus for implementing MLP-MIXER image classification

Abstract

The application discloses a method and device for implementing an MLP-MIXER image classification algorithm. The method comprises the following steps: S1, preprocessing an input image to obtain two-dimensional data; S2, normalizing the data layer to prevent training overfitting, and completing a token MLP; S3, completing layer normalization and a channel MLP; S4, repeating S2-S3 N times, where N is the model depth, to complete the entire feature extraction process; and S5, completing classification through global average pooling and a fully connected layer. In the token MLP and the channel MLP, transposition is completed by using a DRAM array, matrix multiplication of the MLP is completed by using a FLASH array, and skip connection is completed by using an addition circuit. The application reduces the difficulty of designing a control data input / output memory-computing device array and the difficulty of deploying weight parameters to the memory-computing device array.

Description

Technical Field

[0001] This invention relates to a deep learning classification algorithm implemented on very large-scale integrated circuits, belonging to the fields of very large-scale integrated circuits and artificial intelligence. Background Technology

[0002] A Multilayer Perceptron (MLP) is a fully connected artificial neural network consisting of neurons in an input layer, hidden layers, an output layer, and non-linear activation functions. It is trained using a supervised learning technique called backpropagation. MLPs are commonly used as models for image classification tasks. Backpropagation trains the model parameters, enabling it to extract features effectively. MLP-MIXER addresses the challenge of MLPs extracting multi-channel data by unfolding images into two-dimensional data, preserving spatial and channel information. Furthermore, compared to CNNs, the preferred model for image classification, MLPs are computationally simpler, directly performing matrix operations on all pixels of the image data, making them easier to implement in hardware. MLP-MIXER has achieved competitive scores on image classification benchmarks, with pre-training and inference costs comparable to state-of-the-art models.

[0003] Currently, the deployment of MLP-MIXER still faces the following challenges: First, as an advanced model with high precision and simple structure, there is no precedent for implementing MLP-MIXER using a device. Second, MLP-MIXER still retains the characteristics of ANN models, with a large number of parameters. The traditional von Neumann architecture separates storage and computation, requiring access to memory to read and write data, resulting in high power consumption and high latency. Third, if in-memory computing devices, such as FLASH arrays, are used, their limited precision makes it impossible to store float32 parameters. Fourth, the transpose operation in the implementation details is an essential step in the model. Using conventional SRAM to design the corresponding modules would lead to high costs. Fifth, for the device as a whole, the arrangement of modules in the model is not reasonable enough, which may lead to complex hardware module interfaces and chaotic data paths, which is not conducive to design. Summary of the Invention

[0004] To address the aforementioned issues that may be encountered in implementing MLP-MIXER image classification, this invention applies a novel artificial neural network model to an in-memory computing architecture. It aims to provide a method and apparatus for implementing MLP-MIXER image classification, proposing an MLP-MIXER algorithm based on quantized perception training to increase the feasibility of model deployment.

[0005] The technical solution adopted in this invention is as follows:

[0006] A method for implementing MLP-MIXER image classification includes the following steps:

[0007] S1: The input image data is preprocessed and segmented and projected to form two-dimensional image data with the number of labels and the number of channels as two dimensions;

[0008] S2: Perform layer normalization on the two-dimensional image data to obtain layer normalized data. Transpose the layer normalized data to obtain transposed layer normalized data. Perform matrix multiplication on the transposed layer normalized data and the weights of the token MLP to obtain the token MLP processed data, thus completing feature extraction. Then, use skip connections to add the token MLP processed data and the transposed two-dimensional image data to obtain the first processed data.

[0009] S3: Transpose the first processed data to obtain the transposed first processed data, then perform layer normalization, and use matrix multiplication on the layer normalized data and the weights of the channel MLP to obtain the channel MLP processed data, complete feature extraction, and use skip connections to add the channel MLP processed data and the transposed first processed data to obtain the second processed data.

[0010] S4: Repeat steps S2-S3 multiple times to obtain the final feature image data;

[0011] S5: Perform global average pooling on the final feature image data, then complete the mapping from features to categories through a fully connected layer, and finally output a vector with the dimension of the number of categories.

[0012] Further, in step S1, the input image data is preprocessed and segmented for projection. Specifically, the RGB image of size shape(H,W,3) is divided into n blocks along the H and W dimensions. Each block is called a patch, and each patch contains multiple pixels. The pixels of each patch are unfolded into vectors, and linear projection is performed using a projection matrix. The same projection matrix is ​​used for each patch, thus obtaining an encoded image of size... The two-dimensional feature map, in the new feature map, is the number of tags, i.e. the number of patches in the original feature map, which is an integer power of 2. C represents the number of channels, i.e. the length of the vector after the original feature map is segmented and projected.

[0013] Furthermore, steps S2 and S3 both employ a quantization-aware training method. This method pseudo-quantizes the weights and activations during forward propagation, introducing quantization error. During backpropagation, it updates the unquantized weights using gradients to obtain weight parameters and quantization parameters adapted to the quantization error. Before inference, the weights are first fixed as integers, and the quantization coefficients of the activations are fixed, quantizing the activations as integers. During inference, quantized activations and weights are used to complete integer calculations, and then the quantization coefficients are used for scaling, thereby achieving the goal of separating integer and floating-point operations.

[0014] The present invention also provides an apparatus for implementing MLP-MIXER image classification, the apparatus comprising:

[0015] Preprocessing module: used to segment and project the input image data to form two-dimensional image data with the number of labels and the number of channels as two dimensions;

[0016] The MIXER module includes a layer normalization module, a transpose module, a token MLP module, and a channel MLP module, which are used to implement the layer normalization, transpose, and feature extraction functions of the two-dimensional data after passing through the MLP module, respectively. In addition, the MIXER module also includes an addition circuit to complete the skip connection operation.

[0017] The classification module includes a global average pooling layer and a fully connected layer, which are used to perform global average pooling on the feature map and map the features to the category to complete the classification, respectively.

[0018] Furthermore, the transpose module includes a DRAM array, a DAC array at the input end of the DRAM array and an ADC array at the output end, an input address control unit, and an output address control unit. Before entering the DRAM array, the input data is converted into an analog signal by the DAC array. The input address control unit inputs the input data column-wise or row-wise by the input address selection signal. After all the data is input, the output address control unit selects to output the data row-wise or column-wise by the output address selection signal. The output data is converted into a digital signal by the ADC array to obtain the final output signal.

[0019] Furthermore, the token MLP module and the channel MLP module each include two FLASH arrays. The FLASH arrays perform convolution operations in model inference. Each weight is quantized into 8 bits and stored in a single FLASH array device. The data is driven by a high-voltage driving circuit to form a driving voltage, which drives the FLASH array to generate an accumulated current. Finally, the current is first converted into voltage, and then into a digital signal to be transmitted to the next layer.

[0020] Furthermore, in the transpose module, the output data is converted into a digital signal via an ADC array, and its ADC conversion time satisfies: The data retention time in the DRAM array satisfies: T flash <T0<T c Where m is the number of rows of the matrix to be transposed, n is the number of columns of the matrix to be transposed, x is the number of DACs in the DAC array, y is the number of ADCs in the ADC array, and T DAC T is the time it takes for the DAC array to receive digital signals and write analog signals. ADC T is the time it takes for the ADC array to convert an analog signal into a digital signal. cT is the minimum time T is the time it takes for leakage current in a DRAM array device to cause a bit error in the stored value. flash T0 is the time required to complete the calculations of the FLASH array in the token MLP module and the channel MLP module. T0 is the data retention time in the DRAM array. Before the data in the FLASH array and DRAM array completes the calculations, the data in the DRAM array must not have any errors.

[0021] The present invention has the following beneficial effects:

[0022] (1) The present invention designs the MLP-MIXER quantization sensing algorithm based on the physical characteristics of the in-memory computing device array, which can well realize the integration of the algorithm with the in-memory computing architecture.

[0023] (2) Using MLP instead of CNN as the basic unit of the model reduces the difficulty of designing and controlling the data input and output memory computing array and the difficulty of deploying weight parameters to the memory computing array.

[0024] (3) The original model structure of MLP-MIXER was adjusted to speed up the inference pipeline.

[0025] (4) Considering the physical characteristics of DRAM and the transposition requirements of the model, the transposition operation is completed using a DRAM array, avoiding the drawback of DRAM needing to be refreshed and reducing the implementation cost of the entire device. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the structure of the MLP-MIXER image classification device of the present invention;

[0027] Figure 2 This is a schematic diagram of the circuit implementation of the MIXER module of the present invention;

[0028] Figure 3 This is a schematic diagram of the algorithm model of the MLP-MIXER image classification method of the present invention;

[0029] Figure 4 This is a flowchart illustrating the MLP-MIXER image classification method of the present invention. Detailed Implementation

[0030] like Figure 1 As shown, this embodiment provides a structure for an MLP-MIXER image classification device, including:

[0031] Preprocessing module: Used to preprocess three-channel RGB images, segment and project the input image data to form two-dimensional image data with the number of labels and the number of channels as two dimensions;

[0032] The MIXER module mainly includes a layer normalization module, a transpose module, a token MLP module, and a channel MLP module. These modules respectively implement layer normalization, transpose, and feature extraction functions for the 2D data after passing through the MLP modules. Additionally, an addition circuit is included to handle skip connections. The layer normalization module implements the token MLP module in software to process different tags within the same channel dimension. The token MLP module operates independently on each channel, facilitating communication between different tags. The channel MLP module operates independently on each tag, facilitating communication between different channels. The interaction between the two dimensions is achieved through the repeated application of these two MLP modules across the tag and channel dimensions. The token MLP module has the same parameters for different channel dimensions, so one token MLP module is shared for each channel dimension; similarly, one channel MLP module is shared for each tag dimension.

[0033] The classification module includes a global average pooling layer and a fully connected layer, which are used to perform global average pooling on the feature map and map the features to the category to complete the classification. The classification module averages the data along the channel direction to obtain distributed features and maps the distributed features to the sample space to complete the subsequent classification work. The classification module is implemented in software.

[0034] like Figure 2 As shown, this embodiment provides a circuit implementation of a MIXER module, including a layer normalization module, a transpose module, a token MLP module, a channel MLP module, and an adder circuit. Specifically, the transpose module includes a DRAM array, a DAC array at the input of the DRAM array and an ADC array at the output, an input address control unit, and an output address control unit. The token MLP module includes a token FLASH array, a high-voltage drive circuit at the input of the token FLASH array, and an ADC array at the output of the token FLASH array. The channel MLP module includes a channel FLASH array, a high-voltage drive circuit at the input of the channel FLASH array, and an ADC array at the output of the channel FLASH array.

[0035] The layer normalization module normalizes the input two-dimensional data and outputs it as an 8-bit digital signal to the DAC arrays of DRAM array 1 and DRAM array 3. The DAC array of DRAM array 1 converts the two-dimensional data from digital signals to analog signals column by column. The input address control unit receives the address signal and determines which column to write into the DRAM array. After reading all column vectors, the output address control unit receives the address signal, determines which row to read from the DRAM array, and converts it into a digital signal via the ADC array, outputting it to the high-voltage drive circuit of the token MLP array. At this point, the data is still stored in the DRAM array and is not refreshed. The token FLASH array reads the digital signal bit by bit, which is then converted into a drive voltage by the high-voltage drive circuit. The drive voltage drives the devices to generate current, and each column of devices accumulates current, which is then converted into voltage and passed through the FLASH array. The ADC array at the column output terminal converts the signal into a digital signal and sends it to adder circuit 1. The digital signal received by the DAC array of DRAM array 3 is processed according to the processing method of the transpose module where DRAM array 1 is located, that is, written by column and read by row, and the output digital signal is sent to adder circuit 1. After the adder circuit 1 completes the addition operation, it sends the output to the DAC array of DRAM array 2. The DAC array of DRAM array 2 adopts a different processing method than the transpose modules where DRAM array 1 and DRAM array 3 are located, written by row and read by column, and sends the output to the layer normalization module and adder circuit 2 on the software side. The layer normalization module outputs an 8-bit digital signal to the high-voltage drive circuit of the channel FLASH array, which processes it according to the processing method of the token MLP module where the token FLASH array is located, and outputs a digital signal to adder circuit 2. Adder circuit 2 completes the addition operation and outputs an intermediate feature map.

[0036] Figure 3 This is a schematic diagram of the algorithm model of the MLP-MIXER image classification method of this invention. During the training process, the Reald, Linear, LayerNorm, Convld, ReLU, GlobalAveragePooling, and add operators are used: The Reald operator completes image segmentation. Assuming the original image shape is (H, W, 3), where 3 is the number of channels in the original image, and H and W are the height and width of the original image, after dividing the image into n blocks along the height and width, the shape becomes... Following the Rearange operator, the Linear operator performs the image projection mapping, transforming the shape into... The segmentation map is mapped to a shape of The data consists of two dimensions; MIXER-1 contains multiple operators, all of which are divided into two columns. The LayerNorm operator in the left column performs layer normalization, followed by the combination of Linear, ReLU, and Linear operators to perform token MLP operations, and the add operator to perform skip-connection operations. The LayerNorm operator in the right column performs layer normalization, followed by the combination of Convld, ReLU, and Convld operators to perform channel MLP operations, and the add operator to perform skip-connection operations. MIXER is repeated N times to MIXER-N, and the operators and functions in each MIXER are the same. The GlobalAvarage Pooling operator performs global average pooling, that is, averaging the value of each channel. Finally, the Linear operator performs classification. Quantization-aware training is used for Linear operators, Convld operators, add operators, and LayerNorm operators to improve inference accuracy. It should be noted that this invention replaces the GeLU activation function in the existing MLP-MIXER algorithm (Tolstikhin IO, Houlsby N, Kolesnikov A, et al. Mlp-mixer: An all-mlparchitecture for vision[J]. Advances in neural information processing systems, 2021, 34: 24261-24272.) with the ReLU activation function. Due to the inherent truncation characteristics of ReLU, it can be integrated with Conv and Linear operators in quantization-aware training to speed up inference and avoid the design of activation function units in the circuit.

[0037] Token MLP and Channel MLP are the fundamental operations for the entire model computation. The following describes the quantization-aware training process of the basic operators used in these two MLP operations, where both the Linear operator and the Conv1d operator are matrix multiplication operators. Quantization requires quantizing the weights and activations (i.e., the output of each operator). First, the quantization scheme is declared: weights contain positive and negative data; the outputs of the second Linear operator in Token MLP and the second Conv1d operator in Channel MLP, since they do not pass through the ReLU activation function, also contain positive and negative data, so symmetric quantization is used; the first Linear operator in Token MLP and the first Conv1d operator in Channel MLP can be fused with the subsequent ReLU function, and the output of the fused operator does not contain negative data, so affine quantization is used. Then, the zeros in the quantization parameters are explained, denoted by z, q... maxi q mini r maxi and r miniThese represent the maximum and minimum values ​​of the quantization range, and the maximum and minimum values ​​of the floating-point range, respectively. Symmetric quantization can be used to obtain... Using affine quantization The zeros obtained by using both quantization schemes are 0, so during training, only the quantization coefficient q in the quantization parameters needs to be considered.

[0038] The following formula illustrates the quantization of each operator in the MLP. Note that the matrix calculations for the Convld and Linear operators are simplified to the form y = wx + b for easier understanding. The formula involves two functions: round and clamp. round is the rounding function, and clamp(min, max) is the truncation function, where min represents the lower bound of the truncation and max represents the upper bound. In addition, two annotations are included: (train) and (infer). (train) represents the forward propagation formula for the training process, and (infer) represents the integer calculation formula for inference. To ensure that the scaling of integer calculation results can be achieved through bit shifting, the quantization coefficients must be limited to integer powers of 2.

[0039] (a) First, quantize the input data, where x represents floating-point data and x′ represents pseudo-quantized data obtained after quantization and dequantization operations. q q represents the quantized data. x The quantization coefficient of data x:

[0040]

[0041] x q =round(q x x).clamp(-128, 127)(infer)

[0042] (b) Quantization layer normalization operator: First, calculate the mean and variance of the input data, where n represents the number of labels and μ... x and σ x Let x represent the mean and standard deviation, respectively. During inference, the mean and standard deviation can be calculated using the input quantized value and quantization coefficient.

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049] The output of the layer normalization operator is computed in real time. γ and β are hyperparameters fixed during training. l represents the layer normalized data, and l′ represents the pseudo-quantized layer normalized data. q q represents the quantized layer-normalized data. l The quantization coefficients represent the normalized data l of the layer. A and B are the quantization coefficients calculated in real time, and ∈ is a minimum value added to avoid the denominator being 0.

[0050]

[0051]

[0052]

[0053] Affine quantization is applied to the fusion operator of the first operator of the MLP operation and the ReLU operator as follows: y1 is the floating-point output activation of the fusion operator, w1 is the floating-point weight of the fusion operator, b1 is the floating-point bias of the fusion operator, and y′1 and w′1 represent the pseudo-quantized fusion operator output activation and pseudo-quantized fusion operator weights, respectively. q1 w q1 and b q1 q represents the quantized fusion operator output activation, quantized fusion operator weights, and quantized fusion operator biases, respectively. y1 q w1 and q b1 These represent the quantization coefficients of y1, w1, and b1, respectively.

[0054]

[0055]

[0056] The second operator in the MLP operation is symmetrically quantized as follows: y2 is the floating-point output activation of the second operator, w2 is the floating-point weight of the second operator, b2 is the floating-point bias of the second operator, and y′2 and w′2 represent the output activation and weight of the pseudo-quantized second operator, respectively. q2 w q2 and b q2 Let q represent the output activation, weights, and biases of the second quantized operator, respectively. y2 q w2 and q b2 These represent the quantization coefficients of y2, w2, and b2, respectively.

[0057]

[0058]

[0059] Symmetric quantization is used for skip connections:

[0060]

[0061]

[0062] like Figure 4 As shown, the flowchart of an MLP-MIXER image classification method provided in this embodiment, taking the processing of a 32*32*3 image as an example, includes the following steps:

[0063] Step 1: Convert the image data into a tensor and normalize it to (-1,1). Divide the image data into 8 blocks along the height H and width W to obtain 64 patches, each containing 48 pixels. Apply a projection matrix to the pixels of the same patch, set the number of output channels to C, and obtain two-dimensional data with shape (64,C). Quantize the two-dimensional data into 8-bit integer data.

[0064] Step Two: Perform layer normalization on the two-dimensional image data in the software to prevent overfitting during training. First, statistical analysis is performed on the quantized data, calculating the mean of all two-dimensional data. Then calculate the mean of the squares of all the two-dimensional data. Obtain the variance Using the aforementioned statistical data, the layer normalized output is quantized into an 8-bit digital signal and output to the DAC arrays of DRAM array 1 and DRAM array 3. The DAC array of DRAM array 1 converts the two-dimensional data from digital signals to analog signals column by column. The input address control unit receives the address signal and determines which column to write into the DRAM array. After reading all column vectors, the output address control unit receives the address signal and determines which row to read from the DRAM array. The data is then converted into a digital signal by the ADC array and output to the high-voltage drive circuit of the token MLP array. At this time, the data is still stored in the DRAM array and will not be refreshed. The token FLASH array reads the digital signal bit by bit. The high-voltage drive circuit generates a drive voltage, which drives the devices to generate current. Each column of devices generates an accumulated current, which is converted into voltage and then converted into a digital signal by the ADC array at the output of the FLASH array. This signal is then transmitted to the adder circuit 1. The digital signal received by the DAC array of DRAM array 3 is processed according to the processing method of the transpose module where DRAM array 1 is located, i.e., written column by column and read row by row. The output digital signal is then transmitted to the adder circuit 1. After completing the addition operation, the adder circuit 1 transmits the output to the DAC array of DRAM array 2.

[0065] Step 3: Adopting a different processing method than the transpose modules of DRAM array 1 and DRAM array 3, write by row and read by column, and transmit the output to the layer normalization module and adder circuit 2 in the software. The layer normalization module outputs an 8-bit digital signal to the high-voltage drive circuit of the channel FLASH array, processes it according to the processing method of the token MLP module where the token FLASH array is located, and outputs a digital signal to adder circuit 2. Adder circuit 2 completes the addition operation and outputs an intermediate feature map. The DRAM used in the hardware implementation in Step 3 needs to meet certain time constraints. The ADC conversion time satisfies: The data retention time in the DRAM array satisfies: T flash <T0<T c Where m is the number of rows of the matrix to be transposed, n is the number of columns of the matrix to be transposed, x is the number of DACs in the DAC array, y is the number of ADCs in the ADC array, and T DAC T is the time it takes for the DAC array to receive digital signals and write analog signals. ADC T is the time it takes for the ADC array to convert an analog signal into a digital signal. c T is the minimum time T is the time it takes for leakage current in a DRAM array device to cause a bit error in the stored value. flash T0 is the time required to complete the calculation of the FLASH array, and T0 is the data retention time in the DRAM array. To accelerate the pipeline speed, this invention swaps the order of the transpose operation and skip connection operation in the token MLP part of the existing MLP-MIXER algorithm. The MLP operation and layer normalization operation do not change the shape of the two-dimensional data. Assuming the shape of the input two-dimensional data is (P, C), where P is the number of tokens and C is the number of channels, in the existing MLP-MIXER model, the input data is sequentially processed by the layer normalization operation, transpose operation, and token MLP operation to obtain output data with shape (C, P). Before the swap, the output data is transposed to obtain data with shape (P, C) and then added to the input data; after the swap, the input data needs to be transposed beforehand, and the transposed input data and output data are added to obtain data with shape (C, P), and then the addition result is transposed. Since DRAM array 3 needs to wait for the token FLASH behind DRAM array 1 to complete the calculation, in order to prevent data loss, the data transmission on the software side cannot be interrupted. Swapping the order of transposing and skipping connections in the token MLP part of the existing MLP-MIXER algorithm actually advances the data reading of DRAM array 3 (in the original algorithm, DRAM array 3 only plays the role of storing data, because only the original data needs to be read and not transposed. After the algorithm is modified, the transposition function of DRAM array is also effectively utilized). When multiple images need to be inferred, the data in DRAM array 3 can be updated faster, and the pipeline speed can be accelerated.

[0066] Step 4: Set up 8 MIXER modules and repeat steps 2 and 3 8 times to obtain the final feature image digital signal. The number of repetitions here is a hyperparameter, which can be set to 8, 12, 24, or 32 times. The more repetitions, the larger the number of parameters.

[0067] Step 5: Send the digital signal into the software to complete global average pooling and classification.

[0068] The above description is merely a specific embodiment of this disclosure. It should be noted that the image preprocessing device used, the FLASH array used, the image resolution, and the image content do not constitute a limitation on this invention. The scope of protection of this invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for implementing MLP-MIXER image classification, characterized in that, The method comprises the following steps: S1: the input image data is segmented and projected after preprocessing to form two-dimensional image data with the number of markers and the number of channels as two dimensions; S2: the two-dimensional image data is layer normalized to obtain layer normalized data, the layer normalized data is transposed to obtain transposed layer normalized data, the transposed layer normalized data and the weight of the token MLP are subjected to matrix multiplication to obtain token MLP processed data, feature extraction is completed, and the token MLP processed data and the transpose of the two-dimensional image data are added using skip connection to obtain first processing data; S3: the first processing data is transposed to obtain transposed first processing data, and then layer normalization is performed on the data, and the weight of the channel MLP is subjected to matrix multiplication to obtain channel MLP processed data, feature extraction is completed, and the channel MLP processed data and the transposed first processing data are added using skip connection to obtain second processing data; S4: steps S2-S3 are repeated multiple times to obtain final feature image data; S5: the final feature image data is subjected to global average pooling operation, and then feature-to-class mapping is completed through full connection, and finally a vector with a classification number is output.

2. The method for implementing MLP-MIXER image classification according to claim 1, characterized in that, In step S1, the input image data is preprocessed and segmented and projected, specifically: an RGB image with a size of shape(H, W, 3) is divided into n blocks along the H and W dimensions, each block is called a patch, and each patch contains a plurality of pixel points; the pixel points of each patch are unfolded into a vector, and linear projection is performed using a projection matrix, and the projection matrix used by each patch is the same, so that a two-dimensional feature map with a size of is obtained, in the new feature map, is the number of markers, that is, the number of patches of the original feature map, is an integer power of 2, and C represents the number of channels, that is, the length of the vector after segmentation and projection of the original feature map.

3. The method of claim 1, wherein, In steps S2 and S3, a quantization-aware training method is used, which pseudo-quantizes the weight and activation during forward propagation, introduces quantization error, and updates the weight before quantization using gradient during backward propagation to obtain weight parameters and quantization parameters that adapt to the quantization error; before inference, the weight is first fixed as an integer, and the quantization coefficient of the activation is fixed, and the activation is quantized to an integer, in the inference process, the quantization activation and weight are used to complete the integer calculation, and then the quantization coefficient is used for scaling, so as to separate the integer and floating point operations.

4. An apparatus for implementing MLP-MIXER image classification, the apparatus comprising: The device comprises: a preprocessing module for segmenting and projecting the input image data to form two-dimensional image data with the number of markers and the number of channels as two dimensions; a MIXER module comprising a layer normalization module, a transposition module, a token MLP module and a channel MLP module, which are respectively used to realize the layer normalization function, the transposition function and the feature extraction function of the two-dimensional data after the MLP module; in addition, the MIXER module further comprises an addition circuit for completing the skip connection operation; a classification module comprising a global average pooling layer and a full connection layer, which are respectively used for global average pooling of the feature map and mapping of the feature to the class to complete classification.

5. The apparatus for implementing MLP-MIXER image classification according to claim 4, wherein, The transposition module comprises a DRAM array, a DAC array at the input end of the DRAM array and an ADC array at the output end, an input address control unit and an output address control unit, the input data is converted into an analog signal by the DAC array before entering the DRAM array, the input data is input by column or row through the input address control unit by input address selection signal, after all the data are input, the data are output by row or column through the output address control unit by output address selection signal, and the output data are converted into a digital signal through the ADC array to obtain the final output signal.

6. The apparatus for implementing MLP-MIXER image classification according to claim 5, wherein, The token MLP module and the channel MLP module each include two FLASH arrays, the FLASH arrays complete convolution operations in model inference, each weight is quantized into 8 bits and exists in a FLASH array device, data passes through a high-voltage driving circuit to form a driving voltage, the driving voltage drives the FLASH array to generate accumulated current, and finally, the current is converted into a voltage and then into a digital signal and transmitted to the next layer.

7. The apparatus for implementing MLP-MIXER image classification according to claim 5, wherein, In the transposition module, the output data is converted into digital signals through the ADC array, and the ADC conversion time satisfies: The data storage time in the DRAM array satisfies: T flash <T0<T c Wherein m is the number of rows of the matrix to be transposed, n is the number of columns of the matrix to be transposed, x is the number of DACs in the DAC array, y is the number of ADCs in the ADC array, T DAC is the time for the DAC array to receive digital signals and write analog signals, T ADC is the time for the ADC array to convert analog signals into digital signals, T c is the minimum time for the leakage of the DRAM array device to cause errors in the stored values, T flash is the time required to complete the calculation of the FLASH array in the token MLP module and the channel MLP module, and T0 is the data storage time in the DRAM array. Before the data in the FLASH array and the DRAM array complete the operation, the data in the DRAM array cannot have errors.

Images