A handwritten digit recognition method and system based on memristors

Abstract

This invention discloses a handwritten digit recognition method and system based on memristors, relating to the field of memristor technology. The method includes: receiving input handwritten digit image data; performing a convolution operation on the input handwritten digit image data to obtain feature map data; converting the feature map data into a multidimensional array and labeling it as multidimensional feature data; dividing the multidimensional feature data by columns and assigning a weight coefficient to each multidimensional feature data; multiplying and adding the multidimensional feature data of the corresponding column with the corresponding weight coefficient to obtain the calculation results for all columns; multiplying and adding the calculation results of all columns with the multidimensional feature data of the corresponding column to obtain a calculation result array; assigning probabilities to the data in the calculation result array by columns to obtain the probability coefficient corresponding to each column of data; mapping the probability coefficients into a memristor array; inputting the feature map data into the memristor array to obtain the output result; and returning the output result to a computer, thereby completing the handwritten digit image recognition task.

Description

Technical Field

[0001] This invention relates to the field of memristor technology, specifically a handwritten digit recognition method and system based on memristors. Background Technology

[0002] With the continuous development of deep learning, memristor-based neural network accelerators offer a more efficient and low-power architecture, enabling matrix-vector multiplication (MVM) through "physical computation." However, hardware-accelerated systems, represented by convolutional neural networks (CNNs), still face some challenges. For instance, in memristor-based handwritten digit recognition systems, convolution is used to extract features, followed by pooling to downsample and reduce the feature map size. However, both max pooling and average pooling suffer from the problem of losing significant image information, compromising translation invariance, and reducing robustness to affine transformations. This leads to poor performance of neural networks in image-related tasks, such as handwritten digit recognition. In this task, the image contains 60,000 handwritten random digits (0-9), and the neural network must recognize these digits despite random noise, demanding high robustness. An algorithm called dynamic routing proposes to replace pooling with iteration to retain more useful information and improve the robustness of the neural network. The main idea is to use cosine similarity as a metric, assigning a coefficient between 0 and 1 to each input feature map. More important input data receives a larger coupling coefficient, while the coupling coefficients of other input data are decreased. Initially, all input data have the same coupling coefficient, summing to 1. The coupling coefficients are updated by multiplying the input data by their respective coefficients and then applying the result to a SoftMax activation function, continuing until the set number of iterations is reached. This dynamic routing algorithm effectively couples the relationships between input data, improving the network's robustness. However, a drawback is the need for continuous training and updating of the routing coefficients, leading to significant power consumption for the computer. Summary of the Invention

[0003] To address the shortcomings mentioned in the background section, the present invention aims to provide a handwritten digit recognition method and system based on memristors. This method utilizes the multi-valued characteristics of memristors to store iterative floating-point routing coefficients, improving energy efficiency and enabling the deployment of memristor-based handwritten digit recognition tasks.

[0004] The objective of this invention can be achieved through the following technical solution: a handwritten digit recognition method based on memristors, the method comprising the following steps:

[0005] Receive input handwritten digit image data, perform convolution operation on the input handwritten image data to obtain feature map data, convert the feature map data into a multidimensional array mode, and label it as multidimensional feature data;

[0006] The multidimensional feature data is divided into columns, and a weight coefficient is assigned to each multidimensional feature data. The multidimensional feature data of the corresponding column is multiplied and added with the weight coefficient of the corresponding column to obtain the calculation results of all columns.

[0007] Multiply and add the calculation results of all columns with the multidimensional feature data of the corresponding columns to obtain an array of calculation results;

[0008] Assign probabilities to the data in the calculation result array by column to obtain the probability coefficients corresponding to all data. Map the probability coefficients to the memristor array, input the feature map data into the memristor array, and obtain the output results.

[0009] The output is returned to the computer via TIA, SA, and ADC circuits, thereby realizing the function of the adaptive pooling layer and using the inference process of the memristor to complete the recognition of handwritten digit images.

[0010] Preferably, the weight coefficient is the probability corresponding to each feature map data. It is assigned by column. In the initial case, the probability of all data in all columns is the same, which is 1 / the sum of the number of data in each column. After the probability is updated by the SoftMax activation function, after a set number of iterations, the obtained weight coefficient represents the importance of the corresponding feature map data, realizing the effect of adaptive pooling. The iterated adaptive weight coefficient is mapped to the memristor array to perform the inference process. The sum of the weight coefficients of each column of devices is 1.

[0011] Preferably, the feature map data is three-dimensional tensor data obtained after feature extraction through convolution operation. The three-dimensional tensor data is divided into "row" dimension, "column" dimension and "depth" dimension according to different dimensional directions.

[0012] Preferably, the memristor array has multiple conductance states between high and low resistance states, which can replace the cache in a computer to store floating-point data in a single cell, thereby reducing computer power consumption.

[0013] Preferably, the memristor array is responsible for storing the adaptive pooling coefficient matrix corresponding to the feature map data, that is, assigning corresponding coefficients to all input feature map data, and then outputting current through the memristor array, which is then converted by the circuit and returned to the computer to realize the function of adaptive pooling.

[0014] Preferably, parallel computing using a memristor array is used for parallel acceleration, and inference data is output in parallel and then returned to the computer. The inference process can significantly reduce training power consumption and improve training efficiency.

[0015] Preferably, the process of assigning a weight coefficient to each multidimensional feature data is updated using the idea of ​​cosine similarity.

[0016] Preferably, the results obtained when feature map data of different sizes are multiplied with the same weight coefficient column are different. The matrix multiplication is performed again based on the calculation results of all columns of the matrix multiplication and the multidimensional feature data of the corresponding column. The larger the calculation result of the corresponding column, the closer the feature map data is to the calculation result of the corresponding column. Therefore, a larger weight coefficient needs to be assigned.

[0017] Secondly, in order to achieve the above objectives, the present invention discloses a handwritten digit recognition system based on a memristor, comprising:

[0018] Convolution module: It is used to receive input handwritten digit image data, perform convolution operation on the input handwritten image data to obtain feature map data, convert the feature map data into a multidimensional array mode, and label it as multidimensional feature data;

[0019] Data partitioning module: This module is used to partition multidimensional feature data by column, assign a weight coefficient to each multidimensional feature data, and multiply and add the multidimensional feature data of the corresponding column with the weight coefficient of the corresponding column to obtain the calculation results of all columns.

[0020] The multiply-add module is used to multiply and add the calculation results of all columns with the multidimensional feature data of the corresponding columns to obtain an array of calculation results;

[0021] The probability allocation module is used to assign probabilities to the data in the calculation result array by column, obtain the probability coefficients corresponding to all data, map the probability coefficients to the memristor array, input the feature map data into the memristor array, and obtain the output results.

[0022] Adaptive module: Used to return the output results to the computer through TIA, SA, and ADC circuits, thereby realizing the function of adaptive pooling layer.

[0023] The beneficial effects of this invention are:

[0024] This invention better utilizes the parallel computing capabilities of memristors to achieve adaptive pooling, solving the problem of traditional pooling layers over-compressing images and reducing the robustness of neural networks. Based on this invention, it is possible to recognize handwritten digit images. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0027] Figure 2 This is a flowchart illustrating the routing training and inference process of the handwritten digit recognition method and system implementation scheme based on memristors as described in this embodiment of the invention.

[0028] Figure 3 This is a schematic diagram showing the results of long-term pulse enhancement and long-term pulse attenuation of the memristor array according to an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of the handwritten digit recognition method and system based on memristors according to an embodiment of the present invention;

[0030] Figure 5 This is a system structure diagram of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] like Figure 1 As shown, a handwritten digit recognition method based on memristors includes the following steps:

[0033] Receive input handwritten digit image data, perform convolution operation on the input handwritten image data to obtain feature map data, convert the feature map data into a multidimensional array mode, and label it as multidimensional feature data;

[0034] The multidimensional feature data is divided into columns, and a weight coefficient is assigned to each multidimensional feature data. The multidimensional feature data of the corresponding column is multiplied and added with the weight coefficient of the corresponding column to obtain the calculation results of all columns.

[0035] Multiply and add the calculation results of all columns with the multidimensional feature data of the corresponding columns to obtain an array of calculation results;

[0036] The data in the calculation result array are assigned probabilities by column to obtain the probability coefficients corresponding to all data. The probability coefficients are then mapped to the memristor array. The feature map data is used as input and mapped to the memristor array. After passing through circuits such as DAC and SA, it is converted into the form of memristor array read voltage. Matrix multiplication and addition inference operations are performed on the array weights. The voltage range is 0-0.3V, and the output result is obtained.

[0037] The output is returned to the computer via TIA, SA, and ADC circuits, thus realizing the function of handwritten digit image recognition.

[0038] It should be further explained that, in the specific implementation process, the weight coefficient is the probability corresponding to each feature map data. It is allocated by column. In the initial case, the probability of all data in all columns is the same, which is 1 / the sum of the number of data in each column. After the probability is updated by the SoftMax activation function, after reaching the set number of iterations, the obtained weight coefficient represents the importance of the corresponding feature map data, realizing the effect of adaptive pooling. The sum of the weight coefficients of each column of devices is 1.

[0039] It should be further explained that, in the specific implementation process, the feature map data is three-dimensional tensor data obtained after feature extraction through convolution operation. The three-dimensional tensor data is divided into "row" dimension, "column" dimension and "depth" dimension according to different dimensional directions.

[0040] It should be further explained that, in the specific implementation process, the memristor array has multi-conductivity state characteristics between high and low resistance states, which can replace the cache in the computer to store floating-point data in a single cell, thereby reducing the power consumption of the computer.

[0041] It should be further explained that, in the specific implementation process, the memristor array is responsible for storing the adaptive pooling coefficient matrix corresponding to the feature map data, that is, assigning corresponding coefficients to all input feature map data, and then outputting current through the memristor array, which is then converted by the circuit and returned to the computer to realize the function of adaptive pooling.

[0042] It should be further explained that by using memristor arrays for parallel computing to accelerate parallel processing and outputting inference data in parallel, which is then returned to the computer, the inference process can significantly reduce training power consumption and improve training efficiency.

[0043] In this embodiment, the process of assigning a weight coefficient to each multidimensional feature data is updated using the idea of ​​cosine similarity.

[0044] It should be noted that different sizes of feature map data will produce different results when multiplied with the same weight coefficient column. The larger the array of calculated results obtained by multiplying the multi-column data of the corresponding column with the matrix multiplication of all columns, the closer the feature map data is to the calculated results of all corresponding columns. Therefore, a larger weight coefficient needs to be assigned.

[0045] In this embodiment, for the handwritten digit image recognition task of a general neural network, the processing procedure is as follows: The host computer first extracts features from the input handwritten digit image data through a convolutional kernel to obtain tensor feature map data. The coupling coefficients are then updated according to the update scheme of this embodiment, and the coupling coefficients are mapped to a memristor array. The input data is then normalized and scaled from the computer before being imported into the memristor array for matrix multiplication and addition read operations. During the read operation, the read voltage must not exceed 0.3V; otherwise, it will change the array's stored weights, causing errors. Therefore, the input data is first normalized and scaled down. After updating the coefficient matrix and completing the inference process, the output data is then normalized and amplified, allowing the calculation result to be directly returned to the host computer without complex transformations. After converting the input data into a read voltage, it is multiplied and added with the initial coupling coefficients stored in array 2 to obtain the output result. When the number of iterations is 0, the initial weights of array 2 are all consistent, equivalent to the average pooling function of a traditional neural network. When the number of iterations is n, the maximum pooling function can be achieved, where n is a large iteration number.

[0046] After the pooling coefficients are updated, the input data undergoes adaptive pooling through a memristor array storing the pooling coefficient matrix. The output is then returned to the host computer for the subsequent training process of the handwritten digit image recognition task. Utilizing memristors for parallel updates reduces computer throughput and improves energy efficiency.

[0047] Figure 2 This is a flowchart illustrating the training and inference process of the memristor-based handwritten digit recognition method and system described in this embodiment of the invention. Figure 1 The flowchart of the steps describes updating matrix 2, which contains the pooling coefficients C, and then updating the weights on the host computer to train the network until convergence. The training process is completed on the host computer. After training, the inference process is executed. The inference process performs adaptive pooling by mapping the trained coefficient matrix to a memristor array.

[0048] Figure 3 To address the long-term enhancement and long-term reduction results observed in the pulse test of the memristor array described in this embodiment of the invention, the weight update nonlinearity of the memristor array device is extracted to compensate for the weight update accuracy in software, thereby improving the weight modulation efficiency and accuracy.

[0049] Figure 4This is a schematic diagram illustrating the specific operation of the memristor-based handwritten digit recognition method described in this embodiment of the invention. Upon receiving a handwritten digit image, the computer first extracts features through a convolutional layer. Then, the obtained feature map information is combined and encapsulated into blocks, each representing different abstract information. The features of the handwritten digit image are learned by adjusting the weights of different blocks. The obtained feature blocks are then subjected to adaptive pooling coefficient iteration. After iterating a specified number of times, the resulting pooling coefficient matrix is ​​mapped onto a memristor array, and inference operations are performed, implementing adaptive pooling inference functionality on the memristor array. The obtained data is returned to the computer for flattening, and then passed through three fully connected layers for recognition. The digit represented by the neuron with the highest probability among the 10 output neurons is output. In the example shown in the diagram, the third neuron represents the digit 3.

[0050] On the other hand, such as Figure 5 As shown, this embodiment of the invention also discloses a handwritten digit recognition system based on a memristor, comprising:

[0051] Convolution module: It is used to receive input handwritten digit image data, perform convolution operation on the input handwritten image data to obtain feature map data, convert the feature map data into a multidimensional array mode, and label it as multidimensional feature data;

[0052] Data partitioning module: This module is used to partition multidimensional feature data by column, assign a weight coefficient to each multidimensional feature data, and multiply and add the multidimensional feature data of the corresponding column with the weight coefficient of the corresponding column to obtain the calculation results of all columns.

[0053] The multiply-add module is used to multiply and add the calculation results of all columns with the multidimensional feature data of the corresponding columns to obtain an array of calculation results;

[0054] The probability allocation module is used to assign probabilities to the data in the calculation result array by column, obtain the probability coefficients corresponding to all data, map the probability coefficients to the memristor array, input the feature map data into the memristor array, and obtain the output results.

[0055] Adaptive module: Used to return the output results to the computer through TIA, SA, and ADC circuits, thereby realizing the function of adaptive pooling layer.

[0056] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0057] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0058] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0059] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

Claims

1. A handwritten digit recognition method based on memristors, characterized in that, The method includes the following steps: Receive input handwritten image data, perform convolution operation on the input handwritten image data to obtain feature map data, convert the feature map data into a multidimensional array mode, and label it as multidimensional feature data; The multidimensional feature data is divided into columns, and a weight coefficient is assigned to each multidimensional feature data. The multidimensional feature data of the corresponding column is multiplied and added with the weight coefficient of the corresponding column to obtain the calculation results of all columns. The weight coefficients are the probabilities corresponding to each feature map data. They are assigned by column. Initially, all data in all columns have the same probability, which is 1 / the sum of the number of data in each column. The probability is updated by the SoftMax activation function. After a set number of iterations, the resulting weight coefficients represent the importance of the corresponding feature map data, thus achieving the effect of adaptive pooling. Multiply and add the calculation results of all columns with the multidimensional feature data of the corresponding columns to obtain an array of calculation results; Assign probabilities to the data in the calculation result array by column to obtain the probability coefficients corresponding to all data. Map the probability coefficients to the memristor array, input the feature map data into the memristor array, and obtain the output results. The output is returned to the computer via TIA, SA, and ADC circuits, thus enabling the function of handwritten digit image recognition.

2. The handwritten digit recognition method based on memristor according to claim 1, characterized in that, The feature map data is a three-dimensional tensor data obtained after feature extraction through convolution operations. The three-dimensional tensor data is divided into "row" dimension, "column" dimension and "depth" dimension according to different dimensional directions.

3. The handwritten digit recognition method based on memristor according to claim 1, characterized in that, The memristor array has multiple conductance states between high and low resistance states, which can replace the cache in a computer to store floating-point data in a single cell, thereby reducing computer power consumption.

4. The handwritten digit recognition method based on memristor according to claim 3, characterized in that, The memristor array is responsible for storing the adaptive pooling coefficient matrix corresponding to the feature map data. That is, it assigns corresponding coefficients to all input feature map data, and then outputs current through the memristor array, which is converted by the circuit and returned to the computer to realize the function of adaptive pooling.

5. The handwritten digit recognition method based on memristor according to claim 4, characterized in that, Parallel acceleration is achieved by using memristor arrays for parallel computing, and inference data is output in parallel and then returned to the computer. The inference process can significantly reduce training power consumption and improve training efficiency.

6. The handwritten digit recognition method based on memristor according to claim 1, characterized in that, The sum of the weighting coefficients for each column of devices is 1.

7. The handwritten digit recognition method based on memristor according to claim 1, characterized in that, The process of assigning a weight coefficient to each multidimensional feature data is updated using the idea of ​​cosine similarity.

8. The handwritten digit recognition method based on memristor according to claim 1, characterized in that, When feature map data of different sizes are multiplied with the same weight coefficient column, the results are different. Based on the matrix multiplication and addition of the calculation results of all columns, the corresponding multidimensional feature data is multiplied and added again. The larger the calculation result of the corresponding column, the closer the feature map data is to the calculation result of the corresponding column. Therefore, a larger weight coefficient needs to be assigned.

9. A handwritten digit recognition system based on memristors, characterized in that, include: Convolution module: It is used to receive input handwritten digit image data, perform convolution operation on the input handwritten image data to obtain feature map data, convert the feature map data into a multidimensional array mode, and label it as multidimensional feature data; Data partitioning module: This module is used to partition multidimensional feature data by column, assign a weight coefficient to each multidimensional feature data, and multiply and add the multidimensional feature data of the corresponding column with the weight coefficient of the corresponding column to obtain the calculation results of all columns. The weight coefficients are the probabilities corresponding to each feature map data. They are assigned by column. Initially, all data in all columns have the same probability, which is 1 / the sum of the number of data in each column. The probability is updated by the SoftMax activation function. After a set number of iterations, the resulting weight coefficients represent the importance of the corresponding feature map data, thus achieving the effect of adaptive pooling. The multiply-add module is used to multiply and add the calculation results of all columns with the multidimensional feature data of the corresponding columns to obtain an array of calculation results; The probability allocation module is used to assign probabilities to the data in the calculation result array by column, obtain the probability coefficients corresponding to all data, map the probability coefficients to the memristor array, input the feature map data into the memristor array, and obtain the output results. Adaptive module: Used to return the output results to the computer through TIA, SA, and ADC circuits, thereby realizing the function of handwritten digit image recognition.

Images