Emotion recognition method for realizing SE-ConvNeXt network based on memristor

Abstract

The invention belongs to the technical field of neural networks and emotion recognition, and particularly relates to an emotion recognition method for realizing an SE-ConvNeXt network based on a memristor, according to the scheme, a lightweight ConvNeXt model is taken as a substrate, a channel attention module (SE-Block) is fused, an emotion recognition network model based on the SE-ConvNeXt network is constructed, and in the model, a channel attention module (SE-Block) is fused, so that the emotion recognition efficiency is improved. SE-Block dynamically strengthens key emotion representation through feature re-calibration, ConvNeXt effectively avoids the problem of gradient disappearance or gradient explosion of a deep network, and significantly improves the emotion feature discrimination ability; meanwhile, the scheme innovatively combines a convolution kernel priority algorithm (CKF) to optimize an SE-ConvNeXt network model circuit framework realized based on a memristor array, through cross-kernel weight mapping and dynamic calculation path integration, the problem that an invalid calculation flow is not eliminated after intra-kernel pruning of a traditional convolutional neural network circuit is solved, the convolution calculation energy consumption is reduced, and the calculation efficiency is improved. While high-precision emotion recognition is maintained, the overall operation processing efficiency of the SE-ConvNeXt network model circuit is improved.

Description

Technical Field

[0001] This invention belongs to the field of neural network and emotion recognition technology, and in particular relates to an emotion recognition method based on memristor-based SE-ConvNeXt network. Background Technology

[0002] Emotion recognition technology, as an important research area in computer vision, has attracted widespread attention from researchers both domestically and internationally in human-computer interaction applications, including fatigue driving detection, real-time facial expression recognition on mobile devices, and social robots. Facial emotion recognition primarily involves extracting static facial expression features from facial images to analyze and identify a person's emotional state. Deep learning, a research field that has garnered significant attention in recent years, combines multiple abstract data processing layers to form a computational network model, enabling machines to autonomously learn the features of data samples and achieve large-scale data classification and recognition. Therefore, emotion recognition technology incorporating deep learning network models has been widely applied.

[0003] Currently, Convolutional Neural Networks (CNNs) are a commonly used deep learning network model in emotion recognition applications. Their basic architecture mainly includes convolutional layers, pooling layers, and fully connected layers. Most of the computation in CNNs involves convolution operations. For current computing systems, especially in CNNs, the convolutional and pooling layers involve a large amount of intermediate computational data that needs to be stored and then retrieved for computation. This significant data transfer between the processor and memory leads to substantial power consumption and time consumption, a phenomenon known as the von Neumann bottleneck.

[0004] ConvNeXt, an efficient CNN variant proposed in 2022, emerged against the backdrop of slow innovation in CNN architecture since ResNet in 2017 and the rise of Vision Transformer (ViT) in 2020. Its appearance reshaped the competitiveness of CNNs in the field of computer vision. However, lightweight convolutional neural network models like ConvNeXt suffer from insufficient discriminative power when applied to complex image feature recognition, resulting in poor classification accuracy for emotion recognition and less than ideal application performance.

[0005] On the other hand, memristor-based neural networks have seen rapid development in circuit implementation. In 1990, Carver Mead first systematically described the idea of ​​embedding computation into local neurons to achieve a brain-like computing architecture that integrates in-memory computing. Using memristors as neuronal synapses in neuromorphic computing systems was considered the most promising approach to realizing this architecture. In 2017, memristors were applied to simulate the diffusion dynamics of biological synapses to construct spiking neural networks. In 2020, a team from Tsinghua University implemented an 8-bit 1Mb memristor array CNN, achieving 100 times higher energy efficiency than GPUs on the ImageNet task, marking the advent of large-scale in-memory computing arrays. With the increasing maturity of memristors, many neural networks can now be implemented using memristor arrays. However, further research is needed to optimize the circuit structure of memristor arrays and improve their processing efficiency for neural network operations.

[0006] Therefore, how to break through the bottleneck of the von Neumann architecture and achieve accurate and efficient emotion recognition has become a research hotspot in the field of human-computer interaction. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides an emotion recognition method based on a memristor-based SE-ConvNeXt network. This method improves the accuracy of emotion recognition by using an enhanced SE-ConvNeXt network, solving the problem of poor emotion recognition accuracy in lightweight convolutional neural networks. Furthermore, it optimizes the circuit architecture of the memristor-based SE-ConvNeXt network model by incorporating a kernel-first (CKF) algorithm, thus addressing the computational efficiency issue of the SE-ConvNeXt network model circuit and achieving accurate and efficient emotion recognition.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] A method for emotion recognition based on memristors and SE-ConvNeXt networks includes the following steps:

[0010] S1. Construct an emotion recognition network model based on the SE-ConvNeXt network, and train the emotion recognition network model to obtain the trained emotion recognition network model.

[0011] S2. Design a circuit structure based on a memristor array for the trained emotion recognition network model to obtain the emotion recognition network circuit structure.

[0012] S3. Obtain the face image to be identified and perform preprocessing;

[0013] S4. Convert the preprocessed face image into an analog voltage signal matrix using a digital-to-analog converter;

[0014] S5. Input the analog voltage signal matrix into the circuit structure to perform emotion recognition and output the probability distribution of the emotion category of the corresponding face image.

[0015] As a preferred embodiment, the emotion recognition network model includes an initial layer, an attention mechanism deep residual network layer, and a classification head, which are cascaded in sequence. The initial layer is used to perform feature projection transformation on the input face image to obtain an initial feature map. The attention mechanism deep residual network layer is used to extract channel attention features from the initial feature map to obtain a channel attention feature map. The classification head is used to perform emotion classification and recognition processing on the channel attention feature map to obtain the probability distribution of the emotion category of the face image.

[0016] As a preferred embodiment, in the emotion recognition network model:

[0017] The initial layer includes a cascaded convolutional layer and a layer normalization layer; the convolutional layer includes several convolutional modules that perform convolution operations using a kernel-first algorithm, and the convolutional layer is used to perform downsampling transformation processing on the input image to obtain a downsampled feature map; the layer normalization layer is used to normalize the downsampled feature map in the channel dimension to obtain the initial feature map.

[0018] The attention mechanism deep residual network layer comprises four cascaded stage layers, each stage layer consisting of several cascaded attention mechanism deep residual blocks. Each attention mechanism deep residual block includes an attention branch and a residual branch. The attention branch includes a cascaded deep separable convolutional module, a channel attention module, a layer normalization layer, an activation layer, and two point convolutional modules. The output of the deep separable convolutional module serves as the input to the channel attention module. The output of the channel attention module is multiplied by the output of the deep separable convolutional module and then transmitted to the layer normalization layer. After passing through the activation layer and the two point convolutional modules, the output is processed sequentially. The residual branch includes a point convolutional module. The output of the point convolutional module in the residual branch is added to the output of the attention branch to serve as the overall output of the attention mechanism deep residual block.

[0019] The classification head includes a cascaded global average pooling layer and a fully connected layer. The global average pooling layer is used to reduce the dimensionality and normalize the input feature map, and then the fully connected layer performs fully connected and Softmax function activation to obtain the probability distribution prediction result of the emotion category of the face image.

[0020] As a preferred embodiment, the channel attention module includes a globally average pooling module, a first fully connected layer, a ReLU activation function module, a second fully connected layer, and a Sigmoid activation function module, which are cascaded in sequence. The globally average pooling module is used to perform feature compression processing on the input feature map to obtain feature tensor data. The first fully connected layer is used to perform dimensionality reduction processing on the feature tensor data, and then perform ReLU activation operation through the ReLU activation function module. After dimensionality increase processing through the second fully connected layer, it is then subjected to Sigmoid activation operation through the Sigmoid activation function module to obtain the channel weight tensor, which serves as the overall output of the channel attention module.

[0021] As a preferred embodiment, in step S2, the circuit structure of the emotion recognition network obtained by implementing the emotion recognition network model through circuitry includes an input storage circuit, an attention residual operation and processing circuit, and an output storage circuit.

[0022] The input storage circuit includes a number of memristor arrays equal to the number of channels of the input tensor of the attention mechanism deep residual block. Each memristor array stores the voltage pulse amplitude matrix corresponding to the input tensor of a single channel. Each memristor array includes several image storage columns, and each image storage column includes several input-side memristor storage units connected in parallel. The number of image storage columns and the number of input-side memristor storage units connected in parallel in each image storage column are the same as the number of columns and rows of the input tensor of a single channel input to the attention mechanism deep residual block. Each input-side memristor storage unit includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store the voltage pulse amplitude at the corresponding pixel position in the input tensor of a single channel.

[0023] In the attention residual operation processing circuit, the output of each image storage column of each memristor array in the input storage circuit is output to the attention branch circuit and the residual branch circuit respectively through parallel branches. In the attention branch circuit, after being processed by the depthwise separable convolution circuit and the channel attention module circuit in sequence, the output of the channel attention module circuit is multiplied by the output of the depthwise separable convolution circuit through the multiplier circuit, and then passed through the layer normalization circuit, the ReLU activation circuit and two point convolution circuits in sequence. Finally, it is added in parallel with the output processed by the point convolution circuit in the residual branch circuit to obtain the channel attention feature tensor for output.

[0024] In the output storage circuit, two convolutional storage branches are connected in parallel for each column output of the channel attention feature tensor: a positive convolutional storage branch and a negative convolutional storage branch. Each positive convolutional storage branch includes a number of positive memristor storage cells connected in parallel, the same number as the number of rows of the channel attention feature tensor, followed by a bias storage cell connected in parallel. Each positive memristor storage cell includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store positive channel attention feature tensor element values. The bias storage cell also includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store... The negative convolution storage branch includes a number of negative memristor storage cells connected in parallel, the same number as the number of rows of the channel attention feature tensor, and then connected in series with an inverter conversion circuit. Each negative memristor storage cell includes a 1T1R device composed of a memristor and a thyristor connected in series, used to store negative channel attention feature tensor element values. The inverter conversion circuit is used to convert the output of the negative memristor storage cell into a negative value. The outputs of the positive and negative convolution storage branches corresponding to each column output are connected in parallel and added together, then passed through a diode circuit representing the ReLU activation function before being output.

[0025] As a preferred embodiment, the channel attention module circuit in the attention residual calculation processing circuit includes a global average pooling circuit, a first-layer fully connected circuit, a ReLU activation function circuit, a second-layer fully connected circuit, and a Sigmoid activation function circuit connected in sequence. The ReLU activation function circuit is located on the row output side of the first-layer fully connected circuit. After conversion by a current-to-voltage conversion circuit, the converted output voltage is output to the row input side of the second-layer fully connected circuit. The Sigmoid activation function circuit is located on the column output side of the second-layer fully connected circuit, and its output serves as the total output of the channel attention module circuit.

[0026] As a preferred option, in step S3, the circuit structure of the emotion recognition network obtained by implementing the emotion recognition network model through circuitry includes an input part, a convolution kernel operation part, and an output part.

[0027] The input section includes a number of memristor arrays equal to the number of channels in the input tensor of the convolution module. Each memristor array stores the voltage pulse amplitude matrix corresponding to the input tensor of a single channel. Each memristor array includes several image storage columns, and each image storage column includes several input memristor storage units connected in parallel. The number of image storage columns and the number of input memristor storage units connected in parallel in each image storage column are the same as the number of columns and rows of the input tensor of a single channel input to the convolution module. Each input memristor storage unit includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store the voltage pulse amplitude at the corresponding pixel position in the input tensor of a single channel.

[0028] In the convolution kernel operation section, a convolution kernel operation path is provided for each image storage column output by the memristor array in the input section. Each convolution kernel operation path includes a first voltage-to-current conversion circuit. The converted voltage output terminal of the first voltage-to-current conversion circuit is connected in parallel with two convolution processing branches. Each convolution processing branch includes a memristor, a first thyristor, a second voltage-to-current conversion circuit, and a second thyristor connected in series. In the convolution processing branch, the conductance value of the memristor is programmed to represent the convolution weight value, and its output current value is the voltage input value representing the input tensor pixel value and the convolution weight value. The first thyristor controls whether the convolution processing branch is turned on, and the second voltage-to-current conversion circuit converts the output current value of the memristor into a voltage value. The second thyristor controls whether the convolution processing branch participates in the convolution step size calculation. During operation, the two convolution processing branches are alternately turned on by the first thyristor. When one convolution processing branch is turned on and participates in the convolution operation, the other convolution processing branch programs the conductance value of the memristor. The output terminals of each convolution kernel operation path are connected in parallel according to the preset convolution step size setting relationship to obtain a convolution tensor of preset size for output.

[0029] The output section corresponds to two convolution storage branches connected in parallel for each column output of the convolution tensor: a positive convolution storage branch and a negative convolution storage branch. Each positive convolution storage branch includes a number of positive memristor storage cells connected in parallel, the same number as the number of rows of the convolution tensor, and then a bias storage cell connected in parallel. Each positive memristor storage cell includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store positive convolution tensor element values. The bias storage cell also includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store bias values. The negative convolution storage branch includes a number of negative memristor storage cells connected in parallel, the same number as the number of rows of the convolution tensor, and then connected in series with an inverter conversion circuit. Each negative memristor storage cell includes a 1T1R device composed of a memristor and a thyristor connected in series, used to store the absolute values ​​of negative convolution tensor elements. The inverter conversion circuit is used to convert the output of the negative memristor storage cell into a negative value. The outputs of the positive and negative convolution storage branches corresponding to each column output are connected in parallel and added together, then passed through a diode circuit representing the ReLU activation function before being output.

[0030] As a preferred embodiment, in the convolution kernel operation section of the convolution module, when the convolution processing branch programs the conductance value of the memristor, it only programs and sets the corresponding conductance value of the memristor when the convolution weight value is not 0; otherwise, it skips the conductance value programming and setting process and sets the first thyristor in the corresponding convolution processing branch to be disconnected.

[0031] As a preferred embodiment, the global average pooling circuit includes a number of pooling channels equal to the number of channels in the input tensor of the global average pooling circuit; each pooling channel includes a memristor array for storing the voltage pulse amplitude matrix corresponding to the input tensor of a single channel, and a current-to-voltage conversion circuit and a pooling storage memristor connected in series with the output terminal of the memristor array; the memristor array includes several image storage columns, each image storage column including several pooling memristor storage units connected in parallel, the number of image storage columns and the number of pooling memristor storage units connected in parallel in each image storage column being the same as the number of columns and rows of the input tensor of a single channel input to the global average pooling circuit; each pooling memristor storage unit includes a 1T1R device composed of a memristor and a thyristor connected in series, used to store the voltage pulse amplitude at the corresponding pixel position in the input tensor of a single channel; the image storage columns of the memristor array are added in parallel, and then connected in series with the current-to-voltage conversion circuit and the pooling storage memristor.

[0032] As a preferred embodiment, in step S3, the preprocessing method for the face image is as follows: after processing the face image into a grayscale image, it is converted into a face image tensor of a preset size; in step S4, the method for converting the preprocessed face image into an analog voltage signal is as follows: the grayscale value of each pixel position in the face image tensor is converted into a corresponding voltage pulse amplitude value, thereby obtaining the voltage pulse amplitude matrix corresponding to the face image tensor.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention employs an emotion recognition network model based on the SE-ConvNeXt network to achieve emotion recognition. Its model architecture is based on the lightweight ConvNeXt model. To address the issue of insufficient feature discrimination power, it further integrates the channel attention module (SE-Block) and the ConvNeXt Block. The former uses feature recalibration to dynamically enhance key emotional representations (such as periorbital micro-expressions), while the latter eliminates the gradient problem in deep networks through cross-layer identity mapping. The emotion recognition network model formed by the combination of the two, based on the SE-ConvNeXt network architecture, retains the performance advantages of ConvNeXt compared to Transformer while significantly improving the fine-grained feature capture capability, thereby improving the accuracy of emotion feature recognition.

[0035] 2. In this invention, the kernel-first algorithm (CKF) is innovatively combined to optimize the circuit architecture of the SE-ConvNeXt network model based on memristor array implementation. By integrating cross-kernel weight mapping and dynamic computation path, the pain point of invalid computation flow not being eliminated after traditional kernel pruning is solved. The circuit design optimization of pruning is equivalent to reduction of computation, which can effectively reduce the energy consumption of convolution computation. While maintaining high-precision emotion recognition, it can effectively improve the overall computational efficiency of the SE-ConvNeXt network model circuit. Attached Figure Description

[0036] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0037] Figure 1 This is a schematic diagram of the architecture of the emotion recognition network model based on the SE-ConvNeXt network in the present invention.

[0038] Figure 2 A flowchart illustrating an application scenario for emotion recognition using the emotion recognition network model of this invention.

[0039] Figure 3 This is a schematic diagram of the CKF algorithm.

[0040] Figure 4A schematic diagram of a single SE-ConvNeXt Block structure.

[0041] Figure 5 A schematic diagram illustrating the process of generating channel weight tensors for the SE Block.

[0042] Figure 6 This is a circuit diagram of a convolution module based on memristors and the CKF algorithm.

[0043] Figure 7 This is a circuit diagram for programming data into a memristor.

[0044] Figure 8 This is the circuit structure diagram of the global average pooling module.

[0045] Figure 9 This is a circuit structure diagram of the SE-ConvNeXt Block.

[0046] Figure 10 This is the circuit structure diagram of the fully connected layer in the SE Block.

[0047] Figure 11 This is a schematic diagram of the confusion matrix of the model on the test set before and after pruning in the embodiment.

[0048] Figure 12 The image shows a histogram of the frequency distribution of weights in all convolutional layers in the pruned model of this embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0050] To address the problems of existing technologies, this invention proposes an integrated emotion recognition architecture solution based on a memristor-based SE-ConvNeXt network. This solution uses a lightweight ConvNeXt model as a base, integrating a channel attention module (SE-Block) to construct an emotion recognition network model based on the SE-ConvNeXt network. In this model, SE-Block dynamically strengthens key emotion representations through feature recalibration, while ConvNeXt effectively avoids the gradient vanishing or exploding problems of deep networks, significantly improving the discriminative power of emotion features. Simultaneously, this solution innovatively combines the kernel-first algorithm (CKF) to optimize the circuit architecture of the SE-ConvNeXt network model implemented using memristors. Through cross-kernel weight mapping and dynamic computation path integration, it solves the problem of invalid computational flow not being eliminated after kernel pruning in traditional convolutional neural network circuits, reducing the energy consumption of convolutional computation. While maintaining high-precision emotion recognition, it improves the overall computational efficiency of the SE-ConvNeXt network model circuit.

[0051] Specifically, the emotion recognition method based on memristor-based SE-ConvNeXt network proposed in this invention includes the following steps:

[0052] S1. Construct an emotion recognition network model based on the SE-ConvNeXt network, and train the emotion recognition network model to obtain the trained emotion recognition network model.

[0053] S2. Design a circuit structure based on a memristor array for the trained emotion recognition network model to obtain the emotion recognition network circuit structure.

[0054] S3. Obtain the face image to be identified and perform preprocessing;

[0055] S4. Convert the preprocessed face image into an analog voltage signal matrix using a digital-to-analog converter;

[0056] S5. Input the analog voltage signal matrix into the circuit structure to perform emotion recognition and output the probability distribution of the emotion category of the corresponding face image.

[0057] As can be seen, the key technical aspects of this invention lie primarily in two areas: the construction of an emotion recognition network model based on the SE-ConvNeXt network and the implementation of the circuitry for the SE-ConvNeXt network model. These key technical aspects will be elaborated upon below.

[0058] 1. Construction of an emotion recognition network model based on SE-ConvNeXt network

[0059] Figure 1An example of the overall architecture of an emotion recognition network model based on the SE-ConvNeXt network is shown. Figure 1 As shown, the emotion recognition network model includes a cascaded initial layer (Stem Layer), an attention mechanism deep residual network layer (SE-Res Net Layer), and a classification head (Classification Layer). The initial layer is used to perform feature projection transformation on the input face image to obtain an initial feature map. The attention mechanism deep residual network layer is used to extract channel attention features from the initial feature map to obtain a channel attention feature map. The classification head is used to perform emotion classification and recognition processing on the channel attention feature map to obtain the probability distribution of the emotion category of the face image.

[0060] The emotion recognition network model proposed in this invention, based on the SE-ConvNeXt network, uses the lightweight ConvNeXt model as a foundation. To address the issue of insufficient feature discrimination power, it further integrates the channel attention module (SE-Block) and the ConvNeXt Block. The former uses feature recalibration to dynamically enhance key emotional representations (such as periorbital micro-expressions), while the latter eliminates gradient problems in deep networks through cross-layer identity mapping. The resulting SE-ConvNeXt network architecture emotion recognition network model retains the performance advantages of ConvNeXt compared to the Transformer while significantly improving fine-grained feature capture capabilities, thereby enhancing the accuracy of emotion feature recognition.

[0061] In practical applications, for broadly acquired visual images containing faces, there are many mature solutions available for face detection. Considering the favorable data acquisition environment of emotion recognition systems, the classic HOG (Histogram of Oriented Gradient) detector can be used to detect faces in the visual images and output face images in a specified format. Then, this face image is normalized to a 48×48 grayscale image through scaling and channel manipulation. After upsampling and channel duplication, the image is transformed into a 224×224×3 face image tensor, which can then be input into the emotion recognition network model for emotion recognition and classification. The example process for this application scenario is as follows: Figure 2 As shown.

[0062] 1.1 Initial Layer

[0063] The initial layer (Stem Layer) of the SE-ConvNeXt model simulates the idea of ​​segmenting an image into blocks and projecting it linearly. Its structure is simple and efficient, consisting of only one cascaded convolutional layer and one layer normalization layer. The convolutional layer comprises several convolutional modules that perform convolution operations using the Convolutional Kernel First (CKF) algorithm. The convolutional layer is used to downsample the input image to obtain a downsampled feature map. The layer normalization layer normalizes the downsampled feature map along the channel dimension to obtain the initial feature map.

[0064] After preprocessing facial image data and inputting it into the emotion recognition network model, it first enters the convolutional layer in the initial layer. In specific implementations, the convolutional layer can be designed as 3×96 4×4 convolutional modules (3 depends on the number of input channels, 96 depends on the number of output channels), with a convolution stride of 4. This convolutional layer structure aims to achieve non-overlapping image padding and one-step 4x downsampling, effectively avoiding redundant computations in traditional convolution while significantly compressing information. All convolutional module operations in the convolutional layers and the entire network model are implemented using the kernel-first (CKF) algorithm. Traditional convolution operations are implemented using matrix multiplication, which has the limitation that pruning network parameters does not actually reduce the computational load. This is because even if some convolutional kernel weights are set to 0, 0 weights will still participate in the multiplication and accumulation operations during matrix multiplication, thus not reducing the computational load. This presents the problem of "pruning without reducing computation." The CFK algorithm extracts each convolution kernel weight element and multiplies it with the corresponding input value. The design of this invention, during the traversal of each convolution kernel weight element, skips the multiplication and accumulation operation for that element when it is 0, only calculating the non-zero weight elements. This effectively reduces the computational load after pruning, achieving "pruning quantity equals pruning calculation" circuit design optimization. Furthermore, its corresponding memristor-based circuit structure can adapt to any pruned result. Figure 3 The process of demonstrating the principle of the CKF algorithm is shown, and its calculation expression can be represented as:

[0065]

[0066] In formula (1), I and Y are the input matrix (n+2p)×(n+2p) and n, respectively. o ×n o The output matrix is ​​denoted by , where n is the size of the original input matrix, p is the padding size of the original input feature map; K is an m×m convolution kernel, m is the kernel size, K[i,j] is the weight value corresponding to the [i,j]th pixel position in the kernel, s is the convolution stride, and the slice I[r0:r end :s,c0:cend [s] indicates starting from the element corresponding to index (r0, c0) in matrix I, moving with a step size s, and taking n elements corresponding to the indexes. o ×n o The matrix. Formula (2) shows the size n of the output matrix. o The calculation formula.

[0067] After downsampling the image data through the convolution operation of the convolutional layer, in order to maintain consistency with the transformed data, a layer normalization layer is used to normalize the downsampled feature map of each sample in the channel dimension to obtain the initial feature map, which serves as the overall output of the initial layer.

[0068] 1.2 Attention Mechanism in Deep Residual Network Layers (SE-Res Net)

[0069] The attention mechanism deep residual network layer SE-Res Net is a core component of the emotion recognition network model based on the SE-ConvNeXt network. It contains four stage layers, each of which is mainly composed of several attention mechanism deep residual blocks (SE-ConvNeXt Blocks) cascaded together. Figure 4 The structure of a single attention mechanism deep residual block (SE-ConvNeXtBlock) is shown, including an attention branch and a residual branch. The attention branch consists of a cascaded deep separable convolutional module (dwConv), a channel attention module (SE-block), a layer normalization layer (LayerNorm), an activation layer (ReLU), and two point convolutional modules (pwConv). The output of the deep separable convolutional module (dwConv) serves as the input to the channel attention module (SE-block). The output of the channel attention module (SE-block) is multiplied by the output of the deep separable convolutional module (dwConv) and then passed to the layer normalization layer (LayerNorm). After passing through the activation layer (ReLU) and the two point convolutional modules (pwConv), the output is processed sequentially. The residual branch includes a point convolutional module (pwConv). The output of the point convolutional module in the residual branch is added to the output of the attention branch, and this sum serves as the overall output of the attention mechanism deep residual block (SE-ConvNeXt Block).

[0070] In the attention branch, the input feature map is convolved with a large receptive field through a depthwise separable convolutional layer. After each channel is convolved individually, its output is processed by the channel attention module (SE-block) and then normalized. After being expanded by 4 times by the first point convolutional layer, it is compressed back to its original size by another point convolutional layer, forming an inverse bottleneck structure. This structure can effectively learn discriminative features and maintain spatial consistency.

[0071] from Figure 4 As can be seen, the channel attention module (SE-block) includes a globally average pooling module, a first fully connected layer, a ReLU activation function module, a second fully connected layer, and a sigmoid activation function module, which are cascaded in sequence. The globally average pooling module is used to compress the input feature map to obtain feature tensor data. The first fully connected layer is used to reduce the dimensionality of the feature tensor data, and then the ReLU activation function module performs ReLU activation operation. After the second fully connected layer performs dimensionality increase, the sigmoid activation function module performs sigmoid activation operation to obtain the channel weight tensor, which is the overall output of the channel attention module (SE-block).

[0072] Figure 5 This demonstrates the process of the channel attention module (SE-block) generating channel weight tensors. The SE-block process can be divided into two parts: squeezing and excitation. Squeeze compresses the input feature map using a global average pooling module, resulting in a 1×1×C tensor, representing the global distribution of responses across the feature channels. Excitation adaptively calibrates the channel dimensions of the resulting tensor through two fully connected layers, generating channel weight tensors for each channel. The first fully connected layer reduces the channel dimension C to C / r, where r is the compression ratio. The second fully connected layer increases the dimension back to C, ensuring the output channel weight tensor dimension remains consistent with the original input. Multiplying the output F(x) of the depthwise separable convolution module (dwConv) with the channel weight tensor processed by the SE-block yields the channel attention-weighted feature tensor.

[0073] The global average pooling operation in the channel attention module (SE-block) can also be implemented using convolution operations in actual circuit implementations. Therefore, pooling is, in a sense, a special type of convolution. From the perspective of convolution operations, the difference between pooling layers and traditional convolutional layers in convolutional neural networks lies in two aspects. First, the number of input channels and output channels in a traditional convolutional layer are not directly related. The output channel j is obtained by summing the results of the two-dimensional convolution kernel corresponding to the two-dimensional input channels. However, in pooling, the input and output channels are in one-to-one correspondence, i.e., depthwise separable convolution, which is equivalent to performing convolution on each channel individually. Second, the pooling convolution kernel has a specific form. For average pooling, the element value in an m×m convolution kernel is 1 / m. 2 For max pooling, the convolution kernel is a Boolean matrix with only one element equal to 1 and the rest equal to 0. It's worth noting that if global average pooling is implemented using convolution, the kernel value needs to be determined based on the input tensor. Specifically, when global average pooling is implemented using convolution, the kernel's dimensions are equal to the input tensor's dimensions. In summary, convolution and pooling are essentially the same. Therefore, the CKF algorithm is applicable not only to convolution modules but also to global average pooling modules.

[0074] The deep residual processing in the attention mechanism's deep residual block (SE-ConvNeXt Block) is mainly performed through depthwise separable convolutional modules and pointwise convolutional modules. It also features a shortcut connecting the input to the output, the residual branch. The original input feature map can be directly added to the output through this shortcut, thus avoiding the vanishing or exploding gradient problems caused by an excessively deep neural network during gradient descent. It's important to note that due to the tensor addition operation, dimensionality matching is required. If the dimensions of the original input feature map and the output feature map are inconsistent, the residual branch needs to normalize the original input using a 1×1×C pointwise convolutional layer.

[0075] Because of the residual connections, it is necessary to ensure the consistency of the height and width dimensions of the input and output during the residual processing. Therefore, the height and width dimensions of the data will not change after passing through the three convolutional layers of residual processing. It is not difficult to deduce from formula (2) that when p = m / 2 and s = 1, n o=n always holds true. In a depthwise separable convolutional layer, the kernel size m = 7, and the padding size p = 3. Therefore, as long as the padding size is 3, the input and output sizes can be kept consistent. In the residual branch, the point convolutional module itself is a 1×1 kernel and does not change the height and width. In the attention branch, the two point convolutional modules first perform channel expansion and then channel restoration, which also ensures that the channel size remains unchanged. Therefore, if the dimensions of the original input feature map and the output feature map are consistent, point convolution processing can be omitted in the residual branch. Of course, performing point convolution processing does not affect the channel size.

[0076] 1.3 Classification Layer

[0077] The classification layer consists of a cascaded global average pooling layer and a fully connected layer. The global average pooling layer is used to reduce the dimensionality and normalize the input feature map, and then the fully connected layer performs fully connected and Softmax activation to obtain the probability distribution prediction of the emotion category of the face image.

[0078] In the classification head, the input feature map is reduced in dimensionality and normalized after passing through an average pooling layer. To prevent overfitting, 50% of the neurons are randomly discarded before being fed into a fully connected layer. Finally, the output is mapped to the probability space using the Softmax function, outputting the probability distribution of each input image belonging to different emotion categories. This not only outputs more detailed results, but can also be directly applied to the cross-entropy loss function to calculate the cross-entropy loss between the predicted value and the label.

[0079] 1.4 Training of the Emotion Recognition Network Model

[0080] The training method for the emotion recognition network model based on the SE-ConvNeXt network is basically the same as that for the training method of the ConvNeXt network model. Specifically, in the scheme of this invention, during model training, face images that have been pre-labeled with emotion type labels can be used as training samples to form a training sample set, which is input into the emotion recognition network model. The emotion recognition network model processes the training samples to obtain the emotion category recognition prediction results (usually, the emotion category with the highest probability distribution is taken as the emotion category recognition prediction result). Based on the emotion category recognition prediction results of the training samples and the original labeled emotion type labels, the cross-entropy loss of the training is calculated, and the parameters of the emotion recognition network model are optimized inversely based on the cross-entropy loss until the emotion recognition network model converges, thus completing the training process of the emotion recognition network model.

[0081] Implementation of the 2SE-ConvNeXt network model circuit

[0082] The following section details the circuit design and implementation of a memristor array-based emotion recognition network model using the SE-ConvNeXt network. This includes explanations of the convolutional module, global average pooling module, and SE-ConvNeXtBlock circuit design. In addition, the section will describe the circuit implementation of some detailed functions, such as the memristor resistance value programming circuit, the ReLU activation function, and the equivalent circuit of the Sigmoid activation function.

[0083] 2.1 Implementation of the Convolution Module Circuit Structure

[0084] To improve the overall processing efficiency of the emotion recognition network model, the convolutional modules used in the emotion recognition network model of this invention, including convolutional layers, depthwise separable convolutional modules (composed of a combination of channel convolutional modules and point convolutional modules), and the convolutional module units used in the point convolutional modules, are all implemented using the kernel-first (CKF) algorithm. Taking the convolutional module of the convolutional layer in the initial layer (Stem Layer) as an example, Figure 6 The detailed circuit design of a convolution module based on memristors and the CKF algorithm is shown. Figure 6 As can be seen from the example, the circuit structure of each convolution module includes an input part, a kernel operation part, and an output part.

[0085] The input part includes a number of memristor arrays equal to the number of channels in the input tensor of the convolution module. Each memristor array stores the voltage pulse amplitude matrix corresponding to the input tensor of a single channel. Each memristor array includes several image storage columns, and each image storage column includes several input memristor storage units connected in parallel. The number of image storage columns and the number of input memristor storage units connected in parallel in each image storage column are the same as the number of columns and rows of the input tensor of a single channel input to the convolution module. Each input memristor storage unit includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store the voltage pulse amplitude at the corresponding pixel position in the input tensor of a single channel.

[0086] In the kernel operation section (Kernel Part + Calculation Part), a convolution kernel operation path is provided for each image storage column output by the memristor array in the input section. Each convolution kernel operation path includes a first voltage-to-current converter (Converter). The converted voltage output terminal of the first voltage-to-current converter (Converter) is connected in parallel with two convolution processing branches. Each convolution processing branch includes a memristor, a first thyristor (GTO), a second voltage-to-current converter (Converter), and a second thyristor (GTO) connected in series. In the convolution processing branch, the conductance value of the memristor is programmed to represent the convolution weight value, and its output current value represents the input weight value. The product of the voltage input value of the tensor pixel value and the conductance value representing the convolution weight value, the first thyristor is used to control whether the convolution processing branch is turned on, the second voltage-to-current conversion circuit is used to convert the output current value of the memristor into a voltage value, and the second thyristor is used to control whether the convolution processing branch participates in the convolution step size calculation; during operation, the two convolution processing branches are alternately turned on by the first thyristor of one of them. When one convolution processing branch is turned on to participate in the convolution operation, the conductance value of the memristor is programmed and set in the other convolution processing branch; the output terminals of each convolution kernel operation path are connected in parallel according to the preset convolution step size setting relationship to obtain a convolution tensor of preset size for output.

[0087] The output part corresponds to each column output of the convolutional tensor with two parallel convolutional storage branches: a positive convolutional storage branch and a negative convolutional storage branch. Each positive convolutional storage branch includes a number of positive memristor storage cells connected in parallel, the same number as the number of rows of the convolutional tensor, followed by a bias storage cell connected in parallel. Each positive memristor storage cell includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store positive convolutional tensor element values. The bias storage cell also includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store bias values. The negative convolutional storage branch includes... The system comprises a number of negative memristor storage cells connected in parallel, the same number as the number of rows of the convolution tensor, and then connected in series with an inverter circuit. Each negative memristor storage cell includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store the absolute values ​​of negative convolution tensor elements. The inverter circuit is used to convert the output of the negative memristor storage cells into negative values. The outputs of the positive and negative convolution storage branches corresponding to each column output are connected in parallel and added together, then passed through a diode circuit representing the ReLU activation function before being output.

[0088] As can be seen from the circuit structure of the convolution module described above, the control of each part of the circuit is achieved by thyristors. In specific applications, gate turn-off thyristors (GTOs) can be used. The GTO can control not only the conduction of the device but also its turn-off through the gate signal. When the anode voltage is positive and a positive pulse current is applied to the gate, the GTO conducts. After conduction, the gate loses control over the current, and the current is determined by the voltage between the anode and cathode. To turn off the GTO, a negative voltage or negative current needs to be applied to the gate.

[0089] As for memristors, when used as data storage units (such as in the memristor array in the input section), their stored data comes from the input data of the previous layer and is programmed to import, so that the conductance value of the memristor corresponds to the input data. However, when used as a convolution kernel (such as in the kernel part of the convolution kernel operation), its conductance value needs to be pre-programmed as the convolution weight value, and then multiplied with the imported data. Figure 7 This demonstrates how to program data into a memristor (here, "data" specifically refers to the inputs and outputs of each layer or the weights of neurons in a neural network). The Feedback Write mechanism can be used to program the memristor. A comparator checks if the memristor's resistance value is within the target range. If the resistance value is not within the target range, the resistance value is gradually adjusted by changing the width of the voltage pulse (less than the width required for a complete switch). An XOR gate is used to check if the programming was successful. If unsuccessful, the resistance value is adjusted until it reaches the target range. The D-to-A converter converts the digital signal into an analog signal to control the memristor's resistance value. It is important to note that since a threshold memristor only operates in a resistive state under specific voltage conditions, the actual data needs to be mapped to the memristor's operating range using the mapping relationships shown in formulas (3) and (4) before being imported into the circuit for it to function correctly.

[0090]

[0091] Among them, W + W - These represent the positive and negative convolution weight values, respectively; σ + σ - These represent the positive convolution weight values ​​W. + and negative convolution weights W - The corresponding memristor conductance values; σ max σ min These represent the maximum and minimum conductance values ​​of the memristor, respectively; max|W| represents the maximum absolute value of the settable convolution weights.

[0092] exist Figure 6In the convolution module circuit example shown, the memristor array in the input part is an input tensor storage circuit. For example, if the input image is 224×224×3, the memristor array will be 224×224×3. After the input data is imported into the memristor array, the conductance value of the memristor corresponds to the input data. The control signal EL sequentially controls the memristors in rows 1 to n to participate in the operation, setting the input voltage of each column to 1V. After passing through the cross array, the current value of each column is equal to the conductance value of its corresponding memristor. The input data of each row is sequentially passed to the subsequent circuits in the form of current for multiplication and accumulation operations.

[0093] The data stored in the memristor array of the input part is passed to the convolution kernel operation part for convolution operations in the form of row-by-row sliding outputs. The current of each row of sliding outputs is converted into a voltage signal after passing through the first voltage-to-current converter and input to the kernel part of the convolution kernel operation part. In the convolution kernel operation part, each convolution kernel operation path contains two convolution processing branches, and the memristor of each convolution processing branch stores the value of a convolution kernel weight element. This embodies the core idea of ​​the CKF algorithm. In practical application, by traversing the convolution kernel, the convolution kernel elements are programmed one by one into a row of memristors, so that the conductance value of the memristor corresponds to the convolution weight value. Only when the convolution weight value is not 0 is the corresponding conductance value programmed and set for the memristor; otherwise, the conductance value programming and setting process is skipped, and the first thyristor in the corresponding convolution processing branch is turned off. Furthermore, during operation, the two convolution processing branches are alternately turned on and off via the first thyristor. When one convolution processing branch is on and participating in the convolution operation, the other convolution processing branch programs the conductance value of the memristor. This alternating operation of the two convolution processing branches improves the processing efficiency of the convolution module. When the memristor participates in the operation, since the programmed conductance value of the memristor represents the convolution weight value, its output current value is the product of the voltage input value representing the input tensor pixel value and the conductance value representing the convolution weight value, thus representing the convolution result.

[0094] In the Calculation Part of the convolution kernel operation, its function is to control the convolution stride operation between the convolution kernel elements and the input values ​​at specific positions. The principle is the operation rule corresponding to formula (1). The calculation of the number of control signals, conNum, at different positions is as shown in formula (5). Figure 6 In the example shown, the parameters input into this layer yield conNum = 4. In addition, each control signal controls n. o The column product is activated, resulting in 56 columns in this layer. The interval between each column is the convolution stride minus 1, i.e., a gap of 1 column.

[0095]

[0096] The output part is the storage module for the convolution results. Since some convolution kernel elements are positive and some are negative, and the memristors are programmed with the absolute values ​​of the convolution weights, it is necessary to determine the sign of the convolution kernel elements and, based on the Pos or Neg signal, store the positive and negative values ​​of the multiplication integral into two cross arrays of memristors in the positive convolution storage branch (P) and the negative convolution storage branch (N), respectively. The positive convolution storage branch (P) also contains a row of bias storage cells. The convolution results are output row by row, corresponding to the row-by-row input of each layer. During output, the vector of the negative convolution storage branch (N) is converted to a negative value by an inverter circuit, then added to the output vector of the positive convolution storage branch (P) and the bias value (Bias). The result is then output through a diode representing the ReLU function.

[0097] Formula (6) describes the mathematical expression of the ReLU activation function, and the current-voltage characteristic curve of the diode is similar to that of the ReLU function, which can be used for its hardware implementation.

[0098]

[0099] Figure 6 The diagram shows the circuit structure of a convolutional module. In practical applications, the number of convolutional module circuits required corresponds to the number of convolutional channels. For example, if the output channels of the convolutional layer are 96, then 96 different two-dimensional convolutional kernels are needed to convolve each input channel. The hardware implementation would then require 96 such kernels. Figure 6 The convolution module circuit shown.

[0100] Before being input into the emotion recognition network model, the face image needs to be preprocessed. The face image is processed into a grayscale image and then converted into a face image tensor of a preset size. Then, the preprocessed face image is converted into an analog voltage signal. That is, the grayscale value of each pixel in the face image tensor is converted into a corresponding voltage pulse amplitude, resulting in a voltage pulse amplitude matrix corresponding to the face image tensor. Then, the voltage pulse amplitude matrix corresponding to the face image tensor is input into the initial layer of the emotion recognition network model to enter the processing.

[0101] 2.2 Implementation of Global Average Pooling Circuit Structure

[0102] In the emotion recognition network model based on the SE-ConvNeXt network of this invention, all pooling layers used are global average pooling modules. Taking the global average pooling layer of the classification head as an example, the input data is 7×7×768 in size. Performing global average pooling on it is equivalent to performing a depthwise separable convolution on it using a special convolution kernel of size 7×7×768, where the value of each element of the convolution kernel is 1 / 49.

[0103] The structure of a single global average pooling module circuit is as follows: Figure 8 As shown, this represents the global average pooling module circuit for any m-th channel. The global average pooling circuit includes a number of pooling channels equal to the number of channels in the input tensor of the global average pooling circuit. Each pooling channel includes a memristor array for storing the voltage pulse amplitude matrix corresponding to the input tensor of a single channel, and a current-to-voltage conversion circuit and a pooling storage memristor connected in series with the output of the memristor array. The memristor array includes several image storage columns, each image storage column including several pooling memristor storage units connected in parallel. The number of image storage columns and the number of pooling memristor storage units connected in parallel in each image storage column are the same as the number of columns and rows of the input tensor of a single channel input to the global average pooling circuit. Each pooling memristor storage unit includes a 1T1R device composed of a memristor and a thyristor connected in series, used to store the voltage pulse amplitude at the corresponding pixel position in the input tensor of a single channel. The image storage columns of the memristor array are added in parallel and then connected in series with the current-to-voltage conversion circuit and the pooling storage memristor.

[0104] As can be seen, the structure of the global average pooling circuit is similar to... Figure 6 The structure of the convolution module circuit shown is somewhat similar, except that after the memristor array of the global average pooling circuit, after the current-to-voltage conversion circuit, only one memristor is connected as the pooling storage memristor. In this example, the stored convolution weight value is 1 / 49.

[0105] 2.3 Implementation of the Attention Mechanism Deep Residual Block (SE-ConvNeXt Block) Circuit Structure

[0106] The attention mechanism deep residual block (SE-ConvNeXt Block) can be considered as a combination of the SE Block and the ConvNeXt Block, and its circuit implementation structure is as follows: Figure 9 As shown, the circuitry for the deep residual block of the attention mechanism includes an input storage circuit, an attention residual operation and processing circuit, and an output storage circuit.

[0107] The input storage circuit includes a number of memristor arrays equal to the number of channels of the input tensor of the attention mechanism deep residual block. Each memristor array stores the voltage pulse amplitude matrix corresponding to the input tensor of a single channel. Each memristor array includes several image storage columns, and each image storage column includes several input-side memristor storage units connected in parallel. The number of image storage columns and the number of input-side memristor storage units connected in parallel in each image storage column are the same as the number of columns and rows of the input tensor of a single channel input to the attention mechanism deep residual block. Each input-side memristor storage unit includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store the voltage pulse amplitude at the corresponding pixel position in the input tensor of a single channel.

[0108] In the attention residual operation processing circuit, the output of each image storage column of each memristor array in the input storage circuit is output to the attention branch circuit and the residual branch circuit respectively through parallel branches. In the attention branch circuit, after being processed by the depthwise separable convolution circuit and the channel attention module circuit in sequence, the output of the channel attention module circuit is multiplied by the output of the depthwise separable convolution circuit through the multiplier circuit, and then passed through the layer normalization circuit, the ReLU activation circuit and two point convolution circuits in sequence. Finally, it is added in parallel with the output processed by the point convolution circuit in the residual branch circuit to obtain the channel attention feature tensor for output.

[0109] In the output storage circuit, two convolutional storage branches are connected in parallel for each column output of the channel attention feature tensor: a positive convolutional storage branch and a negative convolutional storage branch. Each positive convolutional storage branch includes a number of positive memristor storage cells connected in parallel, the same number as the number of rows of the channel attention feature tensor, followed by a bias storage cell connected in parallel. Each positive memristor storage cell includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store positive channel attention feature tensor element values. The bias storage cell also includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store... The bias value; the negative convolution storage branch includes a number of negative memristor storage cells connected in parallel, the same number as the number of rows of the channel attention feature tensor, and then connected in series with an inverter conversion circuit; each negative memristor storage cell includes a 1T1R device composed of a memristor and a thyristor connected in series, used to store negative channel attention feature tensor element values; the inverter conversion circuit is used to convert the output of the negative memristor storage cell into a negative value; the output terminals of the positive convolution storage branch and the negative convolution storage branch corresponding to each column output are connected in parallel and added together, then passed through a diode circuit representing the ReLU activation function before being output.

[0110] pass Figure 4The attention mechanism architecture shown is a deep residual block (SE-ConvNeXt Block). As a whole, the SE-Block is embedded within the ConvNeXt network structure, following the depthwise separable convolutional layers. The circuit structure of the channel attention module (SE-block) can also be seen to correspond to its processing steps of squeezing and excitation. Figure 5 As shown. The squeezing part consists of a single global average pooling layer. Taking the SE Block in the first ConvNeXt Block as an example, the output tensor after the depthwise separable convolutional layer is 56×56×96. Global pooling is performed on each channel to obtain a 96-dimensional vector A. Next is the excitation part... Figure 10 The circuit structure of the first fully connected layer, ReLU activation function module, second fully connected layer, and Sigmoid activation function module in the activation section of the channel attention module (SE-block) is shown. The ReLU activation function circuit is located on the row output side of the first fully connected circuit. After conversion by a current-to-voltage conversion circuit, the converted output voltage is output to the row input side of the second fully connected circuit. The Sigmoid activation function circuit is located on the column output side of the second fully connected circuit, and its output serves as the total output of the channel attention module (SE-block) circuit. The vector A output from the compression section enters the two fully connected layers of the activation section. Here, the compression ratio r is set to 8, so the number of neurons in the first fully connected layer is 96 / 8, or 12, and the number of neurons in the second layer is 96, consistent with the input dimension.

[0111] The circuit implementation of the Sigmoid function is in Figure 10 The lower left corner shows a commonly used Sigmoid function circuit structure, which consists of 6 MOSFETs, and its output voltage and input current are located at the same point.

[0112] 3 Examples

[0113] The following detailed description provides further information through specific implementation methods.

[0114] In this embodiment, the SE-ConvNeXt network based on memristors, as described in this invention, is applied to emotion recognition processing to verify the effectiveness of the emotion recognition network model based on the SE-ConvNeXt network. In this embodiment, the model is compared horizontally with other models in the same field based on the average accuracy metric. To demonstrate the advantages of the CKF algorithm, the model needs to be pruned. To maximize the sparsity of the model while ensuring good accuracy, we apply different pruning distributions to vertically compare the overall pruning rate and accuracy of the pruned model to obtain the optimal pruning distribution.

[0115] 3.1 Dataset Description

[0116] In this embodiment, the neural network model is evaluated on the classic FER-2013 dataset in the field of facial expression recognition. This dataset contains a total of 35,887 face images, with 28,709 samples in the training set, and 3,589 samples each in the test and validation sets. The dataset is divided into seven categories: “Angry,” “Disgust,” “Fear,” “Happy,” “Sad,” “Surprised,” and “Neutral,” corresponding to the emotions of “anger,” “disgust,” “fear,” “happiness,” “sadness,” “surprise,” and “neutral,” as shown in Table 1.

[0117] Table 1. Distribution of samples by category in the FER-2013 dataset.

[0118]

[0119] As can be seen from Table 1, the data distribution of this dataset is uneven. The Happy class in the training set has the largest number of data points, reaching 7215, accounting for 26.15% of the total data, while the Disgust class has the smallest number of data points, as low as 436, accounting for only 1.52% of the total data.

[0120] 3.2 Model Training Results and Analysis

[0121] Offline training of SE-ConvNeXt was performed using PyTorch, and the classification accuracy and confusion matrix of the model before pruning and under different pruning rates were examined. It should be noted that the pruning here refers to in-kernel weight pruning, and the pruning rate represents the proportion of kernel elements with zeros in each kernel to be pruned. This pruning is only applied to the convolutional kernels of each SE-ConvNeXt block. For pooling layers, all pooling layers in this model are global average pooling layers. Since the kernel elements of a single average pooling layer are equal, the CKF-based model has natural efficiency for average pooling layers. Since the kernel of a point convolutional layer has only one kernel element, no pruning is performed. Therefore, all pruning target layers in the model can be set as [Stem, Stage(i, j)], where Stage(i, j) represents Stage... i The j-th SE-ConvNeXt Block contains depthwise separable convolutional layers. Because the structure of the SE-ConvNeXt Block in this model is as follows... Figure 4 The diagram shows a fixed structure with three convolutional layers. The second and third layers are point convolutions, while only the first layer has a 7×7 kernel size, which allows for pruning. Therefore, a single SE-ConvNeXt Block can uniquely identify a pruning target layer.

[0122] First, define the indicator GPR (General Pruning Rate), as shown in formula (7):

[0123]

[0124] It describes the proportion of pruned weights among all prunable weights, where T is the set of target pruning layers, and for each layer i∈T, W i P represents the total number of elements in the i-th convolutional kernel. i The number of convolution kernel elements pruned in the i-th layer.

[0125] After setting different pruning distributions and conducting multiple experiments, the pruning distribution pattern shown in Table 2 was finally selected, with a GPR of 45.32%. The accuracy of the original model on the test set after training was 73.17%, while the accuracy of the pruned model was 71.86%, a decrease of only 1.31%. The confusion matrix of the model before and after pruning on the test set is shown below. Figure 11 As shown.

[0126] Table 2 shows the pruning distribution used.

[0127]

[0128] Table 3 shows the accuracy of each class before and after pruning. Whether in the original model or the pruned model, the accuracy of the Happy class is much higher than that of other classes. This may be because the Happy class has a large number of training samples and the happiness itself is easy to identify. The Sad class has the lowest accuracy and is easily confused with the Angry, Fear and Neural classes. This phenomenon is also the case for the human eye. On the other hand, the Disgust class, which has the lowest proportion of samples in the training set, has an accuracy higher than the average accuracy. This shows that the model does not rely heavily on the number of training samples and demonstrates the model's powerful feature extraction ability.

[0129] Table 3. Accuracy of each category before and after model pruning

[0130]

[0131] Table 4 shows a comparison of the accuracy of our model with other mainstream models in the field of emotion recognition based on the FER-2013 dataset. It can be seen that while our model's accuracy before pruning is not the best, it is perfectly acceptable in this metric and outperforms the original ConvNeXt model. The accuracy of the pruned model is still not low, but it effectively reduces model weight, and when combined with the CKF algorithm, it can further improve model efficiency and reduce hardware power consumption.

[0132] Table 4. Accuracy of each model based on the FER-2013 dataset

[0133]

[0134] The models in references [1] to [6] are shown in the following documents:

[0135] Literature [1]: Goodfellow, IJ, Erhan, D., Carrier, PL, Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, DH, Zhou, Y. ,Huyghebaert,C.,Chung,J.,Zhang,Y.,Athanasakis,I.,Bégin,D.,Brehard,G.,Gelinlin,P.,Lavoie,A.,…Bengio,Y.(2013).Challenges in Representation Learning:A report on the blackbox learning challenge.arXiv.

[0136] Reference [2]: Minaee, S., Minaei, M., & Abdolrashidi, A. (2021). Deep-Emotion: A simple but strong baseline for facial expression recognition. arXiv.

[0137] Reference [3]: Wang, J., Yang, X., Sun, L., & Zheng, A. (2019). Facial Expression Recognition with Deeply-Supervised Attention Network. Signal Processing: Image Communication, 74, 119–127.

[0138] Reference [4]: Li, H., Li, J., Yu, G., & Zhou, H. Y. (2023). ES-FER: Emotion-Semantics-based Facial Expression Recognition. arXiv.

[0139] Reference [5]: Zhang, J., Hong, Z., Su, G. M., & Liu, X. (2023). POSTER-V2: A Robust and Efficient Pipeline for Facial Expression Recognition. arXiv.

[0140] Reference [6]: de Oliveira, F. F., Cassales, G. L. F., Britto, A. S., & Oliveira, L. S. (2023). Efficiently Trainable Large-Scale Deep Neural Networks for Facial Expression Recognition. IEEE Access, 11, 23746–23759.

[0141] 3.3 Hardware Simulation and Analysis

[0142] The SE-ConvNeXt circuit is built based on a threshold memristor (VTEAM model, Voltage Threshold Adaptive Memristor). It features non-volatility and threshold resistance change characteristics; the memristor maintains its resistance state after power is off, and its resistance reversibly changes when the applied current or voltage exceeds its threshold. By changing parameters such as the amplitude, width, and number of applied voltage or current pulses, the resistance state of the threshold memristor can be precisely controlled. This work uses a method of keeping the voltage amplitude constant while changing the pulse width and number to introduce weights into the memristor. The threshold voltage V of the memristor used... t The value is set to 6V. The mapping relationship between the conductance value compiled into the memristor and the weight of the neural network is shown in formulas (3) and (4). This mapping can ensure that the conductance value of the memristor is within the effective range, so that the memristor maintains its resistance state when participating in the calculation.

[0143] This circuit structure based on the CKF algorithm and memristor cross array can directly adapt to neural network models under arbitrary pruning distributions. To demonstrate the power consumption of the circuit before and after pruning, this embodiment calculates the power consumption of the memristor in a mental health detection system. Formula (8) shows the calculation principle of the total power consumption of the memristor, which is divided into two parts: weight import and execution calculation. The first definite integral is the formula for the power consumption of the weight import process, and the second definite integral is the formula for the power consumption of the convolution operation process. The following will calculate the total power consumption of the memristor from these two aspects. In Formula (8), U represents the voltage applied to the memristor, and G represents the conductance of the memristor. The conductance range of the threshold memristor model used in this invention is [1μS, 1mS], therefore |σ| max =0.001S.

[0144]

[0145] For a memristor storing input and output data, its power consumption is divided into two parts: weighted input and read. The data is normalized, so the maximum value is 1, corresponding to a memristor conductance of 0.001S, the maximum conductance value. First, the power consumption for weighted input is calculated. To more precisely control the conductance change, a small pulse width is fixed, and the voltage amplitude is adjusted based on feedback, requiring approximately 10 pulse cycles. After each pulse cycle, the current conductance value is read and compared with the target value. The initial state of the memristor is an absolutely high-resistance state, i.e., R0. off =1MΩ, because we take the limiting value |σ| max As the target value, its average power was calculated to be approximately 20.05mW, which is the maximum power consumption during the weighting process. When reading the memristor conductance value, the memristor acts as a fixed resistor. Therefore, the read operation is achieved by applying a 1V voltage to read the output current, and its power consumption is a constant, i.e., 1mW.

[0146] For memristors used for convolution operations, their power consumption is divided into two parts: weight input and operation participation. Figure 12 The frequency distribution of the weights of all convolutional layers in the pruned model is shown. The maximum absolute value of the weights is 1.1688, and the mean absolute value of the overall weights is 0.031730. The mapping relationship between the weights and the conductance is shown in equations (3) and (4). |σ| is calculated. mean ≈0.027mS, with |σ| mean The average power calculated using the target weight value is 0.3204mW. The input data for each convolutional layer is normalized, resulting in a maximum input voltage of 1V. The average maximum power consumption for the memristor used in convolution operations is calculated to be 37.037mW. The number of cycles required for a single convolutional kernel depends on the size of the output data; for example, if the output data size of the Stem layer is 56, a single convolutional kernel requires 56 cycles. Therefore, the power consumption reduction ratio for convolution operations is the weighted average sparsity of each layer, where the weight is the proportion of the output data size of each layer to the sum of the output data sizes of all pruned layers. Thus, the power consumption for convolution operations is reduced by approximately 45.32%. It is evident that the co-design of this invention reduces the power consumption of the core convolutional operation by 45.32%, providing an efficient solution for deploying advanced models on resource-constrained devices.

[0147] 3.4 Summary:

[0148] This invention innovatively integrates channel attention and residual structure in the SE-ConvNeXt model, significantly enhancing fine-grained facial expression feature extraction capabilities. The proposed CKF pruning mechanism reconstructs the computation flow from a native hardware perspective, achieving precise synchronization of kernel weight reduction and invalid computation elimination. A dedicated circuit architecture designed with the non-volatile characteristics of memristors unlocks the application potential of in-memory computing technology in energy-sensitive scenarios. Experiments show that this scheme maintains a high recognition accuracy of 71.86% (only a 1.31% loss compared to the original model) while reducing computational energy consumption by 44.54%. Its deeply coupled "model-algorithm-circuit" optimization path provides a reusable technical blueprint for lightweight edge neural networks applied to emotion recognition. On the one hand, the collaborative innovation of the CKF algorithm and memristor circuits establishes a circuit implementation scheme of "pruning is equivalent to subtraction" and designs a corresponding SE-ConvNeXt network circuit structure, which solves the long-standing problem of software and hardware optimization disconnect in the field of neuromorphic computing and makes this hardware-friendly algorithm applicable to the circuit implementation of the model. On the other hand, the cross-layer feature calibration mechanism of the SE-ConvNeXt network provides a scalable architectural foundation for complex tasks such as emotion recognition based on microtables, and also provides an efficient solution for deploying advanced models on resource-constrained devices.

[0149] As can be seen, the technical solution of this invention demonstrates enormous potential and broad application prospects. First, the memristor-based "in-memory computing" hardware platform itself has significant room for exploration, such as attempting to use more complex memristor arrays to simulate more refined "attention" allocation processes in models. Second, the hardware-aware optimization concept of CKF holds great promise for application to data models with time-varying characteristics, such as sound and action sequences (e.g., spiking neural networks), enabling devices to understand richer emotional expressions. Finally, the technical solution of this invention can be applied to the development of ultra-low-power emotion recognition neuromorphic chip circuits.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A method for emotion recognition based on memristors and SE-ConvNeXt networks, characterized in that, Includes the following steps: S1. Construct an emotion recognition network model based on the SE-ConvNeXt network, and train the emotion recognition network model to obtain the trained emotion recognition network model. S2. Design a circuit structure based on a memristor array for the trained emotion recognition network model to obtain the emotion recognition network circuit structure. S3. Obtain the face image to be identified and perform preprocessing; S4. Convert the preprocessed face image into an analog voltage signal matrix using a digital-to-analog converter; S5. Input the analog voltage signal matrix into the circuit structure to perform emotion recognition and output the probability distribution of the emotion category of the corresponding face image.

2. The emotion recognition method based on memristor-based SE-ConvNeXt network according to claim 1, characterized in that, The emotion recognition network model includes an initial layer, an attention mechanism deep residual network layer, and a classification head, which are cascaded in sequence. The initial layer is used to perform feature projection transformation on the input face image to obtain an initial feature map; The deep residual network layer of the attention mechanism is used to extract channel attention features from the initial feature map to obtain a channel attention feature map; The classification head is used to perform emotion classification and recognition processing on the channel attention feature map to obtain the probability distribution results of the emotion category of the face image.

3. The emotion recognition method based on memristor-based SE-ConvNeXt network according to claim 2, characterized in that, In the emotion recognition network model: The initial layer includes a cascaded convolutional layer and a layer normalization layer; the convolutional layer includes several convolutional modules that perform convolution operations using a kernel-first algorithm, and the convolutional layer is used to perform downsampling transformation processing on the input image to obtain a downsampled feature map; the layer normalization layer is used to normalize the downsampled feature map in the channel dimension to obtain the initial feature map. The attention mechanism deep residual network layer comprises four cascaded stage layers, each stage layer consisting of several cascaded attention mechanism deep residual blocks. Each attention mechanism deep residual block includes an attention branch and a residual branch. The attention branch includes a cascaded deep separable convolutional module, a channel attention module, a layer normalization layer, an activation layer, and two point convolutional modules. The output of the deep separable convolutional module serves as the input to the channel attention module. The output of the channel attention module is multiplied by the output of the deep separable convolutional module and then transmitted to the layer normalization layer. After passing through the activation layer and the two point convolutional modules, the output is processed sequentially. The residual branch includes a point convolutional module. The output of the point convolutional module in the residual branch is added to the output of the attention branch to serve as the overall output of the attention mechanism deep residual block. The classification head includes a cascaded global average pooling layer and a fully connected layer. The global average pooling layer is used to reduce the dimensionality and normalize the input feature map, and then the fully connected layer performs fully connected and Softmax function activation to obtain the probability distribution prediction result of the emotion category of the face image.

4. The emotion recognition method based on memristor-based SE-ConvNeXt network according to claim 3, characterized in that, The channel attention module includes a global average pooling module, a first fully connected layer, a ReLU activation function module, a second fully connected layer, and a Sigmoid activation function module, which are cascaded in sequence. The global average pooling module is used to perform feature compression processing on the input feature map to obtain feature tensor data; the first fully connected layer is used to perform dimensionality reduction processing on the feature tensor data, and then perform ReLU activation operation through the ReLU activation function module, and then perform dimensionality increase processing through the second fully connected layer, and then perform Sigmoid activation operation through the Sigmoid activation function module to obtain the channel weight tensor, which is used as the overall output of the channel attention module.

5. The emotion recognition method based on memristor-based SE-ConvNeXt network according to claim 3, characterized in that, Step S2 involves implementing the emotion recognition network model through circuitry. The circuit of the deep residual block for the attention mechanism includes an input storage circuit, an attention residual operation and processing circuit, and an output storage circuit. The input storage circuit includes a number of memristor arrays equal to the number of channels of the input tensor of the attention mechanism deep residual block. Each memristor array stores the voltage pulse amplitude matrix corresponding to the input tensor of a single channel. Each memristor array includes several image storage columns, and each image storage column includes several input-side memristor storage units connected in parallel. The number of image storage columns and the number of input-side memristor storage units connected in parallel in each image storage column are the same as the number of columns and rows of the input tensor of a single channel input to the attention mechanism deep residual block. Each input-side memristor storage unit includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store the voltage pulse amplitude at the corresponding pixel position in the input tensor of a single channel. In the attention residual operation processing circuit, the output of each image storage column of each memristor array in the input storage circuit is output to the attention branch circuit and the residual branch circuit respectively through parallel branches; In the attention branch circuit, after being processed by the depthwise separable convolution circuit and the channel attention module circuit in sequence, the output of the channel attention module circuit is multiplied by the output of the depthwise separable convolution circuit through the multiplier circuit, and then passed through the layer normalization circuit, the ReLU activation circuit and two point convolution circuits in sequence. Finally, it is added in parallel with the output processed by the point convolution circuit in the residual branch circuit to obtain the channel attention feature tensor for output. In the output storage circuit, two convolutional storage branches are connected in parallel for each column output of the channel attention feature tensor: a positive convolutional storage branch and a negative convolutional storage branch. Each positive convolutional storage branch includes a number of positive memristor storage cells connected in parallel, the same number as the number of rows of the channel attention feature tensor, followed by a bias storage cell connected in parallel. Each positive memristor storage cell includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store positive channel attention feature tensor element values. The bias storage cell also includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store... The negative convolution storage branch includes a number of negative memristor storage cells connected in parallel, the same number as the number of rows of the channel attention feature tensor, and then connected in series with an inverter conversion circuit. Each negative memristor storage cell includes a 1T1R device composed of a memristor and a thyristor connected in series, used to store negative channel attention feature tensor element values. The inverter conversion circuit is used to convert the output of the negative memristor storage cell into a negative value. The outputs of the positive and negative convolution storage branches corresponding to each column output are connected in parallel and added together, then passed through a diode circuit representing the ReLU activation function before being output.

6. The emotion recognition method based on memristor-based SE-ConvNeXt network according to claim 5, characterized in that, In the attention residual calculation processing circuit, the channel attention module circuit includes a global average pooling circuit, a first-layer fully connected circuit, a ReLU activation function circuit, a second-layer fully connected circuit, and a Sigmoid activation function circuit connected in sequence. The ReLU activation function circuit is located on the row output side of the first-layer fully connected circuit. After being converted by the current-to-voltage conversion circuit, the converted output voltage is output to the row input side of the second-layer fully connected circuit. The Sigmoid activation function circuit is located on the column output side of the second-layer fully connected circuit, and the output of the Sigmoid activation function circuit serves as the total output of the channel attention module circuit.

7. The emotion recognition method based on memristor-based SE-ConvNeXt network according to claim 3 or 5, characterized in that, In step S3, the circuit structure of the emotion recognition network obtained by implementing the emotion recognition network model includes an input part, a convolution kernel operation part, and an output part. The input section includes a number of memristor arrays equal to the number of channels in the input tensor of the convolution module. Each memristor array stores the voltage pulse amplitude matrix corresponding to the input tensor of a single channel. Each memristor array includes several image storage columns, and each image storage column includes several input memristor storage units connected in parallel. The number of image storage columns and the number of input memristor storage units connected in parallel in each image storage column are the same as the number of columns and rows of the input tensor of a single channel input to the convolution module. Each input memristor storage unit includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store the voltage pulse amplitude at the corresponding pixel position in the input tensor of a single channel. In the convolution kernel operation section, a convolution kernel operation path is provided for each image storage column output by the memristor array in the input section. Each convolution kernel operation path includes a first voltage-to-current conversion circuit. The converted voltage output terminal of the first voltage-to-current conversion circuit is connected in parallel with two convolution processing branches. Each convolution processing branch includes a memristor, a first thyristor, a second voltage-to-current conversion circuit, and a second thyristor connected in series. In the convolution processing branch, the conductance value of the memristor is programmed to represent the convolution weight value, and its output current value is the voltage input value representing the input tensor pixel value and the convolution weight value. The first thyristor controls whether the convolution processing branch is turned on, and the second voltage-to-current conversion circuit converts the output current value of the memristor into a voltage value. The second thyristor controls whether the convolution processing branch participates in the convolution step size calculation. During operation, the two convolution processing branches are alternately turned on by the first thyristor. When one convolution processing branch is turned on and participates in the convolution operation, the other convolution processing branch programs the conductance value of the memristor. The output terminals of each convolution kernel operation path are connected in parallel according to the preset convolution step size setting relationship to obtain a convolution tensor of preset size for output. The output section corresponds to two convolution storage branches connected in parallel for each column output of the convolution tensor: a positive convolution storage branch and a negative convolution storage branch. Each positive convolution storage branch includes a number of positive memristor storage cells connected in parallel, the same number as the number of rows of the convolution tensor, and then a bias storage cell connected in parallel. Each positive memristor storage cell includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store positive convolution tensor element values. The bias storage cell also includes a 1T1R device consisting of a memristor and a thyristor connected in series, used to store bias values. The negative convolution storage branch includes a number of negative memristor storage cells connected in parallel, the same number as the number of rows of the convolution tensor, and then connected in series with an inverter conversion circuit. Each negative memristor storage cell includes a 1T1R device composed of a memristor and a thyristor connected in series, used to store the absolute values ​​of negative convolution tensor elements. The inverter conversion circuit is used to convert the output of the negative memristor storage cell into a negative value. The outputs of the positive and negative convolution storage branches corresponding to each column output are connected in parallel and added together, then passed through a diode circuit representing the ReLU activation function before being output.

8. The emotion recognition method based on memristor-based SE-ConvNeXt network according to claim 7, characterized in that, In the convolution kernel operation section of the convolution module, when the convolution processing branch programs the conductance value of the memristor, it only programs and sets the corresponding conductance value of the memristor when the convolution weight value is not 0. Otherwise, it skips the conductance value programming and sets the first thyristor in the corresponding convolution processing branch to be disconnected.

9. The emotion recognition method based on memristor-based SE-ConvNeXt network according to claim 3, 4, or 6, characterized in that, The global average pooling circuit includes a number of pooling channels equal to the number of channels in the input tensor of the global average pooling circuit. Each pooling channel includes a memristor array for storing the voltage pulse amplitude matrix corresponding to the input tensor of a single channel, and a current-to-voltage conversion circuit and a pooling storage memristor connected in series with the output of the memristor array. The memristor array includes several image storage columns, each image storage column including several pooling memristor storage units connected in parallel. The number of image storage columns and the number of pooling memristor storage units connected in parallel in each image storage column are the same as the number of columns and rows of the input tensor of a single channel input to the global average pooling circuit. Each pooling memristor storage unit includes a 1T1R device composed of a memristor and a thyristor connected in series, used to store the voltage pulse amplitude at the corresponding pixel position in the input tensor of a single channel. The image storage columns of the memristor array are added in parallel and then connected in series with the current-to-voltage conversion circuit and the pooling storage memristor.

10. The emotion recognition method based on memristor-based SE-ConvNeXt network according to claim 2, characterized in that, In step S3, the preprocessing method for the face image is as follows: after processing the face image into a grayscale image, it is converted into a face image tensor of a preset size; In step S4, the method of converting the preprocessed face image into an analog voltage signal is as follows: the gray value of each pixel in the face image tensor is converted into a corresponding voltage pulse amplitude, and the voltage pulse amplitude matrix corresponding to the face image tensor is obtained.

Images